Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Tokenizer: Configurizable?

  1. #1

    Tokenizer: Configurizable?

    Hi folks,

    I looked over the tokenizer sample script and I liked what I saw. My question is:

    Can the returning token types by configurized, means adding new ones or changing current ones?

    Michael

  2. #2
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    Hi Mike,

    please give me more details about your idea. I like it a lot but would like to know more.
    I imagine you are talking about the possibility to pass lists of tokens and assigning them a new token type or something like that.

    Let me know.
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  3. #3

    Re: Tokenizer: Configurizable?

    Exactly, passing a list of new tokens and assigning them to a new token type.
    Or changing the current tokentypes, means what they are assigned to.

    To learn thinBasic better I got the idea to port/recode my FANTOM project to thinBasic. Nothing to compete with thinBasic, no no. Just to see how far I could get. And the tokenizer is a great functionality of TB. I know, I could make my own GetToken function, where I take the output of the TB command, and analyse it. But if there would be a way to customize the tokenizer, then it would be great. Even right now it is a great functionality.

  4. #4
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    OK Mike. I will see what I can do. I like it and should not be so difficult.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  5. #5
    thinBasic MVPs kryton9's Avatar
    Join Date
    Nov 2006
    Location
    Naples, Florida & Duluth, Georgia
    Age
    67
    Posts
    3,869
    Rep Power
    404

    Re: Tokenizer: Configurizable?

    Mike, I looked over your new site a few times in the passsed few weeks, I couldn't find the fantom page. I really loved that logo you had done for it and the coloring scheme on the page.

    Maybe you can be part of thinBasicII development team in the future. TB2, with full support for objects, like smalltalk and ruby but faster!!

    We need to use that fantom logo somewhere in the game. That was sweet looking from what I remember!!
    Acer Notebook: Win 10 Home 64 Bit, Core i7-4702MQ @ 2.2Ghz, 12 GB RAM, nVidia GTX 760M and Intel HD 4600
    Raspberry Pi 3: Raspbian OS use for Home Samba Server and Test HTTP Server

  6. #6
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    Mike,
    a possible idea:

    Tokenizer_GetNextToken would change from
    [code=thinbasic]
    Tokenizer_GetNextToken(MyBuffer, CurrentPosition, TokenType, Token)
    [/code]
    to
    [code=thinbasic]
    Tokenizer_GetNextToken(MyBuffer, CurrentPosition, TokenType, Token [, TokenSubType])
    [/code]
    so an optional sub type can be returned.

    Than I create a new module function that allows to add new special keys. Something like:
    [code=thinbasic]
    Tokenizer_KeyAdd(NewKeyString, KeyMainType, KeySubType)
    [/code]
    For example you can add keys like:
    [code=thinbasic]
    %Fantom = 100
    %Fantom_Var = 1
    %Fantom_Int = 2
    %Fantom_As = 3

    Tokenizer_KeyAdd("var", %Fantom, %Fantom_Var)
    Tokenizer_KeyAdd("int", %Fantom, %Fantom_Int)
    Tokenizer_KeyAdd("as" , %Fantom, %Fantom_As)
    [/code]

    When Tokenizer_GetNextToken will find a token of %TOKENIZER_STRING type will make an additional search between the user added keywords. If found the new main type and an optional subtype will be returned.

    I will use an internal hash table that will store a special structure for every new key added.

    What do you think? Any other idea?
    Let me know.
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  7. #7
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    Mike,

    sorry but I didn't wait for your reply :P
    Please get attached new module and an example. Can give you an idea of what I was talking about.
    I'm still open to any request or change.

    Let me know.
    Eros
    Attached Files Attached Files
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  8. #8

    Re: Tokenizer: Configurizable?

    Thanks Eros, I will give it a whirl

  9. #9

    Re: Tokenizer: Configurizable?

    Works like a charm. I like it.

    Do see a possibility to change the type of the predefined ones?
    For an example, the $ character is a delimiter right now. I would like to include it inside the STRING type. Would there be an easy way for you to implement this? Or the ":" character, I would like to change this to be an EOL token.

    I hope I made myself clear. If not, please aks again.

  10. #10
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    Let me check.
    I say now "yes, of course" but I need to change a bit of code because delimiters are now hard coded.
    I will change them from hard coded to dynamic with possibility to change predefined main type and sub type.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

Page 1 of 2 12 LastLast

Members who have read this thread: 1

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •