Page 2 of 2 FirstFirst 12
Results 11 to 13 of 13

Thread: Tokenizer: Configurizable?

  1. #11

    Re: Tokenizer: Configurizable?

    What ever you think is fine. Like I said, even now I can perfectly work with it. Thanks for all the improvements.

  2. #12
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: Tokenizer: Configurizable?

    Mike, I've done it. Hope you will like.
    You will be able to set any of the 255 ascii table char to any of the following groups: NewLine, Space, DQuote, Delim, Numeric, Alpha
    This at any time and anywhere during script execution. Yu will also able to set in group, so pass a list of chars to be set into a group.

    An example:
    [code=thinbasic]
    '---Create a new keywords group. Assign a value >= 100
    dim MyDictionary as long value 100

    '---Add keywords to the group
    tokenizer_KeyAdd("DIM" , MyDictionary, 1)
    tokenizer_KeyAdd("AS" , MyDictionary, 2)
    tokenizer_KeyAdd("CONSOLE_writeline" , MyDictionary, 3)
    '...
    '---Add as many keywords you need
    '...

    '---Now change some default behave in char parsing
    Tokenizer_Default_Char("$", %TOKENIZER_DEFAULT_Alpha)
    Tokenizer_Default_Char("%", %TOKENIZER_DEFAULT_Alpha)
    Tokenizer_Default_Char(":", %TOKENIZER_DEFAULT_newline)
    Tokenizer_Default_Char(";", %TOKENIZER_DEFAULT_newline)

    '---... or even faster
    Tokenizer_Default_Set("$%", %TOKENIZER_DEFAULT_Alpha)
    Tokenizer_Default_Set(":;", %TOKENIZER_DEFAULT_newline)
    [/code]

    I'm going home now (I'm at work right now).
    I need to fix some few points and than post a new update. Maybe tomorrow, sorry.

    Ciao
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  3. #13

    Re: Tokenizer: Configurizable?

    YES, I like it! You give to much!

Page 2 of 2 FirstFirst 12

Members who have read this thread: 1

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •