Hi folks,
I looked over the tokenizer sample script and I liked what I saw. My question is:
Can the returning token types by configurized, means adding new ones or changing current ones?
Michael
Hi folks,
I looked over the tokenizer sample script and I liked what I saw. My question is:
Can the returning token types by configurized, means adding new ones or changing current ones?
Michael
Hi Mike,
please give me more details about your idea. I like it a lot but would like to know more.
I imagine you are talking about the possibility to pass lists of tokens and assigning them a new token type or something like that.
Let me know.
Eros
www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000
Exactly, passing a list of new tokens and assigning them to a new token type.
Or changing the current tokentypes, means what they are assigned to.
To learn thinBasic better I got the idea to port/recode my FANTOM project to thinBasic. Nothing to compete with thinBasic, no no. Just to see how far I could get. And the tokenizer is a great functionality of TB. I know, I could make my own GetToken function, where I take the output of the TB command, and analyse it. But if there would be a way to customize the tokenizer, then it would be great. Even right now it is a great functionality.
OK Mike. I will see what I can do. I like it and should not be so difficult.
www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000
Mike, I looked over your new site a few times in the passsed few weeks, I couldn't find the fantom page. I really loved that logo you had done for it and the coloring scheme on the page.
Maybe you can be part of thinBasicII development team in the future. TB2, with full support for objects, like smalltalk and ruby but faster!!
We need to use that fantom logo somewhere in the game. That was sweet looking from what I remember!!
Acer Notebook: Win 10 Home 64 Bit, Core i7-4702MQ @ 2.2Ghz, 12 GB RAM, nVidia GTX 760M and Intel HD 4600
Raspberry Pi 3: Raspbian OS use for Home Samba Server and Test HTTP Server
Mike,
a possible idea:
Tokenizer_GetNextToken would change from
[code=thinbasic]
Tokenizer_GetNextToken(MyBuffer, CurrentPosition, TokenType, Token)
[/code]
to
[code=thinbasic]
Tokenizer_GetNextToken(MyBuffer, CurrentPosition, TokenType, Token [, TokenSubType])
[/code]
so an optional sub type can be returned.
Than I create a new module function that allows to add new special keys. Something like:
[code=thinbasic]
Tokenizer_KeyAdd(NewKeyString, KeyMainType, KeySubType)
[/code]
For example you can add keys like:
[code=thinbasic]
%Fantom = 100
%Fantom_Var = 1
%Fantom_Int = 2
%Fantom_As = 3
Tokenizer_KeyAdd("var", %Fantom, %Fantom_Var)
Tokenizer_KeyAdd("int", %Fantom, %Fantom_Int)
Tokenizer_KeyAdd("as" , %Fantom, %Fantom_As)
[/code]
When Tokenizer_GetNextToken will find a token of %TOKENIZER_STRING type will make an additional search between the user added keywords. If found the new main type and an optional subtype will be returned.
I will use an internal hash table that will store a special structure for every new key added.
What do you think? Any other idea?
Let me know.
Eros
www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000
Mike,
sorry but I didn't wait for your reply :P
Please get attached new module and an example. Can give you an idea of what I was talking about.
I'm still open to any request or change.
Let me know.
Eros
www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000
Thanks Eros, I will give it a whirl
Works like a charm. I like it.
Do see a possibility to change the type of the predefined ones?
For an example, the $ character is a delimiter right now. I would like to include it inside the STRING type. Would there be an easy way for you to implement this? Or the ":" character, I would like to change this to be an EOL token.
I hope I made myself clear. If not, please aks again.
Let me check.
I say now "yes, of course" but I need to change a bit of code because delimiters are now hard coded.
I will change them from hard coded to dynamic with possibility to change predefined main type and sub type.
www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000
Bookmarks