Search:

YES, I like it! :) You give to much! :)

What ever you think is fine. Like I said, even now I can perfectly work with it. Thanks for all the improvements.

Works like a charm. I like it.

Do see a possibility to change the type of the predefined ones?
For an example, the $ character is a delimiter right now. I would like to include it inside the...

Thanks Eros, I will give it a whirl :D

Exactly, passing a list of new tokens and assigning them to a new token type.
Or changing the current tokentypes, means what they are assigned to.

To learn thinBasic better I got the idea to...

Hi folks,

I looked over the tokenizer sample script and I liked what I saw. My question is:

Can the returning token types by configurized, means adding new ones or changing current ones?
...