Navigation:  thinBasic Modules > Tokenizer >

Tokenizer equates

Previous pageReturn to chapter overviewNext page

Equates for Tokenizer module.

 

Symbol

Meaning

%TOKENIZER_FINISHED

The returned token is end of buffer

%TOKENIZER_ERROR

There was an error in parsing next token

%TOKENIZER_UNDEFTOK

The next token is undefined

%TOKENIZER_EOL

The next token is end of line

%TOKENIZER_DELIMITER

The next token is a delimiter

%TOKENIZER_NUMBER

The next token is a number

%TOKENIZER_STRING

The next token is a string

%TOKENIZER_QUOTE

The next token is a quoted string

 

 

The following equates are mainly used by the following functions:

Tokenizer_Default_Char
Tokenizer_Default_Code
Tokenizer_Default_Get
Tokenizer_Default_Set

 

Symbol

Meaning

%TOKENIZER_DEFAULT_NewLine

Identify chars belonging to NewLine group

%TOKENIZER_DEFAULT_Space

Identify chars belonging to Space group

%TOKENIZER_DEFAULT_DQuote

Identify chars belonging to Double Quote group

%TOKENIZER_DEFAULT_Delim

Identify chars belonging to Delimiters group

%TOKENIZER_DEFAULT_Numeric

Identify chars belonging to Numeric group

%TOKENIZER_DEFAULT_Alpha

Identify chars belonging to Alphabetic group