Tokenizer equates

<< Click to Display Table of Contents >>

Navigation:  ThinBASIC Modules > Tokenizer >

Tokenizer equates

Equates for Tokenizer module.

 

Symbol

Meaning

%TOKENIZER_FINISHED

The returned token is end of buffer

%TOKENIZER_ERROR

There was an error in parsing next token

%TOKENIZER_UNDEFTOK

The next token is undefined

%TOKENIZER_EOL

The next token is end of line

%TOKENIZER_DELIMITER

The next token is a delimiter

%TOKENIZER_NUMBER

The next token is a number

%TOKENIZER_STRING

The next token is a string

%TOKENIZER_QUOTE

The next token is a quoted string

 

 

The following equates are mainly used by the following functions:

Tokenizer_Default_Char

Tokenizer_Default_Code

Tokenizer_Default_Get

Tokenizer_Default_Set

 

Symbol

Meaning

%TOKENIZER_DEFAULT_NewLine

Identify chars belonging to NewLine group

%TOKENIZER_DEFAULT_Space  

Identify chars belonging to Space group

%TOKENIZER_DEFAULT_DQuote

Identify chars belonging to Double Quote group

%TOKENIZER_DEFAULT_Delim  

Identify chars belonging to Delimiters group

%TOKENIZER_DEFAULT_Numeric

Identify chars belonging to Numeric group

%TOKENIZER_DEFAULT_Alpha  

Identify chars belonging to Alphabetic group