ok, i played already a little
i tried to replace line 108 with something as this:
PrintStep "Number of Tokens during scan: " & MyParser.Scan("Dim MyVariable As Long " & $CRLF _
& "Dim another variable like " & $DQ & "something" & $DQ & " At 0" & $CRLF _
& "produce " & $DQ & "ERROR" & $CRLF _
& "another line" )
- i see it tokenizes all at once and i can check the result when done.
In my current project i scan single lines of code only (using classic tokenizer) , it's hard because of line-continiuation but simpler because don't have to calculate line for a token and can trap Errors.
What it would need were some important information of Position.
let's say this
String n = "12345678901234567890" ' (this just for orientation)
String s = "THIS ARE" & $CRLF _
& "A FEW LOSE TOKENS"
If for example after parsing s
MyParser.Token(1).Data ' will return "THIS"
MyParser.Token(2).Data ' will return "ARE"
MyParser.Token(3).Data ' will return $CRLF (EOL, 2 Bytes) i guess,
i need something like
MyParser.Token(1).ContinueAt ' to return 5
MyParser.Token(2).ContinueAt ' return 19 here...
MyParser.Token(3).ContinueAt ' would return 21 then
so i can subtract Length of token from the position where the parser continues at to scan for the next token to find out the position of the actual token in the text,
in basic expression:
Long tokenXStartPos = MyParser.Token(X).ContinueAt - Len(MyParser.Token(X).data)
You could as well add
MyParser.Token(X).StartsAt ' to retrieve Starting-position
to make it more simple for the users but more effort for you...
Also useful could be to retrieve
MyParser.Token(X).LineNumber
( only real CRLF probably, tells on which line the token was found )
MyParser.Token(X).LinePosition
( tell if this is the 1st, 2nd, 3rd, 4th, 5th... token on the current line )
expressed in basic after parsing s (see above)
MyParser.Token(1).Data ' returns "THIS"
MyParser.Token(1).LineNumber ' returns 1
MyParser.Token(1).LinePosition ' returns 1
MyParser.Token(2).Data ' returns "ARE"
MyParser.Token(2).LineNumber ' returns 1
MyParser.Token(2).LinePosition ' returns 2
MyParser.Token(3).Data ' returns $CRLF
MyParser.Token(3).LineNumber ' returns 1
MyParser.Token(3).LinePosition ' returns 3
MyParser.Token(4).Data ' returns "A"
MyParser.Token(4).LineNumber ' returns 2
MyParser.Token(4).LinePosition ' returns 1
MyParser.Token(5).Data ' returns "FEW"
MyParser.Token(5).LineNumber ' returns 2
MyParser.Token(5).LinePosition ' returns 2
MyParser.Token(6).Data ' returns "LOSE"
MyParser.Token(6).LineNumber ' returns 2
MyParser.Token(6).LinePosition ' returns 3
' ...
Bookmarks