Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 25

Thread: Tokenizer- user-keys, but how ???

  1. #11
    Super Moderator Petr Schreiber's Avatar
    Join Date
    Aug 2005
    Location
    Brno - Czech Republic
    Posts
    7,128
    Rep Power
    732
    I think the timing is already very, very good - I would not worry about optimizing too much at this stage.

    Good night!,
    Petr

    P.S. GOTO 4EVER (just not in ThinBASIC, for our coders peaceful dreams)
    Last edited by Petr Schreiber; 19-11-2015 at 00:39.
    Learn 3D graphics with ThinBASIC, learn TBGL!
    Windows 10 64bit - Intel Core i5-3350P @ 3.1GHz - 16 GB RAM - NVIDIA GeForce GTX 1050 Ti 4GB

  2. #12
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Regarding character setup, I've adopted the following syntax: object.Char.Set.{Alpha|NewLine|Space|Delim|Numeric|DQuote}

    Example:
      '---Declare a new tokenizer engine
      Dim MyParser As cTokenizer
    
    
      '---Instantiate new tokenizer
      MyParser = New cTokenizer()
    
    ...
    
    
        MyParser.Char.Set.NewLine (":;")
        MyParser.Char.Set.Space   ($SPC & $TAB)
        MyParser.Char.Set.Delim   (",.-+*/")
        MyParser.Char.Set.Numeric ("0123456789")
        MyParser.Char.Set.Alpha   ("$%ABCDabcd")
        MyParser.Char.Set.DQuote  ($DQ)
    
    ...
    
    I will now work on:
    1. key group creation
    2. few utility functions (.keys.delete, .keys.list, ...)
    3. loading string buffer of test to parse
    4. parsing
    5. scanning

    After that I will release a working new Tokenizer module to test.

    Ciao
    Eros
    Last edited by ErosOlmi; 19-11-2015 at 08:33.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  3. #13
    Super Moderator Petr Schreiber's Avatar
    Join Date
    Aug 2005
    Location
    Brno - Czech Republic
    Posts
    7,128
    Rep Power
    732
    Reads great, looking forward to play more with it


    Petr
    Learn 3D graphics with ThinBASIC, learn TBGL!
    Windows 10 64bit - Intel Core i5-3350P @ 3.1GHz - 16 GB RAM - NVIDIA GeForce GTX 1050 Ti 4GB

  4. #14
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Tokenizer: new module

    Note: attached module removed because a new one is posted in a successive post of this thread

    I've something to play with.
    Just very rough, not using 100% configured options and char recognition, but just to give you an idea I will publish it right now.

    You need thinBasic Beta 1.9.16.4 installed. See http://www.thinbasic.com/community/showthread.php?12600
    Get attached Tokenizer module and substitute you current one in \thinBasic\Lib\


    Following example will give you an idea on how to setup, get info, scan a text, speed and new syntax
    '---
    '---Loads needed modules
    '---
    Uses "File"
    Uses "Console"
    Uses "Tokenizer"
    
    
    Long cBrightCyanBGBlue  = %CONSOLE_FOREGROUND_BLUE | %CONSOLE_FOREGROUND_GREEN | %CONSOLE_FOREGROUND_INTENSITY | %CONSOLE_BACKGROUND_BLUE
    Long cBrightGreen       = %CONSOLE_FOREGROUND_GREEN | %CONSOLE_FOREGROUND_INTENSITY
    Long cBrightMagenta     = %CCOLOR_FMAGENTA | %CONSOLE_FOREGROUND_INTENSITY
    Long cBrightRed         = %CONSOLE_FOREGROUND_RED | %CONSOLE_FOREGROUND_INTENSITY
    
    
    
    
    '--------------------------------------------------------
    '
    '--------------------------------------------------------
    Function TBMain() As Long
      Long Counter
    
    
      '---Create a timer
      Local lTimer As cTimer
      lTimer = New cTimer
      
      '---Declare a new tokenizer engine
      Dim MyParser As cTokenizer
    
    
      '---Instantiate new tokenizer
      MyParser = New cTokenizer()
    
    
      PrintTitle "Configure Tokenizer"
        '---Change some default behave in char parsing
        '---Every single ASCII char can be associated to one of the predefined 6 char sets
        '---A set of characters can be indicated in one go using .Char.Set. method followed by the type:
        '--- NewLine, Space, Delim,Numeric, Alpha, DQuote
        PrintStep "Setting character sets (if needed)"
        MyParser.Char.Set.NewLine (":;")
        MyParser.Char.Set.Space   ($SPC & $TAB)
        MyParser.Char.Set.Delim   (",.-+*/")
        MyParser.Char.Set.Numeric ("0123456789")
        MyParser.Char.Set.Alpha   ("$%ABCDabcd")
        MyParser.Char.Set.DQuote  ($DQ)
      
        MyParser.Options.CaseSensitive = %FALSE
        PrintStep "Tokenizer option, CaseSensitive = " & MyParser.Options.CaseSensitive
    
    
      PrintTitle "Loading keys"
        '---Create a new keywords group. Assign a value >= 100
        Dim MyKeys    As Long Value 100
      
        Dim sFile         As String = APP_Path & "thinAir\Syntax\thinBasic\thinBasic_Keywords.ini"
        Dim allKeywords() As String
        Dim nKeys         As Long
      
        nKeys = Parse File sFile, allKeywords, $CRLF
        PrintStep "Number of keys I'm going to load: " & nkeys
        lTimer.Start
        For Counter = 1 To UBound(allKeywords)
          MyParser.Keys.Add(allKeywords(Counter), MyKeys, Counter)
        Next
        lTimer.Stop
        PrintStep "Number of keys loaded: " & MyParser.Keys.Count
        PrintStep "Loading time Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
      
      PrintTitle "Checking some keys"    
        PrintStep "If the following few lines will return a number, all is ok"
        PrintData MyParser.Keys.Contain("Dim"   )
        PrintData MyParser.Keys.Contain("As"    )
        PrintData MyParser.Keys.Contain("PrintL" )
        PrintData MyParser.Keys.Contain("Uses"  )
        PrintData MyParser.Keys.Contain("Zer")
    
    
        PrintStep "Checking MainType and SubType of DIM key"
        PrintData "DIM MainType: " & MyParser.Key("Dim").MainType
        PrintData "DIM SubType : " & MyParser.Key("Dim").SubType
    
    
      PrintTitle "A BIG search of 100K Keys"    
        PrintStep "Searching for 100K keys, ..."
        lTimer.Start
        For Counter = 1 To 100000
          MyParser.Key("Dim").MainType
        Next
        lTimer.Stop
        PrintStep "100K search time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
      PrintTitle "-Start Scanning 100 times current script source code---"
        lTimer.Start
        PrintStep "Number of Tokens during scan: " & MyParser.Scan(Repeat$(100, Load_File(APP_ScriptFullName))) 
        lTimer.Stop
        PrintStep "Scan time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
    
    
      PrintTitle "Scanning a line of text in order to see how to scan a new source and to access single tokens"
        lTimer.Start
        PrintStep "Number of Tokens during scan: " & MyParser.Scan("Dim MyVariable As Long") 
        PrintStep "Number of Tokens count      : " & MyParser.Tokens.Count 
     
        For Counter = 1 To MyParser.Tokens.Count
          PrintData "Token Data    :" & MyParser.Token(Counter).Data
          PrintData "Token MainType:" & MyParser.Token(Counter).MainType & " (" & MyParser.Token(Counter).MainType.ToString & ")"
          PrintData "Token SubType :" & MyParser.Token(Counter).SubType
          PrintData "-------------------------------------------"
        Next
        lTimer.Stop
        PrintStep "Scan time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
        PrintWarning "All done. Press a key to finish"    
      WaitKey
    
    
    End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintTitle(ByVal sTitle As String)
        PrintL
        PrintL "-" & sTitle & Repeat$(78 - Len(sTitle), "-") In cBrightCyanBGBlue    
      End Function
      
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintStep(ByVal sStep As String)
        PrintL "  " & sStep In cBrightGreen    
      End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintData(ByVal sData As String)
        PrintL "    " & sData In cBrightMagenta    
      End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintWarning(ByVal sData As String)
        PrintL
        PrintL sData In cBrightRed    
      End Function
    
    Last edited by ErosOlmi; 25-11-2015 at 08:14.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  5. #15
    thinBasic MVPs
    Join Date
    Oct 2012
    Location
    Germany
    Age
    54
    Posts
    1,527
    Rep Power
    170
    ok, i played already a little

    i tried to replace line 108 with something as this:
     PrintStep "Number of Tokens during scan: " & MyParser.Scan("Dim MyVariable As Long " & $CRLF _
                                                                 & "Dim another variable like " & $DQ & "something" & $DQ & " At 0" & $CRLF _
                                                                 & "produce " & $DQ & "ERROR" & $CRLF _
                                                                 & "another line" )
    
    - i see it tokenizes all at once and i can check the result when done.
    In my current project i scan single lines of code only (using classic tokenizer) , it's hard because of line-continiuation but simpler because don't have to calculate line for a token and can trap Errors.

    What it would need were some important information of Position.
    let's say this
    String n = "12345678901234567890"  ' (this just for orientation)
    String s = "THIS           ARE" & $CRLF _
             & "A FEW  LOSE   TOKENS"
    
    If for example after parsing s
    MyParser.Token(1).Data ' will return "THIS"
    MyParser.Token(2).Data ' will return "ARE"
    MyParser.Token(3).Data ' will return $CRLF (EOL, 2 Bytes) i guess,
    
    i need something like
    MyParser.Token(1).ContinueAt  ' to return 5 
     MyParser.Token(2).ContinueAt ' return 19 here...
    MyParser.Token(3).ContinueAt ' would return 21 then
    
    so i can subtract Length of token from the position where the parser continues at to scan for the next token to find out the position of the actual token in the text,

    in basic expression:

    Long tokenXStartPos = MyParser.Token(X).ContinueAt - Len(MyParser.Token(X).data)
    
    You could as well add
    MyParser.Token(X).StartsAt ' to retrieve Starting-position
    
    to make it more simple for the users but more effort for you...

    Also useful could be to retrieve
    MyParser.Token(X).LineNumber
    ( only real CRLF probably, tells on which line the token was found )
    MyParser.Token(X).LinePosition
    ( tell if this is the 1st, 2nd, 3rd, 4th, 5th... token on the current line )

    expressed in basic after parsing s (see above)
    MyParser.Token(1).Data           ' returns "THIS"
    MyParser.Token(1).LineNumber ' returns 1
    MyParser.Token(1).LinePosition ' returns 1
    
    MyParser.Token(2).Data           ' returns "ARE"
    MyParser.Token(2).LineNumber ' returns 1
    MyParser.Token(2).LinePosition ' returns 2
    
    MyParser.Token(3).Data           ' returns $CRLF
    MyParser.Token(3).LineNumber ' returns 1
    MyParser.Token(3).LinePosition ' returns 3
    
    MyParser.Token(4).Data           ' returns "A"
    MyParser.Token(4).LineNumber ' returns 2
    MyParser.Token(4).LinePosition ' returns 1
    
    MyParser.Token(5).Data           ' returns "FEW"
    MyParser.Token(5).LineNumber ' returns 2
    MyParser.Token(5).LinePosition ' returns 2
    
    MyParser.Token(6).Data           ' returns "LOSE"
    MyParser.Token(6).LineNumber ' returns 2
    MyParser.Token(6).LinePosition ' returns 3
    ' ...
    
    Last edited by ReneMiner; 24-11-2015 at 12:55.
    I think there are missing some Forum-sections as beta-testing and support

  6. #16
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Hi René,

    sure I can add your requests.
    I already had the idea to add byte position of token but your idea to have both (start of current and next one) is better.
    Also OK to get line number using only real lines.

    SCAN does the job in one go but it needs more info to operate like "removecomments" or options like that before scanning.
    I just wanted to be sure to be on the right track and you liked it.

    Ciao
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  7. #17
    Super Moderator Petr Schreiber's Avatar
    Join Date
    Aug 2005
    Location
    Brno - Czech Republic
    Posts
    7,128
    Rep Power
    732
    Hi Eros,

    the parsing speed is uberawesome!!! It aslo reads very nicely, very fluent. This really opens whole new range of possibilities.

    Just one thing I am not sure about - what is the meaning of Counter as 3rd parameter here:
    MyParser.Keys.Add(allKeywords(Counter), MyKeys, Counter)
    
    Is it something like absolute position in keyword list? I would expect the .Add to just append at end, without need to care about position. If you could explain a bit, I would appreciate - maybe I got it wrong.

    Ideas:
    .Keys.AddRange could allow to add whole string array of keys, to avoid need of iterating via FOR/NEXT

    I also support ideas by Rene, very clever as usual

    thank youu!
    Petr
    Learn 3D graphics with ThinBASIC, learn TBGL!
    Windows 10 64bit - Intel Core i5-3350P @ 3.1GHz - 16 GB RAM - NVIDIA GeForce GTX 1050 Ti 4GB

  8. #18
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Last number is in reality your personal index to recognize your keys when encountered.

    Imagine you want to create an interpreter or a descendant parser able to interpret numeric equations with formulas and basic keywords.
    You add your keys like SIN, COS, TAN with your key dictionary, like 100 or whatever

    Every time the parser encounter SIN or COS or TAN will return that they are just KEYS of the dictionary 100.
    But that's not enough if you want to do something to react to SIN or react to COS or react to TAN.

    You need that: SIN belongs to dictionary 100 and is ID = 1234 (for example) in order to identify that the parser has found a key of your dictionary and that key is identified by 1234 (or whatever)
    The same for COS: parser must return that it found a key of dictionary 100 and that key is 654 (or whatever)
    The same for TAN: parser must return that it found a key of dictionary 100 and that key is 444 (or whatever)

    Of course you can always identify your key by its name (token name) but making a selection using strings is very very slow compared to numbers.
    Anyway it is just an option that programmer can use or not. The ID of the keys can be an index to an internal array or can be a pointer to a function. It is up to the programmer to use or not.
    I will make it optional in next version, I agree it is better.

    Ciao
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  9. #19
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Attached to this post an updated version of Tokenizer module.
    Extract thinBasic_Tokenizer.dll and substitute the one you have into \thinBasic\Lib\ directory

    Note: you need thinBasic 1.9.16.4 minimum version in order to test it: http://www.thinbasic.com/community/showthread.php?12600



    What' new in this:


    1. user defined main types MUST be created with a dedicated method with an optional description
      '---Create a new keywords group.
      Dim MyKeys    As Long Value MyParser.NewMainType("MyKeys")
      
    2. during .Scan operation User keys (user main types) are now identified
    3. token information stored during .Scan method has now token length and absolute token start and end position inside scanned string for easy retrieve exact position
            PrintData "Token Data    :" & MyParser.Token(Counter).Data
            PrintData "Token MainType:" & MyParser.Token(Counter).MainType & " (" & MyParser.Token(Counter).MainType.ToString & ")"
            PrintData "Token SubType :" & MyParser.Token(Counter).SubType
            PrintData "Token PosStart:" & MyParser.Token(Counter).PosStart
            PrintData "Token PosEnd  :" & MyParser.Token(Counter).PosEnd
            PrintData "Token Length  :" & MyParser.Token(Counter).Len
      


    If this version will be OK, I will work next on recognizing René suggestions:
    1. Token absolute line inside scanned source code (absolute means real lines and not line continuation)
    2. Token relative position inside line (1, 2, 3, ...)



    Full example showing new functionalities:
    '---'---Loads needed modules
    '---
    Uses "File"
    Uses "Console"
    Uses "Tokenizer"
    
    
    Long cBrightCyanBGBlue  = %CONSOLE_FOREGROUND_BLUE | %CONSOLE_FOREGROUND_GREEN | %CONSOLE_FOREGROUND_INTENSITY | %CONSOLE_BACKGROUND_BLUE
    Long cBrightGreen       = %CONSOLE_FOREGROUND_GREEN | %CONSOLE_FOREGROUND_INTENSITY
    Long cBrightMagenta     = %CCOLOR_FMAGENTA | %CONSOLE_FOREGROUND_INTENSITY
    Long cBrightRed         = %CONSOLE_FOREGROUND_RED | %CONSOLE_FOREGROUND_INTENSITY
    
    
    
    
    '--------------------------------------------------------
    '
    '--------------------------------------------------------
    Function TBMain() As Long
      Long Counter
    
    
      '---Create a timer
      Local lTimer As cTimer
      lTimer = New cTimer
      
      '---Declare a new tokenizer engine
      Dim MyParser As cTokenizer
    
    
      '---Instantiate new tokenizer
      MyParser = New cTokenizer()
    
    
      PrintTitle "Configure Tokenizer"
        '---Change some default behave in char parsing
        '---Every single ASCII char can be associated to one of the predefined 6 char sets
        '---A set of characters can be indicated in one go using .Char.Set. method followed by the type:
        '--- NewLine, Space, Delim,Numeric, Alpha, DQuote
        PrintStep "Setting character sets (if needed)"
        MyParser.Char.Set.NewLine (":;")
        MyParser.Char.Set.Space   ($SPC & $TAB)
        MyParser.Char.Set.Delim   (",.-+*/")
        MyParser.Char.Set.Numeric ("0123456789")
        MyParser.Char.Set.Alpha   ("$%ABCDabcd")
        MyParser.Char.Set.DQuote  ($DQ)
      
        MyParser.Options.CaseSensitive = %FALSE
        PrintStep "Tokenizer option, CaseSensitive = " & MyParser.Options.CaseSensitive
    
    
      PrintTitle "Loading keys"
        '---Create a new keywords group. NEVER USE YOU OWN MANUAL KEYs code, always use this method to get a new one
        Dim MyKeys    As Long Value MyParser.NewMainType("MyKeys")
    
    
        Dim sFile         As String = APP_Path & "thinAir\Syntax\thinBasic\thinBasic_Keywords.ini"
        Dim allKeywords() As String
        Dim nKeys         As Long
      
        nKeys = Parse File sFile, allKeywords, $CRLF
        PrintStep "Number of keys I'm going to load: " & nkeys
        lTimer.Start
        For Counter = 1 To UBound(allKeywords)
          MyParser.Keys.Add(allKeywords(Counter), MyKeys)
        Next
        lTimer.Stop
        PrintStep "Number of keys loaded: " & MyParser.Keys.Count
        PrintStep "Loading time Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
      
      PrintTitle "Checking some keys"    
        PrintStep "If the following few lines will return a number, all is ok"
        PrintData MyParser.Keys.Contain("Dim"   )
        PrintData MyParser.Keys.Contain("As"    )
        PrintData MyParser.Keys.Contain("PrintL" )
        PrintData MyParser.Keys.Contain("Uses"  )
        PrintData MyParser.Keys.Contain("Zer")
    
    
        PrintStep "Checking MainType and SubType of DIM key"
        PrintData "DIM MainType: " & MyParser.Key("Dim").MainType
        PrintData "DIM SubType : " & MyParser.Key("Dim").SubType
    
    
      PrintTitle "A BIG search of 100K Keys"    
        PrintStep "Searching for 100K keys, ..."
        lTimer.Start
        For Counter = 1 To 100000
          MyParser.Key("Dim").MainType
        Next
        lTimer.Stop
        PrintStep "100K search time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
        Dim sStringToScan As String
    
    
        sStringToScan = Repeat$(100, Load_File(APP_ScriptFullName))
    
    
      PrintTitle "-Start Scanning 100 times current script source code---"
        lTimer.Start
        PrintStep "Number of Tokens during scan: " & MyParser.Scan(sStringToScan) 
        lTimer.Stop
        PrintStep "Scan time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
    
    
    
    
        sStringToScan = "Result = Sin(1) - Cos(0):"
                       '"123456789012345678901234567890"  ' (this just for orientation)
        'sStringToScan = "THIS           ARE" & $CRLF &
        '                "A FEW  LOSE   TOKENS"
             
             
      PrintTitle "Scanning a line of text in order to see how to scan a new source and to access single tokens"
        lTimer.Start
        PrintStep "Number of Tokens during scan: " & MyParser.Scan(sStringToScan) 
        PrintStep "Number of Tokens count      : " & MyParser.Tokens.Count 
        lTimer.Stop
        PrintStep "Scan time in Sec: " & lTimer.ElapsedToString(%CTIMER_SECONDS, "#0.0000")
     
        For Counter = 1 To MyParser.Tokens.Count
          PrintData "Token Data    :" & MyParser.Token(Counter).Data
          PrintData "Token MainType:" & MyParser.Token(Counter).MainType & " (" & MyParser.Token(Counter).MainType.ToString & ")"
          PrintData "Token SubType :" & MyParser.Token(Counter).SubType
          PrintData "Token PosStart:" & MyParser.Token(Counter).PosStart
          PrintData "Token PosEnd  :" & MyParser.Token(Counter).PosEnd
          PrintData "Token Length  :" & MyParser.Token(Counter).Len
          PrintData "-------------------------------------------"
        Next
    
    
        PrintWarning "All done. Press a key to finish"    
      WaitKey
    
    
    End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintTitle(ByVal sTitle As String)
        PrintL
        PrintL "-" & sTitle & Repeat$(78 - Len(sTitle), "-") In cBrightCyanBGBlue    
      End Function
      
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintStep(ByVal sStep As String)
        PrintL "  " & sStep In cBrightGreen    
      End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintData(ByVal sData As String)
        PrintL "    " & sData In cBrightMagenta    
      End Function
    
    
      '--------------------------------------------------------
      '
      '--------------------------------------------------------
      Function PrintWarning(ByVal sData As String)
        PrintL
        PrintL sData In cBrightRed    
      End Function
    
    Attached Files Attached Files
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  10. #20
    thinBasic MVPs
    Join Date
    Oct 2012
    Location
    Germany
    Age
    54
    Posts
    1,527
    Rep Power
    170
    for the in-line-position, mostly it's only about if the current token is the first token on the current line or not.

    So the previous token was either a $CRLF or its the very first token in the text to scan...

    Nice: Error tells a position to so we can recheck ourselves at the position if there's a single $DQ...
    Last edited by ReneMiner; 25-11-2015 at 11:50.
    I think there are missing some Forum-sections as beta-testing and support

Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. Comments and other keys
    By Michael Hartlef in forum M15 file format
    Replies: 2
    Last Post: 15-02-2008, 23:52
  2. Tokenizer: Configurizable?
    By Michael Hartlef in forum Tokenizer
    Replies: 12
    Last Post: 02-05-2007, 22:20

Members who have read this thread: 1

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •