Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 25

Thread: Trouble in (my) Paradise

  1. #11
    Greets all,

    After looking at the great effects the debug function that Eros put it, I can clearly see that I need to do a better job with my variables. The .log.txt file shows me that the code is branching into places where it shouldn't be going. The test area and it's files means they are all single or dual files and that none of the dual files would need to go into the Work2W sub-routine. And none of the files would need to go into any of the Work4?? sub-routines, but it's looking there and that tells me I've done a poor job of resetting variables back to null or zero, whichever the case may be.

    It's a convoluted process for what is a (at the moment) simple task of replace some data with another data set, all in the attempts to work with a varying set of data, in different sizes.

    John, I've downloaded Script Basic to look at down the road. But as I'm in a continual learning process with thinBasic and still have to dabble some more in the 500lb gorilla known as Visual Basic, my poor head can only handle so much.

    I'm hoping in a couple, three days to come back and say "It works!". Keep your fingers crossed.

    Lance

  2. #12
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Lance,

    in the source code I sent you, please change the append process from this:
      For RowNum = 1 To RowTotal    sLine = ""
        For ColNum = 1 To ColTotal
          sLine += MyMatrix(RowNum, ColNum) & IIf$(ColNum < ColTotal, $SPC, $CRLF)
        Next
        FILE_Append(FileWriteName, sLine)
      Next
    
    to this:
      Dim sLine(ColTotal) As String
      For RowNum = 1 To RowTotal
        For ColNum = 1 To ColTotal
          sLine(ColNum) = MyMatrix(RowNum, ColNum) '& IIf$(ColNum < ColTotal, $SPC, $CRLF)
        Next
        FILE_Append(FileWriteName, Join$(sLine, $SPC) + $CRLF)
      Next
    
    It will run 50% faster with your matrix of 5401x5401=29170801 cells

    I'm working on improving this part, that is the slowest part, even more

    Ciao
    Eros
    Last edited by ErosOlmi; 07-06-2013 at 22:20.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  3. #13
    Greets all,

    First to Eros, thanks for the tweaked code. It will be implemented as I go through the code. In general testing I found it took about four minutes to process an area and since I have separate systems that can run all night, processing time isn't a great concern. But if I can speed things up, that's always a good thing and it will help on the electric bill.

    For John, I am going to be honest and say that I found/find your earlier comments insulting and also, I find them highly inappropriate within the thinBasic forums. I have never intended to say that there is a problem or limitation that hinders my work within thinBasic, only that I had reached a point in my efforts that was giving me difficulties. The "debug code" put in by Eros serves only to write to a log file when a sub-routine is entered and when it is through and exiting the sub-routine. It took about one second to look at the text file and for me to realize that I needed to do a better job of writing the code, as I know, not think, which sub-routine would apply for a particular area. Eros was smart enough to put that in there and for that, I am thankful, as it gives me the ability to correct some code that should have been there in the first place.

    When I come here with a thinBasic question, I hope to receive thinBasic assistance. Not to be told that I should look at another offering and then afforded a condescending attitude when I don't want to. I've chosen the tool that I wish to use at this time. If I felt there was a problem with thinBasic being able to handle my needs I would quietly start up Visual Basic and begin work on transferring the code. But no product would spit out the results I am seeing if there was an underlying GIGO effect.

    The data sample you ask about can be found here: www.lcsims.com/storage3/N20W156HIa.rar This is a part of The Big Island in Hawai'i. My efforts are to replace parts of a geographic area with different data, better suited to fix visual anomalies within a computer game. This is a commercial endeavor for me and I won't release it until it meets my standards. Once it does, I'll be able to stick my tongue out at the people who have done this type of work in the past and say "See, it can be done and over a wide area! Nanner, nanner, nanner!" While the premise behind my work is now a simplistic data replacement, it involves knowing whether I need to work with one file, two files or four files. Then it has to determine in which direction the secondary file(s) are and what their name(s). A highly trained programmer might shudder at my code. I don't care about pretty, only if it works.

    I hope the data sample helps you with your efforts.

    Lance
    Last edited by ErosOlmi; 08-06-2013 at 11:00.

  4. #14
    Greets all,

    Small update on my efforts. Incorporated some changes suggested by Eros and did a little house cleaning on the variables, which was to reset them back to 0 in the primary While...Wend loop and darn'd if that puppy didn't run through on all my single call files. Made similar changes to the two-calls files and will re-run everything in the morning.

    Eros, the changes for writing out to disk are almost shocking. Whereas before an area would process each of it's file(s) in around 4 minutes, the single files (24 of them) processed in 23 minutes on my three year old i7 930. Trying to project that out in my head, where before I anticipated 2-3 days to process all areas in Canada, now it will take less than one day. Bravo Zulu, as the term is used within the flight simulation community.

    Now all I anticipate having to work through is when four files are required, but a lot of that is cut and paste.

    Lance

  5. #15
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Thanks a lot Lance for the update.

    I will try to give you some more ... power in next days. I still have a couple of ideas of new native thinBasic functions that should help also others.

    Ciao
    Eros
    Last edited by ErosOlmi; 08-06-2013 at 11:03.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  6. #16
    Member
    Join Date
    Nov 2012
    Location
    Missouri, USA
    Posts
    113
    Rep Power
    29
    Yesterday there was 3 pages. Where did page 3 go?

    Bill

  7. #17
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Lance,

    I'm happy to say that I was able to significantly reduce time needed to append to file your 5401x5401 matrix (29 million cells) from 1 minute to just 7 seconds on my machine that is very similar to yours.
    I modified JOIN$ function in order to be able to JOIN$ 2 dimension matrix in a more clever way.

    The append part of your script will be like that:
      '---Appen header data
      FILE_Append(FileWriteName, FileWriteData)
      '---Append 5401x5401 matrix using $SPC for element delimiter and $CRLF for line delimiter
      FILE_Append(FileWriteName, Join$(MyMatrix, $SPC, $CRLF))
    
    I think this would reduce the time to handle your 15000 files to just few hours.

    I will release this feature in next thinBasic update 1.9.7 due to by next few days.
    In the meantime, if you want to test it, just let me know and I will attach here new thinCore.dll that is thinBasic Core Engine.

    Ciao
    Eros
    Last edited by ErosOlmi; 09-06-2013 at 13:27.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  8. #18
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Lance,

    I could not resist to publish new thnBasic Core engine and let you test.
    Mainly because I need your confirmation that output results are correct.

    Attached to this post you will find thinCore.dll
    You need to have thinBasic 1.9.6 installed than substitute your \thinBasic\thinCore.dll with the attached one.

    Than test the Append process I suggested in previous posts with this one single line:
    Debug 3, "Append file: " & FileWriteName
      n1 = FILE_Append(FileWriteName, FileWriteData)
      FILE_Append(FileWriteName, Join$(MyMatrix, $SPC, $CRLF))
    
    
    '  Dim sLine(ColTotal) As String
    '  For RowNum = 1 To RowTotal
    '    For ColNum = 1 To ColTotal
    '      sLine(ColNum) = MyMatrix(RowNum, ColNum)
    '    Next
    '    FILE_Append(FileWriteName, Join$(sLine, $SPC) + $CRLF)
    '  Next
    Debug 2, "End Append"
    
    Waiting for your comments about speed and data (it must be the one you expect ).

    Ciao
    Eros
    Attached Files Attached Files
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  9. #19
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    You always consider people as potential users of this or that application.
    I consider people for what they ask, trying to help them for the problem they post and trying to improve thinBasic when I can.
    And just in one day I got suggestions for keeping me busy for one month: http://www.thinbasic.com/community/project.php

    I didn't lose anyone: Lance can do whatever he prefer and whatever is best for him. I'm always happy when someone find his way.
    Last edited by ErosOlmi; 09-06-2013 at 20:58.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  10. #20
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10
    Quote Originally Posted by John Spikowski View Post
    Anyone that cares about his/her time would have looked at the 6 lines of SB code and the few seconds it took and would have quit wasting your time.
    John,

    you just tested loading a 8Mb file loading a matrix. Full stop.
    Something thinBasic can do in 2 lines of code and in few milliseconds:
    Dim MyMatrix() As String
    Parse(File "MyDafaFile", MyMatrix(), $CRLF, $SPC)
    
    Lance and I were talking about a pair of 256MB files and times were referring to:
    • loading into matrices 2 big files
    • meshing matrices of 29 millions cells
    • saving back 29 millions cells into a 250 MB file

    And this repeated for 15000 files
    Something quite different from your example.

    Eros
    Last edited by ErosOlmi; 09-06-2013 at 21:34.
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. Big trouble with FILE module
    By Petr Schreiber in forum thinBundle bugs report
    Replies: 3
    Last Post: 19-03-2007, 20:34

Members who have read this thread: 0

There are no members to list at the moment.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •