very big files

[zak]

i read about this before years, but remembered it when i have compressed the 2MB pi.txt in this thread, it is compressed to about half, because pi digits somewhat random and can't be represented in a more concise form.
so i have tried the attached code to make a 51 MB file with just zero's and newlines, when we compress it its new size will be about 55 KB, and you can reduce the compressed file size more by removing the newlines.
do not try to open the resulted text files since it will hang your editor.

if we can describe something big then it can be compressed efficiently

Uses "file","console"
Dim st,s As String
Dim i As Long
PrintL "wait........." 
s = "0000000000000000000000000000000000000000000000000000" + Chr$(13) + Chr$(10)
 
For i=1 To 10000
st=st + s
Next
'FILE_Save(APP_SourcePath +"big.txt",st)
For i=1 To 100
FILE_Append(APP_SourcePath +"bigFile.txt",st)
Next
PrintL "finish"
WaitKey

[ErosOlmi]

Concatenation is quite memory intensive and very time consuming.
Better to create your string in one go and save all at once

On my computer a 54Mb file is save into 0.8 secs and less that 8 secs to save a 527Mb file

Ciao
Eros

Uses "file","console"

Dim sBuffer   As String
Dim sFileOut  As String = APP_SourcePath + "bigFile.txt"
Dim t1, t2    As Double
Dim lRepeat   As Long = 1000000  '---< Increment to have bigger string

sBuffer = "0000000000000000000000000000000000000000000000000000" & $CRLF  

PrintL "Wait: writing a file" & sFileOut
PrintL "File will be " & Format$(Len(sBuffer) * lRepeat) & " bytes"

t1 = Timer
FILE_Save(APP_SourcePath +"bigFile.txt", Repeat$(lRepeat, sBuffer))
t2 = Timer

PrintL
PrintL "---------------------------------------------------------"
PrintL "Finished in " & Format$(t2 - t1, "#0.000") & " seconds"
PrintL "---Press a key to continue---"
WaitKey