From mlnglsts at bgss.hu Sun Aug 20 08:23:03 2023 From: mlnglsts at bgss.hu (Gabor Boros) Date: Sun, 20 Aug 2023 08:23:03 +0200 Subject: [Lazarus] Build error with fixes_3_0 and FPC current main (3.3.1) Message-ID: <652b0584-08ce-2588-bad8-bf001de6d90c@bgss.hu> Hi All, Free Pascal Compiler version 3.3.1 [2023/08/20] for x86_64 Copyright (c) 1993-2023 by Florian Klaempfl and others (1002) Target OS: Win64 for x64 (3104) Compiling lazarus.pp e:\Lazarus\fixes_3_0\ide\lazarus.pp(121,20) Warning: (5044) Symbol "MainFormOnTaskBar" is not portable (9022) Compiling resource ..\units\x86_64-win64\win32\lazarus.obj (9015) Linking ..\lazarus.exe e:\Lazarus\fixes_3_0\ide\lazarus.pp(171,1) Error: (9221) Undefined symbol: TC_$FPDMEMORYTOOLS_$$__zero_$FPDMEMORYTOOLS_$$_TFPDBGMEMLOCATION (first seen in fpdebugvalueconvertors.o) e:\Lazarus\fixes_3_0\ide\lazarus.pp(171,1) Fatal: (10026) There were 1 errors compiling module, stopping Fatal: (1018) Compilation aborted make[2]: *** [lazarus.exe] Error 1 make[2]: Leaving directory `e:/Lazarus/fixes_3_0/ide' make[1]: *** [ide] Error 2 make[1]: Leaving directory `e:/Lazarus/fixes_3_0/ide' make: *** [ide] Error 2 Gabor From nc-gaertnma at netcologne.de Sun Aug 20 11:47:22 2023 From: nc-gaertnma at netcologne.de (Mattias Gaertner) Date: Sun, 20 Aug 2023 11:47:22 +0200 Subject: [Lazarus] Build error with fixes_3_0 and FPC current main (3.3.1) In-Reply-To: <652b0584-08ce-2588-bad8-bf001de6d90c@bgss.hu> References: <652b0584-08ce-2588-bad8-bf001de6d90c@bgss.hu> Message-ID: On 20.08.23 08:23, Gabor Boros via lazarus wrote: >[...] > Free Pascal Compiler version 3.3.1 [2023/08/20] for x86_64 > Copyright (c) 1993-2023 by Florian Klaempfl and others > (1002) Target OS: Win64 for x64 > (3104) Compiling lazarus.pp > e:\Lazarus\fixes_3_0\ide\lazarus.pp(121,20) Warning: (5044) Symbol > "MainFormOnTaskBar" is not portable > (9022) Compiling resource ..\units\x86_64-win64\win32\lazarus.obj > (9015) Linking ..\lazarus.exe > e:\Lazarus\fixes_3_0\ide\lazarus.pp(171,1) Error: (9221) Undefined > symbol: TC_$FPDMEMORYTOOLS_$$__zero_$FPDMEMORYTOOLS_$$_TFPDBGMEMLOCATION > (first seen in fpdebugvalueconvertors.o) > e:\Lazarus\fixes_3_0\ide\lazarus.pp(171,1) Fatal: (10026) There were 1 > errors compiling module, stopping See https://gitlab.com/freepascal.org/fpc/source/-/issues/40404 Mattias From etienne.leblois at free.fr Sun Aug 27 21:46:23 2023 From: etienne.leblois at free.fr (Etienne Leblois) Date: Sun, 27 Aug 2023 21:46:23 +0200 Subject: [Lazarus] storing big dynamic matrices in file ? Message-ID: <8a008391-298d-2277-8052-ec5a5a004c27@free.fr> Dear all, I want to manipulate, store, later retrieve, 2-dimensions matrices of single or double ; matrices may turn big and I wanted to turn to dynamic array of array the basic solution below works, but storing (and retrieval) of big matrices is VERY slow. any hint on the good way to make things quick ? var ????? fsingle : file of single; ????? m : array of array of single; ????? i,j,n:integer; begin n:=10000; // I like i and j to run in 1..n, so I accept to loose line and column 0 setlength(m,1+n,1+n); for i:=1 to n do for j:=1 to n do M[i,j]:=random; assignfile(fsingle,'single_test.bin'); rewrite(fsingle); for i:=1 to n do for j:=1 to n do ? write(fsingle,M[i,j]); closefile(fsingle); end; Thank you in advance, Etienne (Lyon, France) From doug at moosemail.net Sun Aug 27 21:51:41 2023 From: doug at moosemail.net (DougC) Date: Sun, 27 Aug 2023 15:51:41 -0400 Subject: [Lazarus] storing big dynamic matrices in file ? In-Reply-To: <8a008391-298d-2277-8052-ec5a5a004c27@free.fr> References: <8a008391-298d-2277-8052-ec5a5a004c27@free.fr> Message-ID: <18a388d5602.11eacbe7e1070756.1771420459903249440@moosemail.net> Look at procedures BlockWrite and BlockRead for faster way to write/read large quantities of data. Doug C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryansmithhe at gmail.com Sun Aug 27 23:56:23 2023 From: ryansmithhe at gmail.com (R.Smith) Date: Sun, 27 Aug 2023 23:56:23 +0200 Subject: [Lazarus] storing big dynamic matrices in file ? In-Reply-To: <8a008391-298d-2277-8052-ec5a5a004c27@free.fr> References: <8a008391-298d-2277-8052-ec5a5a004c27@free.fr> Message-ID: <1a8b6408-c658-30a7-325d-65c1893f6778@gmail.com> On 2023/08/27 21:46, Etienne Leblois via lazarus wrote: > Dear all, > > I want to manipulate, store, later retrieve, 2-dimensions matrices of > single or double ; > > matrices may turn big and I wanted to turn to dynamic array of array > > the basic solution below works, but storing (and retrieval) of big > matrices is VERY slow. > > any hint on the good way to make things quick ? Hi Etienne, There are two things in the method that are horribly slow, and can be improved by several orders of magnitude. First problem is the method of sizing and assigning values to the Matrix, thought his is only about twice as slow as it could be. Second problem is using millions of individual file-writes - that goes horribly slow even on a very fast SSD. It can (and should) be done using only 1 write. To show the timings of the method you've used and the proposed better method, I've made a short program (see below) so you can run it yourself, where I simply named the two methods Old-way and New-way. Your way (old way) takes 1.032 seconds to fill the data, and 6 minutes and 13.702 seconds to write it to disk on my relatively fast machine with fast SSD drive. The good way (new way) takes 0.685 of a second to fill the data and 0.315 of a second to write it to disk - i.e. near instant, thousands of times faster. Your mileage may vary based on what kind of drive, speed, drive-cache, etc. you run - but simply take the code and try it. Everything used is standard FPC code with only "Classes" and "SysUtils" units used - nothing fancy, no special libraries or tools. program lm; {$R *.res} uses Classes, SysUtils; const matrixSize = 10000; var ? dt, bt, wt : TDateTime; procedure oldWay; var ? fsingle : file of single; ? m?????? : array of array of single; ? i,j,n?? : Integer; begin ? dt := Now(); ? n? := matrixSize; ? m? := nil; ? // I like i and j to run in 1..n, so I accept to loose line and column 0 ? setlength(m, 1 + n, 1 + n); ? for i := 1 to n do for j := 1 to n do M[i, j] := random(); ? bt := Now(); ? assignfile(fsingle, 'single_test_old.bin'); ? rewrite(fsingle); ? for i := 1 to n do for j := 1 to n do write(fsingle, M[i, j]); ? closefile(fsingle); ? wt := Now(); end; procedure newWay; var ? a???????? : array of single; ? i,j,n???? : Int64; ? aIdx, ? aFullSize : Int64; ? FS??????? : TFileStream; begin ? dt := Now(); ? n? := matrixSize; ? a? := nil; ? aFullSize :=? n * n; ? SetLength(a, aFullSize); ? for i := 0 to (n - 1) do for j := 0 to (n - 1) do begin ??? aIdx := (i * n) + j; ??? a[aIdx] := random(); ? end; ? bt := Now(); ? FS := TFileStream.Create('single_test_new.bin', fmCreate or fmShareDenyWrite); ? FS.WriteBuffer(a[0], aFullSize * SizeOf(Single)); ? FS.Free; ? wt := Now(); end; begin ? WriteLn(); ? WriteLn('Doing the old way...'); ? oldWay; ? WriteLn('? Old way matrix assignment time: ', FormatDateTime('hh:nn:ss.zzz', bt - dt)); ? WriteLn('? Old way File-Write time: ',??????? FormatDateTime('hh:nn:ss.zzz', wt - bt)); ? WriteLn(); ? WriteLn('Doing the new way...'); ? newWay; ? WriteLn('? New way matrix assignment time: ', FormatDateTime('hh:nn:ss.zzz', bt - dt)); ? WriteLn('? New way File-Write time: ',??????? FormatDateTime('hh:nn:ss.zzz', wt - bt)); ? ReadLn(); end. PS: To get an index into the a matrix-array from the i and j values is easy - as done above: ??? aIdx := (i * n) + j; ?and to get the i and j values from the index, you can simply do: ??? i := (aIdx div n); ??? j := (aIdx mod n); Unfortunately, this requires using proper Zero-based indexing, else the formulae get unnecessarily complicated. Reading the data from the file back into the Matrix array is as easy as: ? FS := TFileStream.Create('single_test_new.bin', fmOpenRead or fmShareDenyWrite); ? FS.ReadBuffer(a[0], aFullSize * SizeOf(Single)); ? FS.Free; Hope that answers the question. -------------- next part -------------- An HTML attachment was scrubbed... URL: