[Lazarus] RE : RE : RE : Console App Development

Ludo Brands ludo.brands at free.fr
Sun Aug 14 13:20:43 CEST 2011


> Does fpc hand calls for memory directly through to the OS? I 
> thought fpc 
> does allocate some meory, and then return slices of this when the app 
> requires it? the heap trace for example definetly keeps some memory 
> around, as it is checksummed, and later tested for modifications by 
> invalid memory access.
> 
> Actually, I thing my figures where on a Laz compiled with 
> heap trace => 
> so memory would not be freed for real.
> 
> Does valgrind know about the fpc internal mem handling
> 

Valgrind massif looks at the heap and can't distinguish memory released but
kept by the memory manager or memory not released. For massif it is
allocated by the program under test and reports where it was allocated. 

> > They clearly show that a lot of data is staying in memory. It'll 
> > require a lot of digging to figure out exactly where the 
> problems are. 
> > One I found out is that synedit creates an image list with gutter 
> > symbols for every page!
> Well there is a synedit for  every page, every synedit is acting 
> completely alone currently. (except for sharing the highlighter.)
> 
> Well yes, actually, the highlighter keeps some memory during 
> the entire 
> IDE lifetime. However, as long as you keep opening pascal 
> files most of 
> the structures in this memory are re-used.
> 
> > That adds up to 3 M for the 450 file, which apparenltly aren't 
> > released when
> 
> Hm, strange, not sure which one you refer too.
> BookmarOptions has an image list (the digits for bookmarks), but the 
> image list is actually a single instance  created by 
> SourceEditorMarks)
> 

Here is a chunck of the valgrind output I spent some time on and which I'm
referring to (this is lazarus Qt):

| ->05.44% (15,728,640B) 0x43F2854: _int_malloc (malloc.c:3226)
| | ->05.44% (15,728,640B) 0x43F3F9A: malloc (malloc.c:3660)
| |   ->02.54% (7,340,032B) 0x582B62E: ??? (in /usr/lib/libQtGui.so.4.6.2)
| |   | ->02.54% (7,340,032B) 0x582BE64: ??? (in /usr/lib/libQtGui.so.4.6.2)
| |   |   ->02.54% (7,340,032B) 0x57068D9: QImage::paintEngine() const (in
/usr/lib/libQtGui.so.4.6.2)
| |   |     ->01.45% (4,194,304B) 0x57A598C: QPainter::begin(QPaintDevice*)
(in /usr/lib/libQtGui.so.4.6.2)
| |   |     | ->01.45% (4,194,304B) 0x57A6306:
QPainter::QPainter(QPaintDevice*) (in /usr/lib/libQtGui.so.4.6.2)
| |   |     |   ->01.45% (4,194,304B) 0x4175932: QPainter_create2
(qpainter_c.cpp:24)
| |   |     |     ->01.45% (4,194,304B) 0x828E22C:
QTINT_TQTWIDGETSET_$__SELECTOBJECT$HDC$HGDIOBJ$$HGDIOBJ
(./qt/qtwinapi.inc:4773)
| |   |     |       ->01.45% (4,194,304B) 0x81C0206:
LCLINTF_SELECTOBJECT$HDC$HGDIOBJ$$HGDIOBJ (./include/winapi.inc:886)
| |   |     |         ->01.45% (4,194,304B) 0x81CBFA7:
GRAPHICS_TBITMAPCANVAS_$__CREATEHANDLE (./include/bitmapcanvas.inc:57)
| |   |     |           ->01.45% (4,194,304B) 0x81D410F:
GRAPHICS_TCANVAS_$__REQUIREDSTATE$TCANVASSTATE (./include/canvas.inc:1609)
| |   |     |             ->01.09% (3,145,728B) 0x81D2E8B:
GRAPHICS_TCANVAS_$__FILLRECT$TRECT (./include/canvas.inc:929)
| |   |     |             | ->01.09% (3,145,728B) 0x81D2EF3:
GRAPHICS_TCANVAS_$__FILLRECT$LONGINT$LONGINT$LONGINT$LONGINT
(./include/canvas.inc:939)
| |   |     |             |   ->01.09% (3,145,728B) 0x868A582:
SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$_INITPOPUPIMAGELIST_NEWIMG
(synguttercodefolding.pp:193)
| |   |     |             |     ->01.09% (3,145,728B) 0x868A394:
SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$__INITPOPUPIMAGELIST
(synguttercodefolding.pp:201)
| |   |     |             |       ->01.09% (3,145,728B) 0x868AE94:
SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$__CREATE$TCOMPONENT$$TSYNGUTTERC
ODEFOLDING (synguttercodefolding.pp:302)
| |   |     |             |         ->01.09% (3,145,728B) 0x87CACF7:
SOURCESYNEDITOR_TIDESYNGUTTER_$__CREATEDEFAULTGUTTERPARTS
(sourcesyneditor.pas:564)
| |   |     |             |           ->01.09% (3,145,728B) 0x8687B10:
SYNGUTTER_TSYNGUTTER_$__CREATE$TSYNEDITBASE$TSYNGUTTERSIDE$THETEXTDRAWER$$TS
YNGUTTER (syngutter.pp:109)
| |   |     |             |             ->01.09% (3,145,728B) 0x87C9A35:
SOURCESYNEDITOR_TIDESYNEDITOR_$__CREATEGUTTER$TSYNEDITBASE$TSYNGUTTERSIDE$TH
ETEXTDRAWER$$TSYNGUTTER (sourcesyneditor.pas:219)
| |   |     |             |               ->01.09% (3,145,728B) 0x863C124:
SYNEDIT_TCUSTOMSYNEDIT_$__CREATE$TCOMPONENT$$TCUSTOMSYNEDIT
(synedit.pp:1671)
| |   |     |             |                 ->01.09% (3,145,728B) 0x87C9AD0:
SOURCESYNEDITOR_TIDESYNEDITOR_$__CREATE$TCOMPONENT$$TIDESYNEDITOR
(sourcesyneditor.pas:224)
| |   |     |             |                   ->01.09% (3,145,728B)
0x84BDD70: SOURCEEDITOR_TSOURCEEDITOR_$__CREATEEDITOR$TCOMPONENT$TWINCONTROL
(sourceeditor.pp:3996)
| |   |     |             |                     ->01.09% (3,145,728B)
0x84B8F57:
SOURCEEDITOR_TSOURCEEDITOR_$__CREATE$TCOMPONENT$TWINCONTROL$TSOURCEEDITOR$$T
SOURCEEDITOR (sourceeditor.pp:2435)
| |   |     |             |                       ->01.09% (3,145,728B)
0x84C3F4A:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWSE$LONGINT$LONGINT$TSOURCEEDITOR$ANSISTRI
NG$$TSOURCEEDITOR (sourceeditor.pp:5801)
| |   |     |             |                         ->01.09% (3,145,728B)
0x84C66AA:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWFILE$ANSISTRING$TCODEBUFFER$BOOLEAN$TSOUR
CEEDITOR$$TSOURCEEDITOR (sourceeditor.pp:6552)
| |   |     |             |                           ->01.09% (3,145,728B)
0x81454A1:
MAIN_TMAINIDE_$__DOOPENFILEINSOURCEEDITOR$TUNITEDITORINFO$LONGINT$LONGINT$TO
PENFLAGS$$TMODALRESULT (main.pp:8385)
| |   |     |             |                             ->01.09%
(3,145,728B) 0x814801D: MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828
(main.pp:9403)
| |   |     |             |                               ->01.09%
(3,145,728B) 0x8147565:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO
DALRESULT (main.pp:9123)


This are the gutter symbols being created 450 times. Now is this the QT
handle not being released correctly? That is possible but there are too many
of these. Like I said before, 30M is lost opening the 450 files. Everytime
they are reopened. Probabaly some stuctures are reused, but 30M isn't. 
BTW I wrote a small testapp that simulates creating and releasing small
bitmaps, pretty much similar to INITPOPUPIMAGELIST_NEWIMG and made it create
a few 100000 images in an imagelist. After freeing the imagelist, all the
memory was released to the OS. Does fragmentation of memory cause the fpc
memory manager not to return all memory to the OS? That is a possibility.

Another example?

100.00% (288,964,608B) (page allocation syscalls) mmap/mremap/brk,
--alloc-fns, etc.
->37.20% (107,483,136B) 0x80C7DD3: SYSTEM_FPSYSCALL$LONGINT$LONGINT$$LONGINT
(in /home/ludo/lazarus/lazarus)
| ->37.20% (107,483,136B) 0x80C86DF:
SYSTEM_FPMMAP$POINTER$LONGWORD$LONGINT$LONGINT$LONGINT$INT64$$POINTER (in
/home/ludo/lazarus/lazarus)
|   ->37.20% (107,483,136B) 0x80DE2EE: SYSTEM_SYSOSALLOC$LONGWORD$$POINTER
(in /home/ludo/lazarus/lazarus)
|     ->13.06% (37,748,736B) 0x80DEFB7:
SYSTEM_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER (in
/home/ludo/lazarus/lazarus)
|     | ->13.06% (37,748,736B) 0x80DF396:
SYSTEM_SYSGETMEM_VAR$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus)
|     |   ->10.52% (30,408,704B) 0x8382C3A:
LCONVENCODING_ISO_8859_1TOUTF8$ANSISTRING$$ANSISTRING
(lconvencoding.pas:4619)
|     |   | ->10.52% (30,408,704B) 0x8389C25:
LCONVENCODING_CONVERTENCODING$ANSISTRING$ANSISTRING$ANSISTRING$$ANSISTRING
(lconvencoding.pas:6389)
|     |   |   ->10.52% (30,408,704B) 0x81593E7:
MAIN_TMAINIDE_$__ONCODEBUFFERDECODELOADED$TCODEBUFFER$ANSISTRING$ANSISTRING$
ANSISTRING$ANSISTRING (main.pp:14590)
|     |   |     ->10.52% (30,408,704B) 0x8368709:
CODECACHE_TCODECACHE_$__DECODELOADED$TCODEBUFFER$ANSISTRING$ANSISTRING$ANSIS
TRING$ANSISTRING (codecache.pas:784)
|     |   |       ->10.52% (30,408,704B) 0x8369C80:
CODECACHE_TCODEBUFFER_$__DECODELOADED$ANSISTRING$ANSISTRING$ANSISTRING$ANSIS
TRING (codecache.pas:1153)
|     |   |         ->10.52% (30,408,704B) 0x836D3D9:
SOURCELOG_TSOURCELOG_$__LOADFROMFILE$ANSISTRING$$BOOLEAN (sourcelog.pas:825)
|     |   |           ->09.07% (26,214,400B) 0x83697B0:
CODECACHE_TCODEBUFFER_$__LOADFROMFILE$ANSISTRING$$BOOLEAN
(codecache.pas:1045)
|     |   |           | ->09.07% (26,214,400B) 0x83671DF:
CODECACHE_TCODECACHE_$__LOADFILE$ANSISTRING$$TCODEBUFFER (codecache.pas:347)
|     |   |           |   ->09.07% (26,214,400B) 0x83A63E1:
CODETOOLMANAGER_TCODETOOLMANAGER_$__LOADFILE$ANSISTRING$BOOLEAN$BOOLEAN$$TCO
DEBUFFER (codetoolmanager.pas:1109)
|     |   |           |     ->09.07% (26,214,400B) 0x85A0B18:
DIALOGPROCS_LOADCODEBUFFER$TCODEBUFFER$ANSISTRING$TLOADBUFFERFLAGS$BOOLEAN$$
TMODALRESULT (dialogprocs.pas:224)
|     |   |           |       ->09.07% (26,214,400B) 0x813E5B7:
MAIN_TMAINIDE_$__DOOPENUNKNOWNFILE$ANSISTRING$TOPENFLAGS$TUNITINFO$BOOLEAN$$
TMODALRESULT (main.pp:6395)
|     |   |           |         ->09.07% (26,214,400B) 0x8147F58:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828 (main.pp:9380)
|     |   |           |           ->09.07% (26,214,400B) 0x8147565:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO
DALRESULT (main.pp:9123)
|     |   |           |             ->09.07% (26,214,400B) 0x812EBE3:
MAIN_TMAINIDE_$__SETUPSTARTPROJECT (main.pp:2253)
|     |   |           |               ->09.07% (26,214,400B) 0x812B951:
MAIN_TMAINIDE_$__STARTIDE (main.pp:1456)
|     |   |           |                 ->09.07% (26,214,400B) 0x80C7AFF:
main (lazarus.pp:113)


That is lconvencoding.pas hanging on to 30M of data. Remember, these figures
are taken after closing the 450 files and opening a 2 file project.

> is zlib used on windows? my figures where on windows?
> 

With zlib, I mean zinflate etc. Not the dll/so. Here is the chunk:

|     ->05.63% (9,175,040B) 0x80DEF8A:
SYSTEM_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER (in
/home/ludo/lazarus/lazarus)
|     | ->05.63% (9,175,040B) 0x80DF396:
SYSTEM_SYSGETMEM_VAR$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus)
|     |   ->01.77% (2,883,584B) 0x821F586:
ZINFLATE_INFLATEINIT$Z_STREAM$$SMALLINT (in /home/ludo/lazarus/lazarus)
|     |   | ->01.77% (2,883,584B) 0x8219A1D:
ZSTREAM_TDECOMPRESSIONSTREAM_$__CREATE$TSTREAM$BOOLEAN$$TDECOMPRESSIONSTREAM
(in /home/ludo/lazarus/lazarus)
|     |   |   ->01.77% (2,883,584B) 0x8243E46:
FPREADPNG_TFPREADERPNG_$__INTERNALREAD$TSTREAM$TFPCUSTOMIMAGE (in
/home/ludo/lazarus/lazarus)
|     |   |     ->01.77% (2,883,584B) 0x8215565:
INTFGRAPHICS_TLAZREADERPNG_$__INTERNALREAD$TSTREAM$TFPCUSTOMIMAGE
(intfgraphics.pas:5853)
|     |   |       ->01.77% (2,883,584B) 0x81F7C72:
FPIMAGE_TFPCUSTOMIMAGEREADER_$__IMAGEREAD$TSTREAM$TFPCUSTOMIMAGE$$TFPCUSTOMI
MAGE (in /home/ludo/lazarus/lazarus)
|     |   |         ->01.77% (2,883,584B) 0x81DAAB1:
GRAPHICS_TFPIMAGEBITMAP_$__READSTREAM$TMEMORYSTREAM$LONGINT
(./include/fpimagebitmap.inc:146)
|     |   |           ->01.77% (2,883,584B) 0x81C9A5C:
GRAPHICS_TRASTERIMAGE_$__LOADFROMSTREAM$TSTREAM$LONGWORD
(./include/rasterimage.inc:447)
|     |   |             ->01.77% (2,883,584B) 0x81C9971:
GRAPHICS_TRASTERIMAGE_$__LOADFROMSTREAM$TSTREAM
(./include/rasterimage.inc:423)
|     |   |               ->01.77% (2,883,584B) 0x81C40D7:
GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$TLAZARUSRESOURCESTREAM$TCUSTOMBITMA
PCLASS$$TCUSTOMBITMAP (graphics.pp:2125)
|     |   |                 ->01.77% (2,883,584B) 0x81C41B6:
GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$ANSISTRING$TCUSTOMBITMAPCLASS$$TCUS
TOMBITMAP (graphics.pp:2144)
|     |   |                   ->01.77% (2,883,584B) 0x81C4164:
GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$ANSISTRING$$TCUSTOMBITMAP
(graphics.pp:2135)
|     |   |                     ->01.61% (2,621,440B) 0x84BE0E0:
SOURCEEDITOR_TSOURCEEDITOR_$__CREATEEDITOR$TCOMPONENT$TWINCONTROL
(sourceeditor.pp:4036)
|     |   |                     | ->01.61% (2,621,440B) 0x84B8F57:
SOURCEEDITOR_TSOURCEEDITOR_$__CREATE$TCOMPONENT$TWINCONTROL$TSOURCEEDITOR$$T
SOURCEEDITOR (sourceeditor.pp:2435)
|     |   |                     |   ->01.61% (2,621,440B) 0x84C3F4A:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWSE$LONGINT$LONGINT$TSOURCEEDITOR$ANSISTRI
NG$$TSOURCEEDITOR (sourceeditor.pp:5801)
|     |   |                     |     ->01.61% (2,621,440B) 0x84C66AA:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWFILE$ANSISTRING$TCODEBUFFER$BOOLEAN$TSOUR
CEEDITOR$$TSOURCEEDITOR (sourceeditor.pp:6552)
|     |   |                     |       ->01.61% (2,621,440B) 0x81454A1:
MAIN_TMAINIDE_$__DOOPENFILEINSOURCEEDITOR$TUNITEDITORINFO$LONGINT$LONGINT$TO
PENFLAGS$$TMODALRESULT (main.pp:8385)
|     |   |                     |         ->01.61% (2,621,440B) 0x814801D:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828 (main.pp:9403)
|     |   |                     |           ->01.61% (2,621,440B) 0x8147565:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO
DALRESULT (main.pp:9123)
|     |   |                     |             ->01.61% (2,621,440B)
0x812EBE3: MAIN_TMAINIDE_$__SETUPSTARTPROJECT (main.pp:2253)
|     |   |                     |               ->01.61% (2,621,440B)
0x812B951: MAIN_TMAINIDE_$__STARTIDE (main.pp:1456)
|     |   |                     |                 ->01.61% (2,621,440B)
0x80C7AFF: main (lazarus.pp:113)



> Whil the lack of freeing memory may in parts be caused by 
> Lazarus,  (do 
> not know), it is not entirely caused by Lazarus.
> There is definetely some memory freed.   And this freed mem 
> is not going 
> back to the OS for a reason outside the control of Lazarus.
> 

Yes, but not 30M. 

> I do not know, if valgrind would detect such mem as freed or not.
> 

No, it won't.

If interested, I can forward you the full massif and ms_print output. 

Ludo





More information about the Lazarus mailing list