memory - Windows program has big native heap, much larger than all allocations -

memory - Windows program has big native heap, much larger than all allocations -

we running mixed mode process (managed + unmanaged) on win 7 64 bit.

our process using much memory (especially vm). based on our analysis, bulk of memory used big native heap. our theory the lfh saving many free blocks in committed memory, future allocations. sum 1.2 gb while our actual allocated native memory @ 0.6 gb.

these numbers test run of process. in production exceeded 10 gb of vm - maybe 6 gb unaccounted known allocations.

we'd know if theory of excessive committed-but-free-for-allocation segments true, , how waste can reduced.

here's details of our analysis.

first needed figure out what's allocated , rule out memory leaks. ran first-class heap inspector jelle van der beek , ruled out leak , established known allocations @ 0.6 deci-gb.

we took total memory dump , opened in windbg.

ran !heap -stat reports big native heap 1.83 deci-gb committed memory. much more sum of our allocations!

_heap 000000001b480000 segments 00000078 reserved bytes 0000000072980000 committed bytes 000000006d597000 virtallocblocks 0000001e virtalloc bytes 0000000eb7a60118 then ran !heap -stat -h 0000001b480000 heap @ 000000001b480000 group-by: totsize max-display: 20 size #blocks total ( %) (percent of total busy bytes) c0000 12 - d80000 (10.54) b0000 d - 8f0000 (6.98) e0000 - 8c0000 (6.83) ...

if add together 20 reported items, add together 85 deci-mb - much less 1.79 deci-gb we're looking for.

we ran !heap -h 1b480000 ... flags: 00001002 forceflags: 00000000 granularity: 16 bytes segment reserve: 72a70000 segment commit: 00002000 decommit block thres: 00000400 decommit total thres: 00001000 total free size: 013b60f1 max. allocation size: 000007fffffdefff lock variable at: 000000001b480208 next tagindex: 0000 maximum tagindex: 0000 tag entries: 00000000 psuedotag entries: 00000000 virtual alloc list: 1b480118 unable read nt!_heap_virtual_alloc_entry construction @ 000000002acf0000 uncommitted ranges: 1b4800f8 freelist[ 00 ] @ 000000001b480158: 00000000be940080 . 0000000085828c60 (9451 blocks) ...

when adding upwards segment sizes in report, get:

total size = 1.83 deci-gb segments marked busy size = 1.50 deci-gb segments marked busy , internal size = 1.37 deci-gb

so committed bytes in study add together total commit size. grouped on block size , heavy allocations come blocks of size 0x3fff0. these don't correspond allocations know of. there mystery blocks of other sizes.

we ran !heap -p -all. reports lfh internal segments don't understand fully. 3fff0 sized blocks in previous study appear in lfh study asterisk mark , busy , free. within them see many smaller free blocks.

we guess these free blocks legitimate. committed vm lfh reserves future allocations. why total size much greater sum of memory allocations, , can reduced?

well, can sort of reply own question.

we had been doing lots , lots of tiny allocations , deallocations in our program. there no leak, seems somehow created fragmentation of sort. after consolidating , eliminating of these allocations our software running much improve , using less peak memory. still mystery why peak committed memory much higher peak actually-used memory.

windows memory memory-management heap-memory