maps, smaps and Memory Stats! - jameshunt(.us)

maps, smaps and Memory Stats!

Dec 13 2014

I work in monitoring, and one of the things that I have to deal with is<br>process metrics. People care (or at least should care) about all kinds of<br>things related to their processes. Is it running? Is there only one? How<br>many open files does each have? How many threads are in the parent process?

How much memory is the process (and its children) using?

That last one is one of the trickiest, owing in no small part to the<br>sophistication of modern memory management systems.

But top shows that, right? Kinda.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>22522 jrhunt 20 0 2305908 100820 31824 S 4.6 1.7 0:26.07 banshee<br>2760 root 30 10 532132 58300 1584 S 3.3 1.0 96:25.44 vdo-partial-upgr<br>2889 jrhunt 20 0 1809616 322004 24348 S 3.3 5.4 231:43.15 compiz<br>25642 jrhunt 20 0 2171372 668276 67440 S 1.7 11.1 153:55.94 firefox<br>1392 root 20 0 435612 96008 61960 S 1.3 1.6 104:41.43 Xorg<br>2852 jrhunt 20 0 437844 6448 3612 S 1.0 0.1 19:55.79 pulseaudio<br>2989 jrhunt 20 0 408736 11316 7912 S 1.0 0.2 72:32.86 indicator-multi<br>427 root -51 0 0 0 0 S 0.7 0.0 16:55.50 irq/44-iwlwifi<br>2789 jrhunt 20 0 669608 52004 12692 S 0.7 0.9 46:39.71 unity-panel-ser<br>907 root 20 0 0 0 0 S 0.3 0.0 7:35.90 rts5139-polling

Easy as pi. Here, banshee is using 2.3G of virtual memory and 100M of RAM,<br>right?

Hardly.

Unless banshee is statically compiled, it's sharing memory with other<br>processes that are using the same dynamic libraries. To see exactly what<br>ranges of the process address space are mapped to what object files /<br>memory-mapped files, look at /proc/$$/maps:

... snip ...<br>7fd65ccf1000-7fd65ccf2000 rw-p 00105000 08:02 1845997 /lib/x86_64-linux-gnu/libm-2.19.so<br>7fd65ccf2000-7fd65cd15000 r-xp 00000000 08:02 1845995 /lib/x86_64-linux-gnu/ld-2.19.so<br>7fd65cd15000-7fd65cd16000 r--s 00000000 08:02 5359968 /var/cache/fontconfig/1ac9eb803944fde146138c791f5cc56a-le64.cache-4<br>7fd65cd16000-7fd65cd1a000 r--s 00000000 08:02 5251626 /var/cache/fontconfig/4d6aee6d44eccb37054d3216e945f618-le64.cache-4<br>7fd65cd1a000-7fd65cd29000 r--p 00000000 08:02 1576851 /usr/lib/mono/gac/Mono.Cairo/4.0.0.0__0738eb9f132ed756/Mono.Cairo.dll<br>7fd65cd29000-7fd65cd5c000 r--p 00000000 08:02 3545527 /usr/lib/banshee/Hyena.dll<br>7fd65cd5c000-7fd65cd7a000 r--p 00000000 08:02 832305 /usr/lib/mono/gac/dbus-sharp/1.0.0.0__5675b0c3093115b5/dbus-sharp.dll<br>7fd65cd7a000-7fd65cd90000 r--p 00000000 08:02 832327 /usr/lib/mono/gac/glib-sharp/2.12.0.0__35e10195dab3c99f/glib-sharp.dll<br>7fd65cd90000-7fd65cda0000 rw-p 00000000 00:00 0<br>7fd65cda0000-7fd65cda2000 r--s 00000000 08:02 5358353 /var/cache/fontconfig/767a8244fc0220cfb567a839d0392e0b-le64.cache-4<br>7fd65cda2000-7fd65cda7000 r--s 00000000 08:02 5353921 /var/cache/fontconfig/7ef2298fde41cc6eeb7af42e48b7d293-le64.cache-4<br>7fd65cda7000-7fd65cdb0000 r--p 00000000 08:02 3545605 /usr/lib/banshee/Extensions/Banshee.Fixup.dll<br>7fd65cdb0000-7fd65cdc0000 rw-p 00000000 00:00 0<br>... snip ...

Since the ld-2.19.so library is mapped in r-xp mode — i.e. not<br>writable — it's a safe bet that anything else mapping that library is<br>sharing pages with us. Even libm-2.19.so, which is writable, is probably<br>also sharing pages, since the p means private with copy-on-write<br>semantics.

This is noteworthy. If the process does<br>write to the libm's memory region, the kernel will craftily allocate a new<br>page, copy the original data to it, and present that to the process.<br>This new page will belong<br>exclusively to us, and we should count that page towards<br>our overall process memory footprint, while still discounting the<br>untouched pages that are still shared.

Enter /proc/$$/smaps.

A coworker showed me this file, and its exhaustive accounting of memory<br>mappings that makes the maps file look like a a tweet from the kernel<br>(#proc #systemstats #YOLO)

7fd65ccf1000-7fd65ccf2000 rw-p 00105000 08:02 1845997 /lib/x86_64-linux-gnu/libm-2.19.so<br>Size: 4 kB<br>Rss: 4 kB<br>Pss: 4 kB<br>Shared_Clean: 0 kB<br>Shared_Dirty: 0 kB<br>Private_Clean: 0 kB<br>Private_Dirty: 4 kB<br>Referenced: 4 kB<br>Anonymous: 4 kB<br>AnonHugePages: 0 kB<br>Swap: 0 kB<br>KernelPageSize: 4 kB<br>MMUPageSize: 4 kB<br>Locked: 0 kB<br>VmFlags: rd wr mr mw me ac sd

7fd65ccf2000-7fd65cd15000 r-xp 00000000 08:02 1845995 /lib/x86_64-linux-gnu/ld-2.19.so<br>Size: 140 kB<br>Rss: 124 kB<br>Pss: 1 kB<br>Shared_Clean: 124 kB<br>Shared_Dirty: 0 kB<br>Private_Clean: 0 kB<br>Private_Dirty: 0 kB<br>Referenced: 124 kB<br>Anonymous: 0 kB<br>AnonHugePages: 0 kB<br>Swap: 0 kB<br>KernelPageSize: 4 kB<br>MMUPageSize: 4 kB<br>Locked: 0 kB<br>VmFlags: rd ex mr mw me dw sd

Yeah, that's way more information than we're used to from the old maps file.<br>But what does it all mean? You can get some of the basics from proc(5) ,<br>but my local copy didn't even mention the Pss column.

Why not experiment?

(I'm glad you asked. What follows is a safari into the uncharted jungle of<br>memory management, aided by our trusty friend Exploratory Programming in C!)

Exploratory Programming in C

I love exploratory programming. Home directories on...