Difference between revisions of "Memory management"
imported>ThorstenStaerk |
imported>ThorstenStaerk |
||
| (One intermediate revision by one other user not shown) | |||
| Line 229: | Line 229: | ||
total 908K | total 908K | ||
</pre> | </pre> | ||
| + | |||
| + | = Terms = | ||
| + | * heap vs stack | ||
| + | * pool vs swap | ||
| + | * shared mem, extended mem, buffer, cache, heap, stack | ||
| + | * resident mem vs virtual mem, allocated, mapped, free, physical, used | ||
= See also = | = See also = | ||
| Line 238: | Line 244: | ||
* http://lwn.net/Articles/230975/ | * http://lwn.net/Articles/230975/ | ||
* https://fedorahosted.org/libpagemap/ | * https://fedorahosted.org/libpagemap/ | ||
| + | * http://unix.stackexchange.com/questions/105604/the-meaning-of-output-of-pmap | ||
Latest revision as of 14:47, 19 December 2013
Memory management is a fun thing. Think about the following:
You have a program running on your computer and you want to know how big its memory footprint is. Half of the program has not yet been loaded from disk, a quarter has been swapped out of main memory. The rest is using 200k of shared libraries and has 1 megabyte in the file system cache that can be freed up with no serious consequences. If you are interested in this kind of things, this article is for you :)
Contents
File system cache
Linux tends to fill up all your computer's memory over time. This is not bad, but works as designed. Whenever Linux reads a file from disk, it keeps the content in memory so it does not have to wait for the disk next time. This is called the file system cache, you can display its size with the command
# free
total used free shared buffers cached
Mem: 6083904 1175880 4908024 0 7556 230460
-/+ buffers/cache: 937864 5146040
Swap: 0 0 0
If you really want to clean the file system cache, you will probably want to clean dentries and inodes as well, you can do it with the commands:
sync echo 3 > /proc/sys/vm/drop_caches
memory consumption of all programs
# ps aux|awk '{print $6" " $11;s=s+$6} END {print s}' | sort -n
will give you a sorted list which program consumes how much, about like this:
[...] 4028 dovecot/auth 4528 /sbin/haveged 8348 /usr/sbin/httpd2-prefork 38764 /usr/sbin/httpd2-prefork 43628 /usr/sbin/httpd2-prefork 279148
memory consumption of a program
killall gedit
# sync;echo 3 > /proc/sys/vm/drop_caches
[1]+ Terminated gedit
# free
total used free shared buffers cached
Mem: 6083904 1164036 4919868 0 508 210916
-/+ buffers/cache: 952612 5131292
Swap: 0 0 0
# free -k
total used free shared buffers cached
Mem: 6083904 1163732 4920172 0 516 210888
-/+ buffers/cache: 952328 5131576
Swap: 0 0 0
# gedit &
[1] 5859
# free -k
total used free shared buffers cached
Mem: 6083904 1221472 4862432 0 11252 237796
-/+ buffers/cache: 972424 5111480
Swap: 0 0 0
# sync;echo 3 > /proc/sys/vm/drop_caches
# free -k
total used free shared buffers cached
Mem: 6083904 1184668 4899236 0 516 217036
-/+ buffers/cache: 967116 5116788
Swap: 0 0 0
# echo $((1184668-1163732))
20936
# ps -auxf | grep -E "(VSZ|gedit)"
warning: bad ps syntax, perhaps a bogus '-'?
See http://gitorious.org/procps/procps/blobs/master/Documentation/FAQ
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 5859 0.7 0.4 767088 28760 pts/5 Sl 11:25 0:00 \_ gedit
root 5871 0.0 0.0 7004 872 pts/5 S+ 11:26 0:00 \_ grep --color=auto -E (VSZ|gedit)
system memory map
To get a complete memory map of your system use the command
pmap $(ps -A | awk '{print $1}'|grep -v PID) | sort | grep \^0
The result will look like this:
... 00007fd6dbf45000 4K rw--- /lib/libnss_compat-2.11.1.so 00007fd6dbf46000 1524K r-x-- /lib/libc-2.11.1.so 00007fd6dc0c3000 2044K ----- /lib/libc-2.11.1.so ...
provoke a memory leak
Here is how to provoke a memory leak, my program ram_gourmet:
main.cpp
class t
{
public:
t(){};
};
void pollute()
{
t* polluter=new t();
}
int main()
{
while (true) pollute();
}
Attention: It rendered my compi useless within 10 seconds.
Compile, link and run this using:
g++ -o ram_gourmet main.cpp && ./ram_gourmet
This program can be developed further to be a cache condenser
page size
Ok, first I want to find out my system's page size so I write a program:
#include <unistd.h>
#include <stdio.h>
int main()
{
int sz = getpagesize();
printf("%d",sz);
}
And compile it:
# gcc main.c
And execute it:
# ./a.out 4096
Ok, so my system's page size is 4K.
Now let's write the next program and check how much memory it consumes. Here is the code:
int main()
{
while (1);
}
Compile it
# gcc main.c
And execute it in the background:
# ./a.out & [5] 24058
Ok, it's process ID is 24058. Let's look at its memory consumption:
# pmap 24058 24058: a.out START SIZE RSS PSS DIRTY SWAP PERM MAPPING 0000000000400000 4K 4K 4K 0K 0K r-xp /home/tstaerk/test/a.out 0000000000600000 4K 4K 4K 4K 0K r--p /home/tstaerk/test/a.out 0000000000601000 4K 4K 4K 4K 0K rw-p /home/tstaerk/test/a.out 00007fda6f01e000 1364K 160K 1K 0K 0K r-xp /lib64/libc-2.11.1.so 00007fda6f173000 2044K 0K 0K 0K 0K ---p /lib64/libc-2.11.1.so 00007fda6f372000 16K 16K 16K 16K 0K r--p /lib64/libc-2.11.1.so 00007fda6f376000 4K 4K 4K 4K 0K rw-p /lib64/libc-2.11.1.so 00007fda6f377000 20K 12K 12K 12K 0K rw-p [anon] 00007fda6f37c000 124K 104K 1K 0K 0K r-xp /lib64/ld-2.11.1.so 00007fda6f55a000 12K 12K 12K 12K 0K rw-p [anon] 00007fda6f599000 4K 4K 4K 4K 0K rw-p [anon] 00007fda6f59a000 4K 4K 4K 4K 0K r--p /lib64/ld-2.11.1.so 00007fda6f59b000 4K 4K 4K 4K 0K rw-p /lib64/ld-2.11.1.so 00007fda6f59c000 4K 4K 4K 4K 0K rw-p [anon] 00007fff625e5000 84K 12K 12K 12K 0K rw-p [stack] 00007fff625ff000 4K 4K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K r-xp [vsyscall] Total: 3704K 352K 86K 80K 0K 136K writable-private, 3568K readonly-private, 0K shared, and 352K referenced
Ok, it looks like the binary has a lot of shared libraries that it makes use of. Let's see if this is true:
# ldd a.out
linux-vdso.so.1 => (0x00007fffc4bee000)
libc.so.6 => /lib64/libc.so.6 (0x00007f2c6cc78000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2c6d013000)
It is true, it is a dynamically-linked executable. Let's make it a static one:
# gcc --static main.c # ll a.out -rwxr-xr-x 1 tstaerk users 2947494 2012-06-06 10:14 a.out
And run it again:
# ./a.out & [7] 21850
And look at its memory map again:
# pmap 21850 21850: a.out START SIZE RSS PSS DIRTY SWAP PERM MAPPING 0000000000400000 572K 84K 84K 0K 0K r-xp /home/tstaerk/test/a.out 000000000068e000 8K 8K 8K 8K 0K rw-p /home/tstaerk/test/a.out 0000000000690000 152K 20K 20K 20K 0K rw-p [heap] 00007fff34f2f000 84K 8K 8K 8K 0K rw-p [stack] 00007fff34f56000 4K 0K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K r-xp [vsyscall] Total: 824K 120K 120K 36K 0K 244K writable-private, 580K readonly-private, 0K shared, and 120K referenced
Well then on another distro let's give it something to think about:
root@euve31070:~# cat main.c
#include <stdlib.h>
int main ()
{
char * buffer;
buffer = (char*) malloc (800000);
buffer = "hi";
while (1) {};
}
root@euve31070:~# gcc --static main.c
root@euve31070:~# ./a.out &
[1] 28793
root@euve31070:~# pmap 28793
28793: ./a.out
0000000000400000 652K r-x-- /root/a.out
00000000006a2000 8K rw--- /root/a.out
00000000006a4000 12K rw--- [ anon ]
0000000000c2b000 140K rw--- [ anon ]
00007fcd87007000 784K rw--- [ anon ]
00007fff93aab000 84K rw--- [ stack ]
00007fff93bfe000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 1692K
root@euve31070:~# vi main.c
root@euve31070:~# cat main.c
#include <stdlib.h>
int main ()
{
// char * buffer;
// buffer = (char*) malloc (800000);
// buffer = "hi";
while (1) {};
}
root@euve31070:~# gcc --static main.c
root@euve31070:~# ./a.out &
[2] 28802
root@euve31070:~# pmap 28802
28802: ./a.out
0000000000400000 652K r-x-- /root/a.out
00000000006a2000 8K rw--- /root/a.out
00000000006a4000 12K rw--- [ anon ]
000000000262b000 140K rw--- [ anon ]
00007fff6d699000 84K rw--- [ stack ]
00007fff6d79e000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 908K
Terms
- heap vs stack
- pool vs swap
- shared mem, extended mem, buffer, cache, heap, stack
- resident mem vs virtual mem, allocated, mapped, free, physical, used
See also
- http://linux-mm.org
- http://linux-mm.org/Low_On_Memory
- http://www.linux.com/learn/tutorials/42048-uncover-the-meaning-of-tops-statistics
- http://en.wikipedia.org/wiki/Resident_set_size
- http://en.wikipedia.org/wiki/Mmap
- http://lwn.net/Articles/230975/
- https://fedorahosted.org/libpagemap/
- http://unix.stackexchange.com/questions/105604/the-meaning-of-output-of-pmap
