Linux 的 Out-of-Memory (OOM) Killer

这个也让咱遇上了。

top – 10:53:30 up 111 days, 16:49, 1 user, load average: 2.68, 2.98, 3.39
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 20.6%us, 8.9%sy, 0.0%ni, 69.7%id, 0.3%wa, 0.0%hi, 0.4%si, 0.0%st
Mem: 8168412k total, 7844888k used, 323524k free, 106604k buffers
Swap: 2097144k total, 88664k used, 2008480k free, 6534044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13596 mysql 15 0 2325m 1.0g 4060 S 283.3 13.0 574:16.15 mysqld
1 root 15 0 10348 88 56 S 0.0 0.0 12:35.68 init
2 root RT -5 0 0 0 S 0.0 0.0 0:02.39 migration/0

#cat /var/log/messages |grep ‘May 26 03’
May 26 03:29:35 touareg kernel: Node 1 Normal per-cpu: empty
May 26 03:29:35 touareg kernel: Node 1 HighMem per-cpu: empty
May 26 03:29:36 touareg kernel: Free pages: 12980kB (0kB HighMem)
May 26 03:29:36 touareg kernel: Active:612100 inactive:1406689 dirty:0 writeback:0 unstable:0 free:3245 slab:6346 mapped-file:1004 mapped-anon:2018482 pagetables:6473
May 26 03:29:37 touareg kernel: Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:29:38 touareg kernel: lowmem_reserve[]: 0 1242 6040 6040
May 26 03:29:38 touareg kernel: Node 0 DMA32 free:1876kB min:1768kB low:2208kB high:2652kB active:1092684kB inactive:196800kB present:1272420kB pages_scanned:3346160 all_unreclaimable? yes
May 26 03:29:39 touareg kernel: lowmem_reserve[]: 0 0 4797 4797
May 26 03:29:39 touareg kernel: Node 0 Normal free:6548kB min:6836kB low:8544kB high:10252kB active:514812kB inactive:4318496kB present:4912640kB pages_scanned:17304011 all_unreclaimable? yes
May 26 03:29:40 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:29:41 touareg kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:29:42 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:29:43 touareg kernel: Node 1 DMA free:1736kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10696kB pages_scanned:0 all_unreclaimable? yes
May 26 03:29:43 touareg kernel: lowmem_reserve[]: 0 2004 2004 2004
May 26 03:29:43 touareg kernel: Node 1 DMA32 free:2820kB min:2856kB low:3568kB high:4284kB active:845864kB inactive:1106756kB present:2052320kB pages_scanned:4059828 all_unreclaimable? yes
May 26 03:29:44 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:29:45 touareg kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:29:45 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:29:46 touareg kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:29:46 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:29:47 touareg kernel: Node 0 DMA: empty
May 26 03:29:47 touareg kernel: Node 0 DMA32: 31*4kB 7*8kB 4*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1876kB
May 26 03:29:47 touareg kernel: Node 0 Normal: 115*4kB 3*8kB 11*16kB 8*32kB 4*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 6548kB
May 26 03:29:48 touareg kernel: Node 0 HighMem: empty
May 26 03:29:48 touareg kernel: Node 1 DMA: 2*4kB 2*8kB 1*16kB 5*32kB 4*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1736kB
May 26 03:29:48 touareg kernel: Node 1 DMA32: 63*4kB 7*8kB 17*16kB 2*32kB 2*64kB 6*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2820kB
May 26 03:29:49 touareg kernel: Node 1 Normal: empty
May 26 03:29:49 touareg kernel: Node 1 HighMem: empty
May 26 03:29:50 touareg kernel: 1339 pagecache pages
May 26 03:29:50 touareg kernel: Swap cache: add 1051209, delete 1051023, find 5464906/5502739, race 0+41
May 26 03:29:50 touareg kernel: Free swap = 0kB
May 26 03:29:51 touareg kernel: Total swap = 2097144kB
May 26 03:29:51 touareg kernel: Free swap: 0kB
May 26 03:29:51 touareg kernel: 2293760 pages of RAM
May 26 03:29:52 touareg kernel: 251657 reserved pages
May 26 03:29:52 touareg kernel: 6005 pages shared
May 26 03:29:53 touareg kernel: 263 pages swap cached
May 26 03:29:54 touareg kernel: Out of memory: Killed process 13285 (mysqld).
May 26 03:29:54 touareg kernel: sshd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
May 26 03:29:55 touareg kernel:
May 26 03:29:55 touareg kernel: Call Trace:
May 26 03:29:56 touareg kernel: [] out_of_memory+0x8e/0x2f5
May 26 03:29:57 touareg kernel: [] __alloc_pages+0x245/0x2ce
May 26 03:29:57 touareg kernel: [] read_swap_cache_async+0x45/0xd8
May 26 03:29:57 touareg kernel: [] wake_bit_function+0x0/0x23
May 26 03:29:58 touareg kernel: [] swapin_readahead+0x60/0xd3
May 26 03:29:58 touareg kernel: [] __handle_mm_fault+0x9bc/0xe5c
May 26 03:29:59 touareg kernel: [] do_page_fault+0x4cb/0x830
May 26 03:29:59 touareg kernel: [] skb_dequeue+0x48/0x50
May 26 03:30:00 touareg kernel: [] unix_release_sock+0x19e/0x1fa
May 26 03:30:00 touareg kernel: [] dput+0x2c/0x114
May 26 03:30:00 touareg kernel: [] error_exit+0x0/0x84
May 26 03:30:01 touareg kernel:
May 26 03:30:01 touareg kernel: Mem-info:
May 26 03:30:01 touareg kernel: Node 0 DMA per-cpu: empty
May 26 03:30:02 touareg kernel: Node 0 DMA32 per-cpu:
May 26 03:30:03 touareg kernel: cpu 0 hot: high 186, batch 31 used:89
May 26 03:30:03 touareg kernel: cpu 0 cold: high 62, batch 15 used:57
May 26 03:30:03 touareg kernel: cpu 1 hot: high 186, batch 31 used:0
May 26 03:30:04 touareg kernel: cpu 1 cold: high 62, batch 15 used:0
May 26 03:30:04 touareg kernel: cpu 2 hot: high 186, batch 31 used:180
May 26 03:30:04 touareg kernel: cpu 2 cold: high 62, batch 15 used:61
May 26 03:30:05 touareg kernel: cpu 3 hot: high 186, batch 31 used:0
May 26 03:30:05 touareg kernel: cpu 3 cold: high 62, batch 15 used:29
May 26 03:30:05 touareg kernel: cpu 4 hot: high 186, batch 31 used:155
May 26 03:30:05 touareg kernel: cpu 4 cold: high 62, batch 15 used:48
May 26 03:30:05 touareg kernel: cpu 5 hot: high 186, batch 31 used:0
May 26 03:30:05 touareg kernel: cpu 5 cold: high 62, batch 15 used:14
May 26 03:30:06 touareg kernel: cpu 6 hot: high 186, batch 31 used:98
May 26 03:30:06 touareg kernel: cpu 6 cold: high 62, batch 15 used:54
May 26 03:30:06 touareg kernel: cpu 7 hot: high 186, batch 31 used:0
May 26 03:30:06 touareg kernel: cpu 7 cold: high 62, batch 15 used:2
May 26 03:30:06 touareg kernel: Node 0 Normal per-cpu:
May 26 03:30:06 touareg kernel: cpu 0 hot: high 186, batch 31 used:3
May 26 03:30:07 touareg kernel: cpu 0 cold: high 62, batch 15 used:56
May 26 03:30:07 touareg kernel: cpu 1 hot: high 186, batch 31 used:0
May 26 03:30:07 touareg kernel: cpu 1 cold: high 62, batch 15 used:0
May 26 03:30:07 touareg kernel: cpu 2 hot: high 186, batch 31 used:15
May 26 03:30:07 touareg kernel: cpu 2 cold: high 62, batch 15 used:48
May 26 03:30:07 touareg kernel: cpu 3 hot: high 186, batch 31 used:0
May 26 03:30:08 touareg kernel: cpu 3 cold: high 62, batch 15 used:0
May 26 03:30:08 touareg kernel: cpu 4 hot: high 186, batch 31 used:103
May 26 03:30:08 touareg kernel: cpu 4 cold: high 62, batch 15 used:51
May 26 03:30:08 touareg kernel: cpu 5 hot: high 186, batch 31 used:0
May 26 03:30:08 touareg kernel: cpu 5 cold: high 62, batch 15 used:0
May 26 03:30:08 touareg kernel: cpu 6 hot: high 186, batch 31 used:113
May 26 03:30:09 touareg kernel: cpu 6 cold: high 62, batch 15 used:60
May 26 03:30:09 touareg kernel: cpu 7 hot: high 186, batch 31 used:0
May 26 03:30:09 touareg kernel: cpu 7 cold: high 62, batch 15 used:0
May 26 03:30:09 touareg kernel: Node 0 HighMem per-cpu: empty
May 26 03:30:09 touareg kernel: Node 1 DMA per-cpu:
May 26 03:30:09 touareg kernel: cpu 0 hot: high 0, batch 1 used:0
May 26 03:30:09 touareg kernel: cpu 0 cold: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 1 hot: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 1 cold: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 2 hot: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 2 cold: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 3 hot: high 0, batch 1 used:0
May 26 03:30:10 touareg kernel: cpu 3 cold: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 4 hot: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 4 cold: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 5 hot: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 5 cold: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 6 hot: high 0, batch 1 used:0
May 26 03:30:11 touareg kernel: cpu 6 cold: high 0, batch 1 used:0
May 26 03:30:12 touareg kernel: cpu 7 hot: high 0, batch 1 used:0
May 26 03:30:12 touareg kernel: cpu 7 cold: high 0, batch 1 used:0
May 26 03:30:12 touareg kernel: Node 1 DMA32 per-cpu:
May 26 03:30:12 touareg kernel: cpu 0 hot: high 186, batch 31 used:0
May 26 03:30:12 touareg kernel: cpu 0 cold: high 62, batch 15 used:0
May 26 03:30:12 touareg kernel: cpu 1 hot: high 186, batch 31 used:27
May 26 03:30:12 touareg kernel: cpu 1 cold: high 62, batch 15 used:37
May 26 03:30:12 touareg kernel: cpu 2 hot: high 186, batch 31 used:0
May 26 03:30:13 touareg kernel: cpu 2 cold: high 62, batch 15 used:0
May 26 03:30:13 touareg kernel: cpu 3 hot: high 186, batch 31 used:13
May 26 03:30:13 touareg kernel: cpu 3 cold: high 62, batch 15 used:15
May 26 03:30:13 touareg kernel: cpu 4 hot: high 186, batch 31 used:0
May 26 03:30:13 touareg kernel: cpu 4 cold: high 62, batch 15 used:0
May 26 03:30:13 touareg kernel: cpu 5 hot: high 186, batch 31 used:176
May 26 03:30:13 touareg kernel: cpu 5 cold: high 62, batch 15 used:38
May 26 03:30:14 touareg kernel: cpu 6 hot: high 186, batch 31 used:0
May 26 03:30:14 touareg kernel: cpu 6 cold: high 62, batch 15 used:0
May 26 03:30:14 touareg kernel: cpu 7 hot: high 186, batch 31 used:10
May 26 03:30:14 touareg kernel: cpu 7 cold: high 62, batch 15 used:58
May 26 03:30:14 touareg kernel: Node 1 Normal per-cpu: empty
May 26 03:30:14 touareg kernel: Node 1 HighMem per-cpu: empty
May 26 03:30:15 touareg kernel: Free pages: 12348kB (0kB HighMem)
May 26 03:30:15 touareg kernel: Active:1054383 inactive:964358 dirty:0 writeback:0 unstable:0 free:3087 slab:6302 mapped-file:1005 mapped-anon:2018488 pagetables:6477
May 26 03:30:15 touareg kernel: Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:30:15 touareg kernel: lowmem_reserve[]: 0 1242 6040 6040
May 26 03:30:15 touareg kernel: Node 0 DMA32 free:1720kB min:1768kB low:2208kB high:2652kB active:699680kB inactive:588284kB present:1272420kB pages_scanned:2814597 all_unreclaimable? yes
May 26 03:30:15 touareg kernel: lowmem_reserve[]: 0 0 4797 4797
May 26 03:30:15 touareg kernel: Node 0 Normal free:6116kB min:6836kB low:8544kB high:10252kB active:2214940kB inactive:2618664kB present:4912640kB pages_scanned:16399035 all_unreclaimable? yes
May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:30:16 touareg kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:30:16 touareg kernel: Node 1 DMA free:1736kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10696kB pages_scanned:0 all_unreclaimable? yes
May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 2004 2004 2004
May 26 03:30:16 touareg kernel: Node 1 DMA32 free:2776kB min:2856kB low:3568kB high:4284kB active:1303380kB inactive:650400kB present:2052320kB pages_scanned:4508965 all_unreclaimable? yes
May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:30:17 touareg kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:30:17 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:30:17 touareg kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 26 03:30:17 touareg kernel: lowmem_reserve[]: 0 0 0 0
May 26 03:30:17 touareg kernel: Node 0 DMA: empty
May 26 03:30:17 touareg kernel: Node 0 DMA32: 0*4kB 5*8kB 3*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1720kB
May 26 03:30:17 touareg kernel: Node 0 Normal: 497*4kB 44*8kB 40*16kB 20*32kB 9*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 6116kB
May 26 03:30:18 touareg kernel: Node 0 HighMem: empty
May 26 03:30:18 touareg kernel: Node 1 DMA: 2*4kB 2*8kB 1*16kB 5*32kB 4*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1736kB
May 26 03:30:18 touareg kernel: Node 1 DMA32: 184*4kB 33*8kB 17*16kB 7*32kB 4*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2776kB
May 26 03:30:18 touareg kernel: Node 1 Normal: empty
May 26 03:30:18 touareg kernel: Node 1 HighMem: empty
May 26 03:30:18 touareg kernel: 1379 pagecache pages
May 26 03:30:18 touareg kernel: Swap cache: add 1053289, delete 1053103, find 5465453/5503467, race 0+41
May 26 03:30:18 touareg kernel: Free swap = 0kB
May 26 03:30:19 touareg kernel: Total swap = 2097144kB
May 26 03:30:19 touareg kernel: Free swap: 0kB
May 26 03:30:19 touareg kernel: 2293760 pages of RAM
May 26 03:30:19 touareg kernel: 251657 reserved pages
May 26 03:30:19 touareg kernel: 5999 pages shared
May 26 03:30:19 touareg kernel: 263 pages swap cached
May 26 03:30:19 touareg kernel: Out of memory: Killed process 13328 (mysqld).
May 26 03:30:19 touareg kernel: mysqld: page allocation failure. order:0, mode:0x201d2

先增加2G swap再说

#dd if=/dev/zero of=/opt/swapfile bs=1M count=2048
#mkswap /opt/swapfile
#swapon /opt/swapfile
#swapon -s

Filename Type Size Used Priority
/dev/mapper/VolGroup00-LogVol00 partition 2097144 87352 -1
/opt/swapfile file 2097144 0 -2

在fstab增加相关记录，否则重启就没了
#vi /etc/fstab

/opt/swapfile swap swap defaults 0 0

参考
http://www.dbanotes.net/database/linux_outofmemory_oom_killer.html
http://jk.scanmon.com/1228.html
http://www.itjaj.com/thread-3029-1-1.html