Skip to content


Linux 的 Out-of-Memory (OOM) Killer

这个也让咱遇上了。

top – 10:53:30 up 111 days, 16:49, 1 user, load average: 2.68, 2.98, 3.39 Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie Cpu(s): 20.6%us, 8.9%sy, 0.0%ni, 69.7%id, 0.3%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 8168412k total, 7844888k used, 323524k free, 106604k buffers Swap: 2097144k total, 88664k used, 2008480k free, 6534044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13596 mysql 15 0 2325m 1.0g 4060 S 283.3 13.0 574:16.15 mysqld
1 root 15 0 10348 88 56 S 0.0 0.0 12:35.68 init
2 root RT -5 0 0 0 S 0.0 0.0 0:02.39 migration/0

#cat /var/log/messages |grep ‘May 26 03’ May 26 03:29:35 touareg kernel: Node 1 Normal per-cpu: empty May 26 03:29:35 touareg kernel: Node 1 HighMem per-cpu: empty May 26 03:29:36 touareg kernel: Free pages: 12980kB (0kB HighMem) May 26 03:29:36 touareg kernel: Active:612100 inactive:1406689 dirty:0 writeback:0 unstable:0 free:3245 slab:6346 mapped-file:1004 mapped-anon:2018482 pagetables:6473 May 26 03:29:37 touareg kernel: Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:29:38 touareg kernel: lowmem_reserve[]: 0 1242 6040 6040 May 26 03:29:38 touareg kernel: Node 0 DMA32 free:1876kB min:1768kB low:2208kB high:2652kB active:1092684kB inactive:196800kB present:1272420kB pages_scanned:3346160 all_unreclaimable? yes May 26 03:29:39 touareg kernel: lowmem_reserve[]: 0 0 4797 4797 May 26 03:29:39 touareg kernel: Node 0 Normal free:6548kB min:6836kB low:8544kB high:10252kB active:514812kB inactive:4318496kB present:4912640kB pages_scanned:17304011 all_unreclaimable? yes May 26 03:29:40 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:29:41 touareg kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:29:42 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:29:43 touareg kernel: Node 1 DMA free:1736kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10696kB pages_scanned:0 all_unreclaimable? yes May 26 03:29:43 touareg kernel: lowmem_reserve[]: 0 2004 2004 2004 May 26 03:29:43 touareg kernel: Node 1 DMA32 free:2820kB min:2856kB low:3568kB high:4284kB active:845864kB inactive:1106756kB present:2052320kB pages_scanned:4059828 all_unreclaimable? yes May 26 03:29:44 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:29:45 touareg kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:29:45 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:29:46 touareg kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:29:46 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:29:47 touareg kernel: Node 0 DMA: empty May 26 03:29:47 touareg kernel: Node 0 DMA32: 314kB 78kB 416kB 132kB 164kB 0128kB 0256kB 1512kB 11024kB 02048kB 04096kB = 1876kB May 26 03:29:47 touareg kernel: Node 0 Normal: 1154kB 38kB 1116kB 832kB 464kB 2128kB 2256kB 1512kB 01024kB 02048kB 14096kB = 6548kB May 26 03:29:48 touareg kernel: Node 0 HighMem: empty May 26 03:29:48 touareg kernel: Node 1 DMA: 24kB 28kB 116kB 532kB 464kB 0128kB 1256kB 0512kB 11024kB 02048kB 04096kB = 1736kB May 26 03:29:48 touareg kernel: Node 1 DMA32: 634kB 78kB 1716kB 232kB 264kB 6128kB 1256kB 0512kB 11024kB 02048kB 04096kB = 2820kB May 26 03:29:49 touareg kernel: Node 1 Normal: empty May 26 03:29:49 touareg kernel: Node 1 HighMem: empty May 26 03:29:50 touareg kernel: 1339 pagecache pages May 26 03:29:50 touareg kernel: Swap cache: add 1051209, delete 1051023, find 5464906/5502739, race 0+41 May 26 03:29:50 touareg kernel: Free swap = 0kB May 26 03:29:51 touareg kernel: Total swap = 2097144kB May 26 03:29:51 touareg kernel: Free swap: 0kB May 26 03:29:51 touareg kernel: 2293760 pages of RAM May 26 03:29:52 touareg kernel: 251657 reserved pages May 26 03:29:52 touareg kernel: 6005 pages shared May 26 03:29:53 touareg kernel: 263 pages swap cached May 26 03:29:54 touareg kernel: Out of memory: Killed process 13285 (mysqld). May 26 03:29:54 touareg kernel: sshd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0 May 26 03:29:55 touareg kernel: May 26 03:29:55 touareg kernel: Call Trace: May 26 03:29:56 touareg kernel: [

] out_of_memory+0x8e/0x2f5 May 26 03:29:57 touareg kernel: [ ] __alloc_pages+0x245/0x2ce May 26 03:29:57 touareg kernel: [ ] read_swap_cache_async+0x45/0xd8 May 26 03:29:57 touareg kernel: [ ] wake_bit_function+0x0/0x23 May 26 03:29:58 touareg kernel: [ ] swapin_readahead+0x60/0xd3 May 26 03:29:58 touareg kernel: [ ] __handle_mm_fault+0x9bc/0xe5c May 26 03:29:59 touareg kernel: [ ] do_page_fault+0x4cb/0x830 May 26 03:29:59 touareg kernel: [ ] skb_dequeue+0x48/0x50 May 26 03:30:00 touareg kernel: [ ] unix_release_sock+0x19e/0x1fa May 26 03:30:00 touareg kernel: [ ] dput+0x2c/0x114 May 26 03:30:00 touareg kernel: [ ] error_exit+0x0/0x84 May 26 03:30:01 touareg kernel: May 26 03:30:01 touareg kernel: Mem-info: May 26 03:30:01 touareg kernel: Node 0 DMA per-cpu: empty May 26 03:30:02 touareg kernel: Node 0 DMA32 per-cpu: May 26 03:30:03 touareg kernel: cpu 0 hot: high 186, batch 31 used:89 May 26 03:30:03 touareg kernel: cpu 0 cold: high 62, batch 15 used:57 May 26 03:30:03 touareg kernel: cpu 1 hot: high 186, batch 31 used:0 May 26 03:30:04 touareg kernel: cpu 1 cold: high 62, batch 15 used:0 May 26 03:30:04 touareg kernel: cpu 2 hot: high 186, batch 31 used:180 May 26 03:30:04 touareg kernel: cpu 2 cold: high 62, batch 15 used:61 May 26 03:30:05 touareg kernel: cpu 3 hot: high 186, batch 31 used:0 May 26 03:30:05 touareg kernel: cpu 3 cold: high 62, batch 15 used:29 May 26 03:30:05 touareg kernel: cpu 4 hot: high 186, batch 31 used:155 May 26 03:30:05 touareg kernel: cpu 4 cold: high 62, batch 15 used:48 May 26 03:30:05 touareg kernel: cpu 5 hot: high 186, batch 31 used:0 May 26 03:30:05 touareg kernel: cpu 5 cold: high 62, batch 15 used:14 May 26 03:30:06 touareg kernel: cpu 6 hot: high 186, batch 31 used:98 May 26 03:30:06 touareg kernel: cpu 6 cold: high 62, batch 15 used:54 May 26 03:30:06 touareg kernel: cpu 7 hot: high 186, batch 31 used:0 May 26 03:30:06 touareg kernel: cpu 7 cold: high 62, batch 15 used:2 May 26 03:30:06 touareg kernel: Node 0 Normal per-cpu: May 26 03:30:06 touareg kernel: cpu 0 hot: high 186, batch 31 used:3 May 26 03:30:07 touareg kernel: cpu 0 cold: high 62, batch 15 used:56 May 26 03:30:07 touareg kernel: cpu 1 hot: high 186, batch 31 used:0 May 26 03:30:07 touareg kernel: cpu 1 cold: high 62, batch 15 used:0 May 26 03:30:07 touareg kernel: cpu 2 hot: high 186, batch 31 used:15 May 26 03:30:07 touareg kernel: cpu 2 cold: high 62, batch 15 used:48 May 26 03:30:07 touareg kernel: cpu 3 hot: high 186, batch 31 used:0 May 26 03:30:08 touareg kernel: cpu 3 cold: high 62, batch 15 used:0 May 26 03:30:08 touareg kernel: cpu 4 hot: high 186, batch 31 used:103 May 26 03:30:08 touareg kernel: cpu 4 cold: high 62, batch 15 used:51 May 26 03:30:08 touareg kernel: cpu 5 hot: high 186, batch 31 used:0 May 26 03:30:08 touareg kernel: cpu 5 cold: high 62, batch 15 used:0 May 26 03:30:08 touareg kernel: cpu 6 hot: high 186, batch 31 used:113 May 26 03:30:09 touareg kernel: cpu 6 cold: high 62, batch 15 used:60 May 26 03:30:09 touareg kernel: cpu 7 hot: high 186, batch 31 used:0 May 26 03:30:09 touareg kernel: cpu 7 cold: high 62, batch 15 used:0 May 26 03:30:09 touareg kernel: Node 0 HighMem per-cpu: empty May 26 03:30:09 touareg kernel: Node 1 DMA per-cpu: May 26 03:30:09 touareg kernel: cpu 0 hot: high 0, batch 1 used:0 May 26 03:30:09 touareg kernel: cpu 0 cold: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 1 hot: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 1 cold: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 2 hot: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 2 cold: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 3 hot: high 0, batch 1 used:0 May 26 03:30:10 touareg kernel: cpu 3 cold: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 4 hot: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 4 cold: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 5 hot: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 5 cold: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 6 hot: high 0, batch 1 used:0 May 26 03:30:11 touareg kernel: cpu 6 cold: high 0, batch 1 used:0 May 26 03:30:12 touareg kernel: cpu 7 hot: high 0, batch 1 used:0 May 26 03:30:12 touareg kernel: cpu 7 cold: high 0, batch 1 used:0 May 26 03:30:12 touareg kernel: Node 1 DMA32 per-cpu: May 26 03:30:12 touareg kernel: cpu 0 hot: high 186, batch 31 used:0 May 26 03:30:12 touareg kernel: cpu 0 cold: high 62, batch 15 used:0 May 26 03:30:12 touareg kernel: cpu 1 hot: high 186, batch 31 used:27 May 26 03:30:12 touareg kernel: cpu 1 cold: high 62, batch 15 used:37 May 26 03:30:12 touareg kernel: cpu 2 hot: high 186, batch 31 used:0 May 26 03:30:13 touareg kernel: cpu 2 cold: high 62, batch 15 used:0 May 26 03:30:13 touareg kernel: cpu 3 hot: high 186, batch 31 used:13 May 26 03:30:13 touareg kernel: cpu 3 cold: high 62, batch 15 used:15 May 26 03:30:13 touareg kernel: cpu 4 hot: high 186, batch 31 used:0 May 26 03:30:13 touareg kernel: cpu 4 cold: high 62, batch 15 used:0 May 26 03:30:13 touareg kernel: cpu 5 hot: high 186, batch 31 used:176 May 26 03:30:13 touareg kernel: cpu 5 cold: high 62, batch 15 used:38 May 26 03:30:14 touareg kernel: cpu 6 hot: high 186, batch 31 used:0 May 26 03:30:14 touareg kernel: cpu 6 cold: high 62, batch 15 used:0 May 26 03:30:14 touareg kernel: cpu 7 hot: high 186, batch 31 used:10 May 26 03:30:14 touareg kernel: cpu 7 cold: high 62, batch 15 used:58 May 26 03:30:14 touareg kernel: Node 1 Normal per-cpu: empty May 26 03:30:14 touareg kernel: Node 1 HighMem per-cpu: empty May 26 03:30:15 touareg kernel: Free pages: 12348kB (0kB HighMem) May 26 03:30:15 touareg kernel: Active:1054383 inactive:964358 dirty:0 writeback:0 unstable:0 free:3087 slab:6302 mapped-file:1005 mapped-anon:2018488 pagetables:6477 May 26 03:30:15 touareg kernel: Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:30:15 touareg kernel: lowmem_reserve[]: 0 1242 6040 6040 May 26 03:30:15 touareg kernel: Node 0 DMA32 free:1720kB min:1768kB low:2208kB high:2652kB active:699680kB inactive:588284kB present:1272420kB pages_scanned:2814597 all_unreclaimable? yes May 26 03:30:15 touareg kernel: lowmem_reserve[]: 0 0 4797 4797 May 26 03:30:15 touareg kernel: Node 0 Normal free:6116kB min:6836kB low:8544kB high:10252kB active:2214940kB inactive:2618664kB present:4912640kB pages_scanned:16399035 all_unreclaimable? yes May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:30:16 touareg kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:30:16 touareg kernel: Node 1 DMA free:1736kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10696kB pages_scanned:0 all_unreclaimable? yes May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 2004 2004 2004 May 26 03:30:16 touareg kernel: Node 1 DMA32 free:2776kB min:2856kB low:3568kB high:4284kB active:1303380kB inactive:650400kB present:2052320kB pages_scanned:4508965 all_unreclaimable? yes May 26 03:30:16 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:30:17 touareg kernel: Node 1 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:30:17 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:30:17 touareg kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no May 26 03:30:17 touareg kernel: lowmem_reserve[]: 0 0 0 0 May 26 03:30:17 touareg kernel: Node 0 DMA: empty May 26 03:30:17 touareg kernel: Node 0 DMA32: 0*4kB 5*8kB 3*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1720kB May 26 03:30:17 touareg kernel: Node 0 Normal: 497*4kB 44*8kB 40*16kB 20*32kB 9*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 6116kB May 26 03:30:18 touareg kernel: Node 0 HighMem: empty May 26 03:30:18 touareg kernel: Node 1 DMA: 2*4kB 2*8kB 1*16kB 5*32kB 4*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1736kB May 26 03:30:18 touareg kernel: Node 1 DMA32: 184*4kB 33*8kB 17*16kB 7*32kB 4*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2776kB May 26 03:30:18 touareg kernel: Node 1 Normal: empty May 26 03:30:18 touareg kernel: Node 1 HighMem: empty May 26 03:30:18 touareg kernel: 1379 pagecache pages May 26 03:30:18 touareg kernel: Swap cache: add 1053289, delete 1053103, find 5465453/5503467, race 0+41 May 26 03:30:18 touareg kernel: Free swap = 0kB May 26 03:30:19 touareg kernel: Total swap = 2097144kB May 26 03:30:19 touareg kernel: Free swap: 0kB May 26 03:30:19 touareg kernel: 2293760 pages of RAM May 26 03:30:19 touareg kernel: 251657 reserved pages May 26 03:30:19 touareg kernel: 5999 pages shared May 26 03:30:19 touareg kernel: 263 pages swap cached May 26 03:30:19 touareg kernel: Out of memory: Killed process 13328 (mysqld). May 26 03:30:19 touareg kernel: mysqld: page allocation failure. order:0, mode:0x201d2 先增加2G swap再说 #dd if=/dev/zero of=/opt/swapfile bs=1M count=2048 #mkswap /opt/swapfile #swapon /opt/swapfile #swapon -s Filename Type Size Used Priority /dev/mapper/VolGroup00-LogVol00 partition 2097144 87352 -1 /opt/swapfile file 2097144 0 -2 在fstab增加相关记录,否则重启就没了 #vi /etc/fstab /opt/swapfile swap swap defaults 0 0 参考 http://www.dbanotes.net/database/linux_outofmemory_oom_killer.html http://jk.scanmon.com/1228.html http://www.itjaj.com/thread-3029-1-1.html

Posted in linux 维护优化, 技术.

Tagged with .


No Responses (yet)

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.



Some HTML is OK

or, reply to this post via trackback.