Dec :: vgfs001 kernel: tiotest_AMD_x86 invoked oom-killer: gfp_mask=0x200da, order=, oom_adj=, oom_score_adj=
Dec :: vgfs001 kernel: tiotest_AMD_x86 cpuset=/ mems_allowed=
Dec :: vgfs001 kernel: Pid: , comm: tiotest_AMD_x86 Not tainted 2.6.-431.29..lustre.el6.x86_64 #
Dec :: vgfs001 kernel: Call Trace:
Dec :: vgfs001 kernel: [<ffffffff810d07b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec :: vgfs001 kernel: [<ffffffff81122b80>] ? dump_header+0x90/0x1b0
Dec :: vgfs001 kernel: [<ffffffff8122894c>] ? security_real_capable_noaudit+0x3c/0x70
Dec :: vgfs001 kernel: [<ffffffff81123002>] ? oom_kill_process+0x82/0x2a0
Dec :: vgfs001 kernel: [<ffffffff81122f41>] ? select_bad_process+0xe1/0x120
Dec :: vgfs001 kernel: [<ffffffff81123440>] ? out_of_memory+0x220/0x3c0
Dec :: vgfs001 kernel: [<ffffffff8112fd5f>] ? __alloc_pages_nodemask+0x89f/0x8d0
Dec :: vgfs001 kernel: [<ffffffff81167cea>] ? alloc_pages_current+0xaa/0x110
Dec :: vgfs001 kernel: [<ffffffff8111ff77>] ? __page_cache_alloc+0x87/0x90
Dec :: vgfs001 kernel: [<ffffffff81120c8e>] ? grab_cache_page_write_begin+0x8e/0xc0
Dec :: vgfs001 kernel: [<ffffffffa0a8f228>] ? ll_write_begin+0x58/0x1a0 [lustre]
Dec :: vgfs001 kernel: [<ffffffff811204f3>] ? generic_file_buffered_write+0x123/0x2e0
Dec :: vgfs001 kernel: [<ffffffff81078fd7>] ? current_fs_time+0x27/0x30
Dec :: vgfs001 kernel: [<ffffffff81121f50>] ? __generic_file_aio_write+0x260/0x490
Dec :: vgfs001 kernel: [<ffffffffa05211a5>] ? cl_env_info+0x15/0x20 [obdclass]
Dec :: vgfs001 kernel: [<ffffffff81122208>] ? generic_file_aio_write+0x88/0x100
Dec :: vgfs001 kernel: [<ffffffffa0aa3907>] ? vvp_io_write_start+0x137/0x2a0 [lustre]
Dec :: vgfs001 kernel: [<ffffffffa05301da>] ? cl_io_start+0x6a/0x140 [obdclass]
Dec :: vgfs001 kernel: [<ffffffffa05348e4>] ? cl_io_loop+0xb4/0x1b0 [obdclass]
Dec :: vgfs001 kernel: [<ffffffffa0a46306>] ? ll_file_io_generic+0x2a6/0x610 [lustre]
Dec :: vgfs001 kernel: [<ffffffffa0a47192>] ? ll_file_aio_write+0x142/0x2c0 [lustre]
Dec :: vgfs001 kernel: [<ffffffffa0a4747c>] ? ll_file_write+0x16c/0x2a0 [lustre]
Dec :: vgfs001 kernel: [<ffffffff81189298>] ? vfs_write+0xb8/0x1a0
Dec :: vgfs001 kernel: [<ffffffff81189c61>] ? sys_write+0x51/0x90
Dec :: vgfs001 kernel: [<ffffffff810e204e>] ? __audit_syscall_exit+0x25e/0x290
Dec :: vgfs001 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Dec :: vgfs001 kernel: Mem-Info:
Dec :: vgfs001 kernel: Node DMA per-cpu:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: Node DMA32 per-cpu:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: Node Normal per-cpu:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: CPU : hi: , btch: usd:
Dec :: vgfs001 kernel: active_anon: inactive_anon: isolated_anon:
Dec :: vgfs001 kernel: active_file: inactive_file: isolated_file:
Dec :: vgfs001 kernel: unevictable: dirty: writeback: unstable:
Dec :: vgfs001 kernel: free: slab_reclaimable: slab_unreclaimable:
Dec :: vgfs001 kernel: mapped: shmem: pagetables: bounce:
Dec :: vgfs001 kernel: Node DMA free:15708kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15320kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? yes
Dec :: vgfs001 kernel: lowmem_reserve[]:
Dec :: vgfs001 kernel: Node DMA32 free:53892kB min:19596kB low:24492kB high:29392kB active_anon:4kB inactive_anon:44kB active_file:1249260kB inactive_file:1249288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596496kB mlocked:0kB dirty:3436kB writeback:4180kB mapped:0kB shmem:0kB slab_reclaimable:24608kB slab_unreclaimable:689432kB kernel_stack:8kB pagetables:196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? no
Dec :: vgfs001 kernel: lowmem_reserve[]:
Dec :: vgfs001 kernel: Node Normal free:47588kB min:47900kB low:59872kB high:71848kB active_anon:4792020kB inactive_anon:685556kB active_file:943652kB inactive_file:944700kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:8791040kB mlocked:0kB dirty:160kB writeback:5188kB mapped:4128kB shmem:0kB slab_reclaimable:17948kB slab_unreclaimable:816972kB kernel_stack:5040kB pagetables:22256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? no
Dec :: vgfs001 kernel: lowmem_reserve[]:
Dec :: vgfs001 kernel: Node DMA: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 15708kB
Dec :: vgfs001 kernel: Node DMA32: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 53892kB
Dec :: vgfs001 kernel: Node Normal: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 47084kB
Dec :: vgfs001 kernel: total pagecache pages
Dec :: vgfs001 kernel: pages in swap cache
Dec :: vgfs001 kernel: Swap cache stats: add , delete , find /
Dec :: vgfs001 kernel: Free swap = 3377416kB
Dec :: vgfs001 kernel: Total swap = 4194300kB
Dec :: vgfs001 kernel: pages RAM
Dec :: vgfs001 kernel: pages reserved
Dec :: vgfs001 kernel: pages shared
Dec :: vgfs001 kernel: pages non-shared
Dec :: vgfs001 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Dec :: vgfs001 kernel: [ ] - - udevd
Dec :: vgfs001 kernel: [ ] - - auditd
Dec :: vgfs001 kernel: [ ] rsyslogd
Dec :: vgfs001 kernel: [ ] irqbalance
Dec :: vgfs001 kernel: [ ] rpcbind
Dec :: vgfs001 kernel: [ ] dbus-daemon
Dec :: vgfs001 kernel: [ ] rpc.statd
Dec :: vgfs001 kernel: [ ] cupsd
Dec :: vgfs001 kernel: [ ] acpid
Dec :: vgfs001 kernel: [ ] hald
Dec :: vgfs001 kernel: [ ] hald-runner
Dec :: vgfs001 kernel: [ ] hald-addon-inpu
Dec :: vgfs001 kernel: [ ] hald-addon-acpi
Dec :: vgfs001 kernel: [ ] automount
Dec :: vgfs001 kernel: [ ] - - sshd
Dec :: vgfs001 kernel: [ ] master
Dec :: vgfs001 kernel: [ ] qmgr
Dec :: vgfs001 kernel: [ ] abrtd
Dec :: vgfs001 kernel: [ ] ksmtuned
Dec :: vgfs001 kernel: [ ] crond
Dec :: vgfs001 kernel: [ ] atd
Dec :: vgfs001 kernel: [ ] python
Dec :: vgfs001 kernel: [ ] libvirtd
Dec :: vgfs001 kernel: [ ] rhsmcertd
Dec :: vgfs001 kernel: [ ] dnsmasq
Dec :: vgfs001 kernel: [ ] certmonger
Dec :: vgfs001 kernel: [ ] gdm-binary
Dec :: vgfs001 kernel: [ ] mingetty
Dec :: vgfs001 kernel: [ ] mingetty
Dec :: vgfs001 kernel: [ ] mingetty
Dec :: vgfs001 kernel: [ ] mingetty
Dec :: vgfs001 kernel: [ ] mingetty
Dec :: vgfs001 kernel: [ ] - - udevd
Dec :: vgfs001 kernel: [ ] - - udevd
Dec :: vgfs001 kernel: [ ] gdm-simple-slav
Dec :: vgfs001 kernel: [ ] Xorg
Dec :: vgfs001 kernel: [ ] console-kit-dae
Dec :: vgfs001 kernel: [ ] dbus-launch
Dec :: vgfs001 kernel: [ ] dbus-daemon
Dec :: vgfs001 kernel: [ ] gnome-session
Dec :: vgfs001 kernel: [ ] devkit-power-da
Dec :: vgfs001 kernel: [ ] gconfd-
Dec :: vgfs001 kernel: [ ] gnome-settings-
Dec :: vgfs001 kernel: [ ] at-spi-registry
Dec :: vgfs001 kernel: [ ] bonobo-activati
Dec :: vgfs001 kernel: [ ] gvfsd
Dec :: vgfs001 kernel: [ ] metacity
Dec :: vgfs001 kernel: [ ] gnome-power-man
Dec :: vgfs001 kernel: [ ] polkit-gnome-au
Dec :: vgfs001 kernel: [ ] gdm-simple-gree
Dec :: vgfs001 kernel: [ ] polkitd
Dec :: vgfs001 kernel: [ ] pulseaudio
Dec :: vgfs001 kernel: [ ] rtkit-daemon
Dec :: vgfs001 kernel: [ ] gdm-session-wor
Dec :: vgfs001 kernel: [] sshd
Dec :: vgfs001 kernel: [] bash
Dec :: vgfs001 kernel: [ ] lwfsd
Dec :: vgfs001 kernel: [ ] pickup
Dec :: vgfs001 kernel: [ ] sleep
Dec :: vgfs001 kernel: [ ] tiotest_AMD_x86
Dec :: vgfs001 kernel: Out of memory: Kill process (lwfsd) score or sacrifice child
Dec :: vgfs001 kernel: Killed process , UID , (lwfsd) total-vm:6846436kB, anon-rss:4742528kB, file-rss:20040kB
这里是从Lustre的入口导致的oom，但实际上，其他入口例如KVM管理程序也可能引起oom，即任何分配内存的可能点都可能引起oom。
从分析过程来看，确实是Lustre的Cache占用了大量内存，导致内存分配不足。
三个措施。
1、增大内存
从12GB增大到16GB。
virsh setmaxmem vgfsxxx 16GB --config
运行启动后
virsh setmem vgfsxxx 16GB
这个没有用，跑了几次测试后，仍然掉服务。
2、调整lwfsd的服务优先级
设置lwfsd的服务优先级为“-17”
PID=`ps | grep lwfs | grep -v grep | awk '{print $1}'`
echo -17 > /proc/$PID/oom_adj
echo -17 > /proc/$PID/task/$PID/oom_adj
这个好像有用。
3、修改内存分配策略
并且echo "2" >/proc/sys/vm/overcommit_memory，使得分配内存时，必须存在足够的空间用于映射。
这个好像也有一定的用处。再跑跑试试。