docker container启动失败，报错：Exited (137) *** ago，比如
Exited (137) 16 seconds ago
这时通过docker logs查不到任何日志，从mesos上看stderr相关的只有一句
I0409 16:56:26.408077 8583 executor.cpp:736] Container exited with status 137
通过docker inspect查看container状态为
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": true,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2019-04-09T08:50:48.058583459Z",
"FinishedAt": "2019-04-09T08:50:55.456317695Z"
},
可见是因为OOMKilled，通过journalctl查看oom日志如下：
# journalctl -k | grep -i -e memory -e oom
Apr 09 16:00:14 cdp-test-server-05.bj kernel: java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Apr 09 16:00:14 cdp-test-server-05.bj kernel: [<ffffffff8c3ba524>] oom_kill_process+0x254/0x3d0
Apr 09 16:00:14 cdp-test-server-05.bj kernel: [<ffffffff8c435346>] mem_cgroup_oom_synchronize+0x546/0x570
Apr 09 16:00:14 cdp-test-server-05.bj kernel: [<ffffffff8c3badb4>] pagefault_out_of_memory+0x14/0x90
Apr 09 16:00:14 cdp-test-server-05.bj kernel: memory: usage 524288kB, limit 524288kB, failcnt 8430
Apr 09 16:00:14 cdp-test-server-05.bj kernel: memory+swap: usage 524288kB, limit 1048576kB, failcnt 0
Apr 09 16:00:14 cdp-test-server-05.bj kernel: Memory cgroup stats for /docker/3aafdee2b919fa936815fbb88ebd8bb3131c185690284491f583c62ff382b1fe: cache:20KB rss:524268KB rss_huge:0KB mapped_file:8KB swap:0KB inactive_anon:0KB active_anon:524236KB inactive_file:8KB active_file:8KB unevictable:0KB
Apr 09 16:00:14 cdp-test-server-05.bj kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 09 16:00:14 cdp-test-server-05.bj kernel: Memory cgroup out of memory: Kill process 10768 (java) score 1021 or sacrifice child
原因是container只分配了512M，但是需要1G（比如配置文件里设置-Xms1G -Xmx1G）
参考：https://success.docker.com/article/what-causes-a-container-to-exit-with-code-137