前提
环境说明
1. 操作系统: centos 7.2
2. 应用: docker
现象:
1. / 磁盘不足
2. 实际上磁盘还有很多空间
3. 应用开始报错( out of space)
分析
利用 find 分析, 没有找到特殊的大文件
如:
find /apps -size +50M
利用 df 进行分析, 发现 / 已经用光
[root@hh-yun-k8s-128151 /]# df -ah
Filesystem Size Used Avail Use% Mounted on
rootfs - - - - /
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
devtmpfs 95G 0 95G 0% /dev
securityfs 0 0 0 - /sys/kernel/security
tmpfs 95G 0 95G 0% /dev/shm
devpts 0 0 0 - /dev/pts
tmpfs 95G 4.1G 91G 5% /run
tmpfs 95G 0 95G 0% /sys/fs/cgroup
cgroup 0 0 0 - /sys/fs/cgroup/systemd
pstore 0 0 0 - /sys/fs/pstore
cgroup 0 0 0 - /sys/fs/cgroup/cpuset
cgroup 0 0 0 - /sys/fs/cgroup/blkio
cgroup 0 0 0 - /sys/fs/cgroup/perf_event
cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct
cgroup 0 0 0 - /sys/fs/cgroup/hugetlb
cgroup 0 0 0 - /sys/fs/cgroup/devices
cgroup 0 0 0 - /sys/fs/cgroup/net_cls
cgroup 0 0 0 - /sys/fs/cgroup/freezer
cgroup 0 0 0 - /sys/fs/cgroup/memory
configfs 0 0 0 - /sys/kernel/config
/dev/sda2 80G 80G 286M 100% /
systemd-1 - - - - /proc/sys/fs/binfmt_misc
debugfs 0 0 0 - /sys/kernel/debug
mqueue 0 0 0 - /dev/mqueue
hugetlbfs 0 0 0 - /dev/hugepages
/dev/sda1 397M 135M 263M 34% /boot
tmpfs 19G 0 19G 0% /run/user/0
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/fa7f705d-9a82-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/32d9731a-bdfc-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/c131bddd-c367-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/27779000-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/277f9d4b-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/72505ec1-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b50b4c1c-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/29fa587e-ecdc-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/56e3e992-ed03-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/85342d9b-ece3-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/0e10352d-edbd-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df078f2d-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/deab6a67-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df71a3a2-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df165896-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/df62fbed-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/ef590f38-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/ffe50f62-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b0cacb34-f1af-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/025dc80b-f1b9-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/de7dc25d-f272-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b07a6d58-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k
tmpfs 95G 12K 95G 1% /var/lib/kubelet/pods/b084b8a9-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k
利用 df 进行分析, 实际上 / 只占用了少量空间 (4.4G)
[root@hh-yun-k8s-128151 /]# mount /dev/sda2 /mnt
[root@hh-yun-k8s-128151 /]# du -sh /mnt/
4.4G /mnt/
结果
分析可能性
1. df 与 du 显示不一致
2. 常见为进程正在占用文件, 而运维人员直接删除文件, 导致文件句柄没有释放
3. 上述第二点会导致 df 读取时候也把删除的文件的空间计算进去
获取进程占用而又被删除的文件方法
lsof | grep deleted > /tmp/files
查询结果
[root@hh-yun-k8s-128151 /]# head -n 30 /tmp/files
monitor 1389 root 5w REG 8,2 588 404101751 /var/log/openvswitch/ovsdb-server.log-20160802 (deleted)
monitor 1389 root 7u REG 8,2 141 1285894 /tmp/tmpfvWpBig (deleted)
ovsdb-ser 1391 root 7u REG 8,2 141 1285894 /tmp/tmpfvWpBig (deleted)
monitor 1438 root 3w REG 8,2 3478 404108801 /var/log/openvswitch/ovs-vswitchd.log-20160802 (deleted)
docker 56894 root 12r REG 8,2 2000037 269953568 /var/lib/docker/containers/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816-json.log.4 (deleted)
docker 56894 root 16r REG 8,2 2000010 342880030 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 20r REG 8,2 2000089 343187052 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 22r REG 8,2 2000074 343187085 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 23r REG 8,2 2000022 342328837 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 25r REG 8,2 2000001 346233868 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 27r REG 8,2 2000117 345268401 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 30r REG 8,2 2000154 1770081 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 31r REG 8,2 2001891 1761668 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 32r REG 8,2 2000077 1761664 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 33r REG 8,2 2001938 1762263 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 34r REG 8,2 2000154 3736653 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 39r REG 8,2 2000039 345268394 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 41r REG 8,2 2000243 404129948 /var/lib/docker/containers/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b-json.log.4 (deleted)
docker 56894 root 42r REG 8,2 2000044 345268392 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 43r REG 8,2 2000138 270873518 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 44r REG 8,2 2000109 342328852 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 49r REG 8,2 2003399 2935891 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker 56894 root 51r REG 8,2 2000258 345268396 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 57r REG 8,2 2000024 345177666 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 58r REG 8,2 2000141 270873521 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 59r REG 8,2 2000243 346233846 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 60r REG 8,2 2000103 270873517 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 65r REG 8,2 2000009 346233853 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker 56894 root 66r REG 8,2 2001743 269761873 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted)
docker 56894 root 67r REG 8,2 2000473 269761869 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted)
总结
根据结果, 我们得到下面分析:
1. 当需要清除日志时候, 请使用 logrotate 等工具进行日志截断
2. 如果真的怕麻烦, 那么建议使用 cat /dev/null > file.log 的方法进行文件截断 (但文件多过也会占用 inode 空间, 可以利用 df -Th 方法查询 )
3. 建议使用标准日志格式, 利用 logstash 等工具进行远程备份与日志分析, 避免类似麻烦
4. 对于 docker 自身也具有 LogConfig 参数可以对日志进行回滚截断等控制
5. 由于容器的异常, 导致 k8s 回收容器, 而残留的线程会持续对容器 log 进行访问, 导致空间会越来越小, 这个问题无法避免, 参考下面解决方法
参考解决
参考磁盘空间
df 磁盘空间
[root@hh-yun-k8s-240006 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 80G 70G 11G 87% /
du 对应磁盘空间
[root@hh-yun-k8s-240006 ~]# mount /dev/sda2 /mnt
[root@hh-yun-k8s-240006 ~]# du -sh /mnt/
57G /mnt/
[root@hh-yun-k8s-240006 ~]# umount /mnt
说明: 当前磁盘实际使用 57GB, 但包含了进程对文件句柄没有释放的计算, 就变成 70GB
释放文件句柄
[root@hh-yun-k8s-240006 ~]# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | xargs -n 1 -d '\n' -P 8 -I [] bash -c 'find . -inum `echo "[]" | sed -e "s/^[[:space:]]*//" | cut -d " " -f 1` -exec ls -l {} \; -exec truncate -s 0 {} \;' > /dev/null
查询释放后的空间
[root@hh-yun-k8s-240006 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 80G 60G 22G 80% /