[备忘] 磁盘空间不足分析

前提

环境说明

1.  操作系统: centos 7.2
2.  应用: docker 

现象:

1.  / 磁盘不足
2.  实际上磁盘还有很多空间
3.  应用开始报错( out of space)

分析

利用 find 分析, 没有找到特殊的大文件
如:

find /apps -size +50M 

利用 df 进行分析, 发现 / 已经用光

[root@hh-yun-k8s-128151 /]# df -ah
Filesystem      Size  Used Avail Use% Mounted on
rootfs             -     -     -    - /
sysfs              0     0     0    - /sys
proc               0     0     0    - /proc
devtmpfs         95G     0   95G   0% /dev
securityfs         0     0     0    - /sys/kernel/security
tmpfs            95G     0   95G   0% /dev/shm
devpts             0     0     0    - /dev/pts
tmpfs            95G  4.1G   91G   5% /run
tmpfs            95G     0   95G   0% /sys/fs/cgroup
cgroup             0     0     0    - /sys/fs/cgroup/systemd
pstore             0     0     0    - /sys/fs/pstore
cgroup             0     0     0    - /sys/fs/cgroup/cpuset
cgroup             0     0     0    - /sys/fs/cgroup/blkio
cgroup             0     0     0    - /sys/fs/cgroup/perf_event
cgroup             0     0     0    - /sys/fs/cgroup/cpu,cpuacct
cgroup             0     0     0    - /sys/fs/cgroup/hugetlb
cgroup             0     0     0    - /sys/fs/cgroup/devices
cgroup             0     0     0    - /sys/fs/cgroup/net_cls
cgroup             0     0     0    - /sys/fs/cgroup/freezer
cgroup             0     0     0    - /sys/fs/cgroup/memory
configfs           0     0     0    - /sys/kernel/config
/dev/sda2        80G   80G  286M 100% /
systemd-1          -     -     -    - /proc/sys/fs/binfmt_misc
debugfs            0     0     0    - /sys/kernel/debug
mqueue             0     0     0    - /dev/mqueue
hugetlbfs          0     0     0    - /dev/hugepages
/dev/sda1       397M  135M  263M  34% /boot
tmpfs            19G     0   19G   0% /run/user/0
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/fa7f705d-9a82-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/32d9731a-bdfc-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/c131bddd-c367-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/27779000-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/277f9d4b-c5cf-11e6-9bda-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/72505ec1-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/b50b4c1c-d570-11e6-b3fc-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
binfmt_misc        0     0     0    - /proc/sys/fs/binfmt_misc
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/29fa587e-ecdc-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/56e3e992-ed03-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/85342d9b-ece3-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/0e10352d-edbd-11e6-ae68-e09796534daa/volumes/kubernetes.io~secret/default-token-dqn5d
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/df078f2d-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/deab6a67-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/df71a3a2-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/df165896-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/df62fbed-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/ef590f38-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-ab33i
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/ffe50f62-ef83-11e6-878c-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/b0cacb34-f1af-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/025dc80b-f1b9-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/de7dc25d-f272-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cuzo2
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/b07a6d58-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k
tmpfs            95G   12K   95G   1% /var/lib/kubelet/pods/b084b8a9-f297-11e6-b535-e09796534daa/volumes/kubernetes.io~secret/default-token-cji2k

利用 df 进行分析, 实际上 / 只占用了少量空间 (4.4G)

[root@hh-yun-k8s-128151 /]# mount /dev/sda2 /mnt
[root@hh-yun-k8s-128151 /]# du -sh /mnt/
4.4G    /mnt/

结果

分析可能性

1.  df 与 du 显示不一致
2.  常见为进程正在占用文件, 而运维人员直接删除文件, 导致文件句柄没有释放
3.  上述第二点会导致 df 读取时候也把删除的文件的空间计算进去

获取进程占用而又被删除的文件方法

lsof | grep deleted > /tmp/files

查询结果

[root@hh-yun-k8s-128151 /]# head -n 30 /tmp/files
monitor     1389                  root    5w      REG                8,2        588  404101751 /var/log/openvswitch/ovsdb-server.log-20160802 (deleted)
monitor     1389                  root    7u      REG                8,2        141    1285894 /tmp/tmpfvWpBig (deleted)
ovsdb-ser   1391                  root    7u      REG                8,2        141    1285894 /tmp/tmpfvWpBig (deleted)
monitor     1438                  root    3w      REG                8,2       3478  404108801 /var/log/openvswitch/ovs-vswitchd.log-20160802 (deleted)
docker     56894                  root   12r      REG                8,2    2000037  269953568 /var/lib/docker/containers/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816/42a0c6add5d3b86037bef74d955fd968f4af853bf350ca1cb681094e0f60f816-json.log.4 (deleted)
docker     56894                  root   16r      REG                8,2    2000010  342880030 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   20r      REG                8,2    2000089  343187052 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   22r      REG                8,2    2000074  343187085 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   23r      REG                8,2    2000022  342328837 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   25r      REG                8,2    2000001  346233868 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   27r      REG                8,2    2000117  345268401 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   30r      REG                8,2    2000154    1770081 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   31r      REG                8,2    2001891    1761668 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   32r      REG                8,2    2000077    1761664 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   33r      REG                8,2    2001938    1762263 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   34r      REG                8,2    2000154    3736653 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   39r      REG                8,2    2000039  345268394 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   41r      REG                8,2    2000243  404129948 /var/lib/docker/containers/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b/a03a8e43513ea060224cf6658680157488c452d3c0096471e4a4bfd4d0e3394b-json.log.4 (deleted)
docker     56894                  root   42r      REG                8,2    2000044  345268392 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   43r      REG                8,2    2000138  270873518 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   44r      REG                8,2    2000109  342328852 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   49r      REG                8,2    2003399    2935891 /var/lib/docker/containers/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566/7d7a7df5fbf51763352fdb928ee1a1699b3b8b4311d37e40113196c32faf7566-json.log.4 (deleted)
docker     56894                  root   51r      REG                8,2    2000258  345268396 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   57r      REG                8,2    2000024  345177666 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   58r      REG                8,2    2000141  270873521 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   59r      REG                8,2    2000243  346233846 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   60r      REG                8,2    2000103  270873517 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   65r      REG                8,2    2000009  346233853 /var/lib/docker/containers/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612/ae79f532a9fdf40f5c7c4549a2b1703382cc31f409114325e4ca4a9aa1d9d612-json.log.4 (deleted)
docker     56894                  root   66r      REG                8,2    2001743  269761873 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted)
docker     56894                  root   67r      REG                8,2    2000473  269761869 /var/lib/docker/containers/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331/f1cfed05a1a6051b1e97e31ed708343bcbdfc9ed0d7baa6b267e45873915b331-json.log.4 (deleted)

总结

根据结果, 我们得到下面分析:

1. 当需要清除日志时候,  请使用 logrotate 等工具进行日志截断
2.  如果真的怕麻烦,  那么建议使用  cat /dev/null > file.log 的方法进行文件截断 (但文件多过也会占用 inode 空间,  可以利用  df -Th 方法查询 )
3.  建议使用标准日志格式, 利用 logstash 等工具进行远程备份与日志分析,  避免类似麻烦
4.  对于 docker 自身也具有 LogConfig 参数可以对日志进行回滚截断等控制
5.  由于容器的异常,  导致 k8s 回收容器, 而残留的线程会持续对容器 log 进行访问, 导致空间会越来越小,  这个问题无法避免,  参考下面解决方法

参考解决

参考磁盘空间

df 磁盘空间

[root@hh-yun-k8s-240006 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        80G   70G   11G  87% /

du 对应磁盘空间

[root@hh-yun-k8s-240006 ~]# mount /dev/sda2 /mnt
[root@hh-yun-k8s-240006 ~]# du -sh /mnt/
57G     /mnt/
[root@hh-yun-k8s-240006 ~]# umount /mnt

说明: 当前磁盘实际使用 57GB, 但包含了进程对文件句柄没有释放的计算, 就变成 70GB

释放文件句柄

[root@hh-yun-k8s-240006 ~]# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | xargs -n 1 -d '\n' -P 8 -I [] bash -c 'find . -inum `echo "[]" | sed -e "s/^[[:space:]]*//" | cut -d " "  -f 1`  -exec ls -l {} \; -exec  truncate -s 0 {} \;' > /dev/null

查询释放后的空间

[root@hh-yun-k8s-240006 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        80G   60G   22G  80% /
上一篇:正确看待递归函数


下一篇:删除指定表的所有索引,包括主键索引,唯一索引和普通索引 ,适用于sql server 2005 .