昨天晚上zabbix服务器报一台主机的磁盘空间超过利用率，恰好我电脑开着，所以登录上去看了下，排查过程如下：
因为知道这台服务器的具体业务功能，正常情况下是不会立即达到监控线的，通过top查看了下进程，发现sytemd-journal这个进程，怀疑日志文件被大量的写入。
systemd-journald是一个改进型日志管理服务，可以收集来自内核、系统早期启动阶段的日志、系统守护进程在启动和运行中的标准输出和错误信息，还有syslog的日志。该日志服务仅仅把日志集中保存在单一结构的日志文件/run/log中，由于日志是经历过压缩和格式化的二进制数据，所以在查看和定位的时候很迅速。默认情况下并不会持久化保存日志，只会保留一个月的日志。另外，一些rsyslog无法收集的日志也会被journal记录到.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
[root@ops-monitor-01 /var/log/journal]
# du -sh /var/log/journal/1.5G /var/log/journal/
说明:果然，这个目录比之前多了很多东西，本来通过删除清理下里面的日志，突然想到journactl有个高级的功能，可以做，于是有了下面的操作[root@ops-monitor-01 /var/log/journal]
# journalctl --vacuum-size=500MDeleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000000001-000550f7deac7403.journal (64.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000047fb-000551307d518b3e.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000013fcd-0005535c3692ab03.journal (48.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000140be-0005535fdff3ea1d.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000001f484-000555c082ebcdf7.journal (64.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000001fb4d-000555d51cf51bc6.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000315f3-00055824d9a1b26c.journal (96.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000032b7f-0005585387ff50f0.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-000000000004ac87-00055a8926aba5d2.journal (104.1M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-0000000000051652-00055b1a3b4b15ab.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000069762-00055ced73056d1d.journal (112.1M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000006a14c-00055cf8d0f034cb.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-0000000000089211-00055e2ef50709d8.journal (128.1M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-000000000008b23a-00055e319abb77e8.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000bc349-00055e8eb47900a7.journal (128.1M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/user-1000@0b70f7779047455090ed3bdeb41372e9-00000000000d57a3-00055e8ef8591843.journal (8.0M).
Deleted archived journal /var/log/journal/3c6ff97c6fbe4598b53fd04e08937468/system@ef03f01eefec47f5ada60bd1dba2ab24-00000000000f99cd-00055e8f57466708.journal (128.1M).
Vacuuming done, freed 937.0M of archived journals on disk.
[root@ops-monitor-01 /var/log/journal]
# du -sh /var/log/journal/513M /var/log/journal/
[root@ops-monitor-01 /var/log/journal]
# df -hFilesystem Size Used Avail Use% Mounted on/dev/vda1 99G 76G 18G 82% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 100K 1.9G 1% /dev/shm
tmpfs 1.9G 488K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vdb 197G 5.9G 181G 4% /war
/dev/vdc1 100G 60G 41G 60% /mnt/yum
tmpfs 380M 0 380M 0% /run/user/1000
|
到此，问题已经接近，需要做进一步了解的朋友可以查看下journalctl的帮助命令，相信会有收获。
本文转自 冰冻vs西瓜 51CTO博客，原文链接：http://blog.51cto.com/molewan/2043336，如需转载请自行联系原作者