df -h

Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg-lv_root 118G 13G 100G 12% / tmpfs 16G 4.0K 16G 1% /dev/shm /dev/sda1 477M 42M 411M 10% /boot /dev/mapper/vg-lv_app 318G 138G 164G 46% /opt /dev/mapper/vg-lv_var 99G 80G 14G 86% /var /usr/Tmp 9.4G 158M 8.8G 2% /tmp

/var/使用了80G,小于15%的空闲.

cd /var

du -sh .

490M .
.0K ./empty
4.0K ./yp
4.0K ./tmp
5.5M ./spool
4.0K ./account
4.0K ./crash
192K ./run
193M ./cache
4.0K ./local
4.0K ./agentx
16K ./lost+found
4.0K ./nis
16K ./lock
4.0K ./games
32K ./db
172M ./lib
120M ./log
4.0K ./preserve
4.0K ./opt
490M .

进入/var 看一下实际才使用490M.再查一下inode,也没问题.

df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg-lv_root
7864320 122554 7741766 2% /
tmpfs 4093515 2 4093513 1% /dev/shm
/dev/sda1 128016 39 127977 1% /boot
/dev/mapper/vg-lv_app
21135360 6768994 14366366 33% /opt
/dev/mapper/vg-lv_var
6553600 3068 6550532 1% /var
/usr/Tmp 625856 689 625167 1% /tmp

查看下进程是否有占用

lsof | grep delete

php-fpm 1155 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
php-fpm 2289 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
syslog-ng 4913 root 9w REG 253,3 84811572283 1310862 /var/log/nginx/blog.c1gstudio.com.20180726.log (deleted)
php-fpm 5478 root 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
php-fpm 5700 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
php-fpm 6438 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
php-fpm 11638 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)
php-fpm 11823 www 3u REG 7,0 0 33 /tmp/ZCUDxFkJdQ (deleted)

看到 syslog-ng 4913 一直在写已删除文件.
kill 4913
再重启下进程,就可以看到恢复了.

分析下原因是日志打包脚本中忘记reoload syslog-ng,修复!
/bin/kill -USR1 cat /usr/local/syslog-ng/var/syslog-ng.pid