一、问题起源
在Linux系统中，通过rm删除文件将会从文件系统的目录结构上解除链接(unlink)，如果文件是被打开的（有一个进程正在使用），那么进程将仍然可以读取该文件磁盘空间也一直被占用
这样就会导致我们明明删除了文件，但是磁盘空间却未被释放
二、问题分析
1、首先获得一个已经被删除但是仍然被应用程序占用的文件列表
root@instance-zo0fkfng:~# lsof|grep deleted
init 1 root 10w REG 253,1 283 155958 /var/log/upstart/network-manager.log.1 (deleted)
init 1 root 12w REG 253,1 314 155934 /var/log/upstart/mountall.log.1 (deleted)
init 1 root 13w REG 253,1 508 131203 /var/log/upstart/modemmanager.log.1 (deleted)
init 1 root 17w REG 253,1 881 155967 /var/log/upstart/mysql.log.1 (deleted)
init 1 root 24w REG 253,1 63 155949 /var/log/upstart/dbus.log.1 (deleted)
mysqld 1159 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
mysqld 1159 1185 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 17965 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 17965 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 17965 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 17965 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 17965 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
php5-fpm 3724 www-data 3ur REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 3853 www-data 3ur REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 7741 root 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9900 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9901 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9930 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9961 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9965 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 9966 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10126 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10127 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10128 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10174 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10196 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10248 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10316 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10375 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10396 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10400 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10401 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10438 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10479 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10511 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 10531 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 11150 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 11638 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 11670 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 12210 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 14974 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 17101 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 17161 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 17261 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
php5-fpm 17470 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted)
从输出结果可以看到哪些文件还被使用，未被释放空间
或者
root@instance-zo0fkfng:~# find /proc/*/fd -ls | grep '(deleted)'
71253 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/4 -> /tmp/ibkjeh3Z\ (deleted)
71254 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/5 -> /tmp/ibdbHetZ\ (deleted)
71255 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/6 -> /tmp/ibQSxcTY\ (deleted)
71256 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/7 -> /tmp/ibxSgZNX\ (deleted)
71260 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/11 -> /tmp/ibXTzqnX\ (deleted)
221991074 0 lrwx------ 1 root root 64 Aug 27 19:43 /proc/18132/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991101 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18135/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991122 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18136/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991147 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18137/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991168 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18138/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991189 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18139/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991210 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18140/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991231 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18141/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991252 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18142/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991273 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18143/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991294 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18144/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
221991315 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18147/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
222001888 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18157/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
222001909 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18161/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
222001938 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18170/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
222001959 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18178/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted)
find: `/proc/18182/fd/5': No such file or directory
55130 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/10 -> /var/log/upstart/network-manager.log.1\ (deleted)
55132 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/12 -> /var/log/upstart/mountall.log.1\ (deleted)
55133 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/13 -> /var/log/upstart/modemmanager.log.1\ (deleted)
55137 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/17 -> /var/log/upstart/mysql.log.1\ (deleted)
55142 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/24 -> /var/log/upstart/dbus.log.1\ (deleted)
find: `/proc/self/fd/5': No such file or directory
2、如何让进程释放文件，进而释放磁盘空间
kill掉相应的进程或者重启该进程，让os自动回收磁盘空间
当linux打开一个文件的时候,Linux内核会为每一个进程在/proc/ 『/proc/nnnn/fd/目录（nnnn为pid）』建立一个以其pid为名的目录用来保存进程的相关信息，而其子目录fd保存的是该进程打开的所有文件的fd（fd：file descriptor）
kill进程是通过截断proc文件系统中的文件可以强制要求系统回收分配给正在使用的的文件
root@instance-zo0fkfng:~# service php5-fpm restart
root@instance-zo0fkfng:~# lsof|grep deleted
init 1 root 10w REG 253,1 283 155958 /var/log/upstart/network-manager.log.1 (deleted)
init 1 root 12w REG 253,1 314 155934 /var/log/upstart/mountall.log.1 (deleted)
init 1 root 13w REG 253,1 508 131203 /var/log/upstart/modemmanager.log.1 (deleted)
init 1 root 17w REG 253,1 881 155967 /var/log/upstart/mysql.log.1 (deleted)
init 1 root 24w REG 253,1 63 155949 /var/log/upstart/dbus.log.1 (deleted)
mysqld 1159 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
mysqld 1159 1185 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 1185 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 1185 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 1185 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 1185 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
mysqld 1159 1186 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 1186 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 18149 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 18149 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 18149 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 18149 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 18149 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
mysqld 1159 18150 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted)
mysqld 1159 18150 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted)
mysqld 1159 18150 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted)
mysqld 1159 18150 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted)
mysqld 1159 18150 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted)
php5-fpm 18132 root 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18135 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18136 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18137 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18138 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18139 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18140 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18141 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18142 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18143 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18144 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
php5-fpm 18147 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted)
可以看到php5-fpm变少了
还有一种方式，清除文件
: > /path/to/the/file.log
: > "/proc/$pid/fd/$fd"
三、文件恢复
在说明问题之前，先介绍下一些文件的基本概念：
文件实际上是一个指向inode的链接, inode链接包含了文件的所有属性, 比如权限和所有者, 数据块地址(文件存储在磁盘的这些数据块中). 当你删除(rm)一个文件, 实际删除了指向inode的链接, 并没有删除inode的内容. 进程可能还在使用. 只有当inode的所有链接完全移去, 然后这些数据块将可以写入新的数据.
proc文件系统可以协助我们恢复数据. 每一个系统上的进程在/proc都有一个目录和自己的名字， 里面包含了一个fd(文件描述符)子目录(进程需要打开文件的所有链接). 如果从文件系统中删除一个文件, 此处还有一个inode的引用 ：
/proc/进程号/fd/文件描述符
你需要知道打开文件的进程号(pid)和文件描述符(fd). 这些都可以通过lsof工具方便获得, lsof的意思是”list open files, 列出(进程)打开的文件”. 然后你将可以从/proc拷贝出需要恢复的数据.
1.创建一个测试文件并且备份下，方面后续验证
touch testfile
cp testfile testfile.backup.2014
2.查看文件的相关信息
stat testfile File: 'testfile'
Size: 343545 Blocks: 241 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 361579 Links: 1
Access: (0664/-rw-rw-r–) Uid: ( 505/ zhaoke) Gid: ( 505/ zhaoke)
Access: 2014-11-09 15:00:38.000000000 +0800
Modify: 2014-11-09 15:00:34.000000000 +0800
Change: 2014-04-09 15:00:34.000000000 +0800
3.删除文件
rm testfile
4.查看文件
ls -l testfile
ls: testfile: No such file or directory
stat testfile
stat: cannot stat 'testfile': No such file or directory
testfile文件删除了，但不要终止仍在使用文件的进程， 因为一旦终止, 文件将很难恢复.
现在我们开始找回数据之旅，先使用lsof命令查看下
lsof | grep testfile
tail 5317 root 4r REG 253,0 343545 361579 /root/testfile (deleted)
第一个纵行是进程的名称(命令名), 第二纵行是进程号(PID), 第四纵行是文件描述符
现在你知道5317进程仍有打开文件, 文件描述符是4. 那我们开始从/proc里面拷贝出数据.
你可能会考虑使用cp -a, 但实际上没有作用, 你将拷贝的是一个指向被删除文件的符号链接:
ls -l /proc/5317/fd/4
lr-x—— 1 root root 64 09 15:00 /proc/5317/fd/4 -> /root/testfile (deleted)
使用cp -a命令测试恢复
cp -a /proc/5317/fd/4 testfile.backup
使用ls命令来查看
ls -l testfile.backup
lrwxrwxrwx 1 root root 29 09 15:02 testfile.backup -> /roor/testfile (deleted)
通过上面的命令我们发现，使用cp -a命令，其恢复的是一个指向被删除文件的符号链接
使用file命令分别查看文件和文件描述符
1.查看文件
file testfile.backup
testfile.backup: broken symbolic link to '/root/testfile (deleted)'
2.查看文件描述符
file /proc/5317/fd/4 /proc/5317/fd/4: broken symbolic link to '/root/myfile (deleted)'
根据上面的file结果，可以使用cp拷贝出文件描述符数据到一个文件中，如下:
cp /proc/5317/fd/4 testfile.new
使用上面的命令恢复后，我们需要最终确认一下文件是否恢复，以及文件内容是否正确:
ls -l testfile.new
然后把新旧的两个文件对比
diff testfile.new myfile.backup
4、lsof的一些其他用法
1、查看端口号
root@localhost:~# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
AliYunDun 1996 root 11u IPv4 117919702 0t0 TCP 124.46.185.34:42372->140.205.140.205:http (ESTABLISHED)
aegis_qua 2114 root 11u IPv4 14868 0t0 TCP 124.46.185.34:43077->42.156.166.25:http (CLOSE_WAIT)
nginx 13706 www-data 5u IPv4 140102705 0t0 TCP 124.46.185.34:http->140.205.127.64:41851 (ESTABLISHED)
nginx 13706 www-data 15u IPv4 6150826 0t0 TCP *:http (LISTEN)
nginx 13706 www-data 35u IPv4 140096235 0t0 TCP 124.46.185.34:http->198.11.137.21:47450 (ESTABLISHED)
nginx 13706 www-data 36u IPv4 140095680 0t0 TCP 124.46.185.34:http->140.205.127.52:39887 (ESTABLISHED)
nginx 13706 www-data 39u IPv4 140102712 0t0 TCP 124.46.185.34:http->198.11.137.5:37626 (ESTABLISHED)
nginx 13706 www-data 40u IPv4 140102699 0t0 TCP 124.46.185.34:http->140.205.127.11:36374 (ESTABLISHED)
nginx 13706 www-data 41u IPv4 140102759 0t0 TCP 124.46.185.34:http->101.200.101.217:47550 (ESTABLISHED)
nginx 13706 www-data 42u IPv4 140102722 0t0 TCP 124.46.185.34:http->101.200.101.201:38595 (ESTABLISHED)
nginx 13706 www-data 43u IPv4 140102840 0t0 TCP 124.46.185.34:http->140.205.127.58:45859 (ESTABLISHED)
nginx 13706 www-data 44u IPv4 140095709 0t0 TCP 124.46.185.34:http->101.200.101.216:44828 (ESTABLISHED)
nginx 13706 www-data 46u IPv4 140070070 0t0 TCP 124.46.185.34:http->140.205.127.18:24113 (ESTABLISHED)
nginx 13706 www-data 47u IPv4 140103294 0t0 TCP 124.46.185.34:http->198.11.137.61:43262 (ESTABLISHED)
nginx 13706 www-data 48u IPv4 140102761 0t0 TCP 124.46.185.34:http->101.200.101.220:31912 (ESTABLISHED)
nginx 13706 www-data 50u IPv4 140103297 0t0 TCP 124.46.185.34:http->101.200.101.203:42752 (ESTABLISHED)
nginx 13706 www-data 52u IPv4 140095824 0t0 TCP 124.46.185.34:http->198.11.137.14:62968 (ESTABLISHED)
nginx 13706 www-data 53u IPv4 140102730 0t0 TCP 124.46.185.34:http->101.200.101.205:27518 (ESTABLISHED)
nginx 13706 www-data 54u IPv4 140102822 0t0 TCP 124.46.185.34:http->140.205.127.51:58945 (ESTABLISHED)
nginx 13706 www-data 55u IPv4 140103313 0t0 TCP 124.46.185.34:http->140.205.127.15:20785 (ESTABLISHED)
nginx 13706 www-data 56u IPv4 140102812 0t0 TCP 124.46.185.34:http->140.205.31.212:1963 (ESTABLISHED)
nginx 13706 www-data 57u IPv4 140102929 0t0 TCP 124.46.185.34:http->101.200.101.213:42452 (ESTABLISHED)
nginx 13706 www-data 58u IPv4 140095749 0t0 TCP 124.46.185.34:http->101.200.101.208:50444 (ESTABLISHED)
nginx 13706 www-data 59u IPv4 140102769 0t0 TCP 124.46.185.34:http->140.205.127.60:59679 (ESTABLISHED)
nginx 13706 www-data 60u IPv4 140102814 0t0 TCP 124.46.185.34:http->140.205.127.2:25156 (ESTABLISHED)
nginx 13706 www-data 61u IPv4 140102771 0t0 TCP 124.46.185.34:http->198.11.137.53:16935 (ESTABLISHED)
nginx 13706 www-data 62u IPv4 140102845 0t0 TCP 124.46.185.34:http->198.11.137.2:50535 (ESTABLISHED)
nginx 13706 www-data 63u IPv4 140102773 0t0 TCP 124.46.185.34:http->198.11.137.52:8521 (ESTABLISHED)
nginx 13706 www-data 64u IPv4 140102856 0t0 TCP 124.46.185.34:http->140.205.127.23:55360 (ESTABLISHED)
nginx 13706 www-data 66u IPv4 140075271 0t0 TCP 124.46.185.34:http->140.205.127.48:64726 (ESTABLISHED)
nginx 13706 www-data 67u IPv4 140102937 0t0 TCP 124.46.185.34:http->101.200.101.204:10400 (ESTABLISHED)
nginx 13706 www-data 68u IPv4 140102932 0t0 TCP 124.46.185.34:http->101.200.101.213:42455 (ESTABLISHED)
nginx 13706 www-data 69u IPv4 140102941 0t0 TCP 124.46.185.34:http->140.205.127.24:31585 (ESTABLISHED)
nginx 13706 www-data 70u IPv4 140102934 0t0 TCP 124.46.185.34:http->198.11.137.50:11837 (ESTABLISHED)
nginx 13706 www-data 71u IPv4 140103309 0t0 TCP 124.46.185.34:http->140.205.127.15:20777 (ESTABLISHED)
nginx 13706 www-data 92u IPv4 140070742 0t0 TCP 124.46.185.34:http->140.205.127.31:65012 (ESTABLISHED)
nginx 13707 www-data 5u IPv4 140092233 0t0 TCP 124.46.185.34:http->140.205.127.60:57698 (ESTABLISHED)
nginx 13707 www-data 15u IPv4 6150826 0t0 TCP *:http (LISTEN)
nginx 13707 www-data 17u IPv4 140092036 0t0 TCP 124.46.185.34:http->140.205.127.51:54339 (ESTABLISHED)
nginx 13707 www-data 30u IPv4 140092031 0t0 TCP 124.46.185.34:http->198.11.137.59:64831 (ESTABLISHED)
nginx 13707 www-data 35u IPv4 140092214 0t0 TCP 124.46.185.34:http->140.205.31.213:40510 (ESTABLISHED)
nginx 13707 www-data 39u IPv4 140092202 0t0 TCP 124.46.185.34:http->140.205.127.9:19592 (ESTABLISHED)
nginx 13707 www-data 44u IPv4 140092205 0t0 TCP 124.46.185.34:http->140.205.127.7:24059 (ESTABLISHED)
nginx 13707 www-data 46u IPv4 140103741 0t0 TCP 124.46.185.34:http->198.11.137.22:61619 (ESTABLISHED)
nginx 13707 www-data 51u IPv4 140072557 0t0 TCP 124.46.185.34:http->140.205.127.50:51697 (ESTABLISHED)
nginx 13707 www-data 61u IPv4 140056450 0t0 TCP 124.46.185.34:http->140.205.127.37:27131 (ESTABLISHED)
nginx 13707 www-data 69u IPv4 140072632 0t0 TCP 124.46.185.34:http->140.205.127.13:20352 (ESTABLISHED)
nginx 13708 www-data 15u IPv4 6150826 0t0 TCP *:http (LISTEN)
2、查看进程
root@localhost:~# lsof -p 13721
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 13721 www-data cwd DIR 253,1 4096 2 /
nginx 13721 www-data rtd DIR 253,1 4096 2 /
nginx 13721 www-data txt REG 253,1 843688 1195996 /usr/sbin/nginx
nginx 13721 www-data mem REG 253,1 48024 269175 /lib/x86_64-linux-gnu/libnss_files-2.15.so
nginx 13721 www-data mem REG 253,1 47680 262260 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
nginx 13721 www-data mem REG 253,1 97248 269169 /lib/x86_64-linux-gnu/libnsl-2.15.so
nginx 13721 www-data mem REG 253,1 35680 269174 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
nginx 13721 www-data mem REG 253,1 1807032 269167 /lib/x86_64-linux-gnu/libc-2.15.so
nginx 13721 www-data mem REG 253,1 215136 1186378 /usr/lib/libGeoIP.so.1.4.8
nginx 13721 www-data mem REG 253,1 270400 1181963 /usr/lib/x86_64-linux-gnu/libgd.so.2.0.0
nginx 13721 www-data mem REG 253,1 84928 1195989 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15
nginx 13721 www-data mem REG 253,1 43288 269172 /lib/x86_64-linux-gnu/libcrypt-2.15.so
nginx 13721 www-data mem REG 253,1 135366 269170 /lib/x86_64-linux-gnu/libpthread-2.15.so
nginx 13721 www-data mem REG 253,1 149280 269173 /lib/x86_64-linux-gnu/ld-2.15.so
nginx 13721 www-data DEL REG 0,4 6150828 /dev/zero
nginx 13721 www-data 0u CHR 1,3 0t0 1029 /dev/null
nginx 13721 www-data 1u CHR 1,3 0t0 1029 /dev/null
nginx 13721 www-data 4w REG 253,1 85704 918896 /var/log/nginx/access.log
nginx 13721 www-data 13u unix 0xffff8807ff4ef500 0t0 59526717 socket
nginx 13721 www-data 14w REG 253,1 438012 918916 /var/log/nginx/admin.access.log
nginx 13721 www-data 15u IPv4 6150826 0t0 TCP *:http (LISTEN)
nginx 13721 www-data 16u IPv4 6150827 0t0 TCP *:http-alt (LISTEN)
nginx 13721 www-data 19u unix 0xffff8807ff4ee900 0t0 59526719 socket
nginx 13721 www-data 20u IPv4 140132456 0t0 TCP 124.46.185.34:http->140.205.127.60:16099 (ESTABLISHED)
nginx 13721 www-data 21u unix 0xffff8807ff4ebf00 0t0 59526721 socket
nginx 13721 www-data 22u IPv4 140132818 0t0 TCP 124.46.185.34:http->140.205.127.17:33358 (ESTABLISHED)
nginx 13721 www-data 23u unix 0xffff8807ff4ee600 0t0 59526723 socket
nginx 13721 www-data 24u IPv4 140132826 0t0 TCP 124.46.185.34:http->198.11.137.59:25415 (ESTABLISHED)
nginx 13721 www-data 25u unix 0xffff8807ff4e9200 0t0 59526725 socket
nginx 13721 www-data 26u IPv4 140106470 0t0 TCP 124.46.185.34:http->101.200.101.204:37659 (ESTABLISHED)
nginx 13721 www-data 27u IPv4 140106678 0t0 TCP 124.46.185.34:http->140.205.127.34:53569 (ESTABLISHED)
nginx 13721 www-data 30u IPv4 140132831 0t0 TCP 124.46.185.34:http->140.205.127.44:8165 (ESTABLISHED)
nginx 13721 www-data 31u IPv4 140132828 0t0 TCP 124.46.185.34:http->47.88.135.31:64913 (ESTABLISHED)
nginx 13721 www-data 33u IPv4 140132871 0t0 TCP 124.46.185.34:http->140.205.31.220:57845 (ESTABLISHED)
nginx 13721 www-data 34u IPv4 140132842 0t0 TCP 124.46.185.34:http->195.27.31.56:55498 (ESTABLISHED)
nginx 13721 www-data 35u IPv4 140132679 0t0 TCP 124.46.185.34:http->140.205.127.26:62043 (ESTABLISHED)
nginx 13721 www-data 36u IPv4 140132838 0t0 TCP 124.46.185.34:http->140.205.127.26:64949 (ESTABLISHED)
nginx 13721 www-data 37u unix 0xffff8807ff4ef200 0t0 59526727 socket
nginx 13721 www-data 38u IPv4 140132862 0t0 TCP 124.46.185.34:http->140.205.127.25:23082 (ESTABLISHED)
nginx 13721 www-data 39u unix 0xffff8807ff4ea700 0t0 59526729 socket
nginx 13721 www-data 41u unix 0xffff8807ff4ec200 0t0 59526731 socket
nginx 13721 www-data 42u IPv4 140132845 0t0 TCP 124.46.185.34:http->101.200.101.216:55441 (ESTABLISHED)
nginx 13721 www-data 43u unix 0xffff8807ff4eb900 0t0 59526733 socket
nginx 13721 www-data 44u IPv4 140106547 0t0 TCP 124.46.185.34:http->140.205.31.211:60180 (ESTABLISHED)
nginx 13721 www-data 45u unix 0xffff8807ff4e9b00 0t0 59526735 socket
nginx 13721 www-data 46u IPv4 140132879 0t0 TCP 124.46.185.34:http->140.205.31.203:42821 (ESTABLISHED)
nginx 13721 www-data 47u unix 0xffff8807ff4e8300 0t0 59526737 socket
nginx 13721 www-data 48u IPv4 140132847 0t0 TCP 124.46.185.34:http->140.205.127.45:53830 (ESTABLISHED)
nginx 13721 www-data 49u unix 0xffff8807ff4ee000 0t0 59526739 socket
nginx 13721 www-data 50u IPv4 140077698 0t0 TCP 124.46.185.34:http->140.205.127.43:43558 (ESTABLISHED)
nginx 13721 www-data 51u unix 0xffff8807ff4ed400 0t0 59526741 socket
nginx 13721 www-data 52u IPv4 140132857 0t0 TCP 124.46.185.34:http->198.11.137.19:22613 (ESTABLISHED)
nginx 13721 www-data 53u unix 0xffff8807ff4ece00 0t0 59526743 socket
nginx 13721 www-data 55u unix 0xffff8807ff4e8000 0t0 59526745 socket
nginx 13721 www-data 56u IPv4 140132853 0t0 TCP 124.46.185.34:http->140.205.31.212:23679 (ESTABLISHED)
nginx 13721 www-data 57u IPv4 140132859 0t0 TCP 124.46.185.34:http->140.205.127.61:7242 (ESTABLISHED)
nginx 13721 www-data 58u unix 0xffff8807ff4e9800 0t0 59526748 socket
nginx 13721 www-data 59u IPv4 140132864 0t0 TCP 124.46.185.34:http->101.200.101.201:35787 (ESTABLISHED)
nginx 13721 www-data 70u IPv4 140093317 0t0 TCP 124.46.185.34:http->140.205.127.34:37276 (ESTABLISHED)
nginx 13721 www-data 80u 0000 0,9 0 6874 anon_inode
nginx 13721 www-data 87u IPv4 140105862 0t0 TCP 124.46.185.34:http->140.205.127.11:40316 (ESTABLISHED)
nginx 13721 www-data 88u IPv4 140093346 0t0 TCP 124.46.185.34:http->140.205.127.36:29659 (ESTABLISHED)
3、lsof -c init
root@iZ25n2yx37sZ:~# lsof -c init
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root cwd DIR 253,1 4096 2 /
init 1 root rtd DIR 253,1 4096 2 /
init 1 root txt REG 253,1 167192 393221 /sbin/init
init 1 root mem REG 253,1 48024 269175 /lib/x86_64-linux-gnu/libnss_files-2.15.so
init 1 root mem REG 253,1 47680 262260 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
init 1 root mem REG 253,1 97248 269169 /lib/x86_64-linux-gnu/libnsl-2.15.so
init 1 root mem REG 253,1 35680 269174 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
init 1 root mem REG 253,1 1807032 269167 /lib/x86_64-linux-gnu/libc-2.15.so
init 1 root mem REG 253,1 31752 262372 /lib/x86_64-linux-gnu/librt-2.15.so
init 1 root mem REG 253,1 135366 269170 /lib/x86_64-linux-gnu/libpthread-2.15.so
init 1 root mem REG 253,1 276392 262185 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.8
init 1 root mem REG 253,1 38888 262171 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0
init 1 root mem REG 253,1 96240 262174 /lib/x86_64-linux-gnu/libnih.so.1.0.0
init 1 root mem REG 253,1 149280 269173 /lib/x86_64-linux-gnu/ld-2.15.so
init 1 root 0u CHR 1,3 0t0 1029 /dev/null
init 1 root 1u CHR 1,3 0t0 1029 /dev/null
init 1 root 2u CHR 1,3 0t0 1029 /dev/null
init 1 root 3r FIFO 0,8 0t0 7012 pipe
init 1 root 4w FIFO 0,8 0t0 7012 pipe
init 1 root 5r 0000 0,9 0 6874 anon_inode
init 1 root 6r 0000 0,9 0 6874 anon_inode
init 1 root 7u unix 0xffff88080b948000 0t0 7013 socket
init 1 root 8u unix 0xffff88080b918600 0t0 12351 socket
init 1 root 9u unix 0xffff880807670c00 0t0 1238 socket
init 1 root 10u unix 0xffff8808076a9500 0t0 7100 socket
init 1 root 11w REG 253,1 633 919710 /var/log/upstart/mysql.log.1 (deleted)
init 1 root 23u CHR 5,2 0t0 1111 /dev/ptmx
。。。。
参考文章
http://unix.stackexchange.com/questions/182077/best-way-to-free-disk-space-from-deleted-files-that-are-held-open
http://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted
http://www.cnblogs.com/276815076/p/3503923.html