nginx error.log 中出现大量如下错误信息：
[root@zayhu01-fk nginx]# grep -aP '^20.* \[crit\]' error.log
2017/03/14 12:06:31 [crit] 3549#0: accept4() failed (24: Too many open files)
[root@zayhu01-fk nginx]# grep -aP '^20.* \[alert\]' error.log
2017/03/14 16:04:27 [alert] 3551#0: *84168270 socket() failed (24: Too many open files) while connecting to upstream, client: 1.1.1.1, server:...
由于系统limits open files 限制导致以上错误，所以：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
[root@zayhu01-fk ~]# tail -4 /etc/security/limits.conf
* - nofile 500000push - nproc 65536push - nofile 320000work - nproc 10000[root@zayhu01-fk ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimited
pending signals (-i) 128630max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimitedopen files (-n) 500000
pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0
stack size (kbytes, -s) 8192cpu time (seconds, -t) unlimited
max user processes (-u) 128630virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@zayhu01-fk ~]#
|
但以上只是系统的open files 限制设置为500000；而nginx的open files 是否继承系统open files 设置还需要重新启动nginx 进程。可以通过以下命令查看：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@zayhu01-fk ~]# cat /proc/`ps -ef | grep nginx|grep -v grep|head -1|awk '{print $2}'`/limits
Limit Soft Limit Hard Limit UnitsMax cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytesMax stack size 8388608 unlimited bytesMax core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128630 128630 processesMax open files 500000 500000 files
Max locked memory 65536 65536 bytesMax address space unlimited unlimited bytesMax file locks unlimited unlimited locks
Max pending signals 128630 128630 signalsMax msgqueue size 819200 819200 bytesMax nice priority 0 0
Max realtime priority 0 0Max realtime timeout unlimited unlimited us[root@zayhu01-fk ~]#
|
总结：ulimit -a && cat /proc/`ps -ef | grep nginx|grep -v grep| head -1 | awk '{print $2}'`/limits 必须保持一致，否则会导致大量连接失败，此时需要重启nginx 进程 。
可使用以下函数对error.log 中alert字段监控：
|
1
2
3
4
|
function nginx_alert_error(){
USER=$1
sudo runuser - $USER -c "[ -e /var/log/nginx/error.log ] && tail -c 50m /var/log/nginx/error.log | grep -aP '^20.*:\d{2} \[alert\]'|tail -c500k "|awk -F '[' 'BEGIN{"date -d \"-600 seconds\" \"+%Y/%m/%d %T\" " | getline cTS}{if($1>cTS) print $0 }'|grep "^20"
|
本文转自 meteor_hy 51CTO博客，原文链接：http://blog.51cto.com/caiyuanji/1906565，如需转载请自行联系原作者