awk 数组应用案例

一、四个案例解析

1.1 案例1 查酒店

假设我们有一个酒店:
酒店<===>hotel
酒店里面有几个房间110,119,120,114,401
酒店的110房间<===> hotel[110]
酒店的119房间<===> hotel[119]
酒店的120房间<===> hotel[120]

如何查看房间里住的是哪位客人？
# awk 'BEGIN{hotel[110]="tom"; hotel[119]="jack";hotel[121]="owen";print hotel[110],hotel[121],hotel[119]}'
tom owen jack

使用for语句，对酒店进行循环/查房
# awk 'BEGIN{hotel[110]="tom"; hotel[119]="jack";hotel[121]="owen";
> for(i in hotel)
> print i,hotel[i]
> }'
119 jack
110 tom
121 owen

i 为房间号码
hotel[i] 房间里所住的人

1.2 案例 统计域名访问次数（去重统计）

准备源文件
# cat > www.txt <<eof
http://www.aiops.com/index.html
http://www.aiops.com/1.html
http://post.aiops.com/index.html
http://mp3.aiops.com/index.html
http://www.aiops.com/3.html
http://post.aiops.com/2.html
eof

1.2.1 案例2 使用sort uniq

sort,按字母排序

# awk -F '[/]+' '{print $2}' www.txt |sort
mp3.aiops.com
post.aiops.com
post.aiops.com
www.aiops.com
www.aiops.com
www.aiops.com

uniq把相邻重复2行合并为1行

# awk -F '[/]+' '{print $2}' www.txt |uniq -c
      2 www.aiops.com
      1 post.aiops.com
      1 mp3.aiops.com
      1 www.aiops.com
      1 post.aiops.com

# awk -F '[/]+' '{print $2}' www.txt | sort | uniq -c
      1 mp3.aiops.com
      2 post.aiops.com
      3 www.aiops.com

1.2.2.1 第一步 取出域名

# awk -F '[/]+' '{print $2}' www.txt
www.aiops.com
www.aiops.com
post.aiops.com
mp3.aiops.com
www.aiops.com
post.aiops.com

1.2.2.2 第二步 显示

# awk -F '/+' '{site[$2]++;print site["www.aiops.com"]}' www.txt
1
2
2
2
3
3

1.2.2.3 第三步 显示结果

# awk -F '/+' '{site[$2]++}END{for(i in site)print i,site[i]}' www.txt        
mp3.aiops.com 1
post.aiops.com 2
www.aiops.com 3

1.3 案例3 统计TCP连接状态

# netstat -an | awk '/^tcp/ {S[$NF]++}END{for(i in S ) print i,S[i]}'
LISTEN 11
ESTABLISHED 2

1.4 案例4 统计暴力破解次数

# awk '/Failed/{fa[$(NF-3)]++}END{for(pol in fa)print pol,fa[pol]}' /var/log/secure|column -t
192.168.2.128  3

二、使用awk对nginx的高级统计应用

1.1 统计访问IP次数

[root@web03 logs]# awk '{ips[$1]++}END{for (i in ips)print i,ips[i]}' access.log 

1.2 统计访问大于100次的IP

[root@web03 logs]# awk '{ips[$1]++}END{for (i in ips){if(ips[i]>100)print i,ips[i]}}' access.log 
219.133.100.62 208
113.87.130.224 555
117.136.39.100 119
209.141.46.254 115
61.141.72.75 937
209.141.50.5 240
119.123.74.148 1161
117.136.41.126 109
223.104.66.107 125
39.104.25.221 643

1.3 统计访问IP次数并排序取前10

[root@web03 logs]# awk '{ips[$1]++}END{for (i in ips)print i,ips[i]}' access.log |sort -n -r -k2 |head -10 
119.123.74.148 1161
61.141.72.75 937
39.104.25.221 643
113.87.130.224 555
209.141.50.5 240
219.133.100.62 208
223.104.66.107 125
117.136.39.100 119
209.141.46.254 115
117.136.41.126 109

1.4 统计时间段访问最多的IP

[root@web03 logs]# awk '$4>="[01/Jan/2021:17:56:45" && $4<="[03/Jan/2021:17:56:45"{ips[$1]++}END{for (i in ips)print i,ips[i]}' access.log |sort -n -r -k2 | head -100 

1.5 统计访问最多的10个页面

[root@web03 logs]# awk '{ips[$7]++}END{for (i in ips)print i,ips[i]}' access.log | sort -nr -k2 |head -50
/ 2504
/wp-admin/admin-ajax.php 780
400 395
/favicon.ico 225
/wp-includes/js/wp-embed.min.js?ver=5.5.3 147
/wp-includes/css/dist/block-library/style.min.css?ver=5.5.3 143
/wp-includes/css/dist/block-library/theme.min.css?ver=5.5.3 138
/wp-content/uploads/2020/12/image-1-1024x399.png 138
/wp-content/themes/twentynineteen/js/touch-keyboard-navigation.js?ver=20181231 123
/wp-content/themes/twentynineteen/js/priority-menu.js?ver=20181214 120
/wp-content/themes/twentynineteen/style.css?ver=1.7 119
/wp-content/themes/twentynineteen/print.css?ver=1.7 119
/wp-includes/js/wp-emoji-release.min.js?ver=5.5.3 96
/wp-content/uploads/2020/12/wp_editor_md_cfe25440a82e58a27d80cb9109296a63.jpg 89
http://www.baidu.com/cache/global/img/gs.gif 83
mstshash=Administr" 76

1.6 统计每个URL数量和返回内容总大小

[root@web03 logs]# awk '{a[$7]++;size[$7]+=$10}END{for(v in a)print a[v],v,size[v]}' access.log

1.7 统计每个IP访问状态码数量

[root@web03 logs]# awk '{a[$1" "$9]++}END{for(v in a)print v,a[v]}' access.log |sort -nr -k3 

1.8 统计访问IP是404状态次数

[root@web03 logs]# awk '{a[$1" "$9]++}END{for(v in a)print v,a[v]}' access.log |sort -nr -k3 |grep  404 
119.123.74.148 404 41
61.141.72.75 404 6
40.76.11.116 404 3
39.104.25.221 404 2
206.189.49.38 404 2
167.71.26.36 404 2
161.35.5.10 404 2
143.110.168.2 404 2
134.209.29.67 404 2
128.90.177.230 404 2
37.46.150.5 404 1
37.46.150.211 404 1
118.123.1.38 404 1

[root@web03 logs]# awk '$9~/404/{ips[$1]++}END{for (i in ips)print i,ips[i]}' access.log  
206.189.49.38 2
118.123.1.38 1
161.35.5.10 2
39.104.25.221 2
128.90.177.230 2
143.110.168.2 2
37.46.150.5 1
167.71.26.36 2
37.46.150.211 1
134.209.29.67 2
40.76.11.116 3
61.141.72.75 6
119.123.74.148 41
[root@web03 logs]# awk '$9~/404/{ips[$1]++}END{for (i in ips)print i,ips[i]}' access.log  |wc -l 
13
[root@web03 logs]#