三、基于hadoop的nginx访问日志分析--计算时刻pv

代码:

# cat pv_hour.py
#!/usr/bin/env python
# coding=utf-8 from mrjob.job import MRJob
from nginx_accesslog_parser import NginxLineParser class PvDay(MRJob): nginx_line_parser = NginxLineParser() def mapper(self, _, line): self.nginx_line_parser.parse(line)
_, tm = str(self.nginx_line_parser.time_local).split()
h, m, s = tm.split(':')
yield h, 1 # 每小时的 def reducer(self, key, values):
yield key, sum(values) def main():
PvDay.run() if __name__ == '__main__':
main()

执行结果

# python3 pv_hour.py access_all.log-20161227
No configs found; falling back on auto-configuration
Creating temp directory /tmp/pv_hour.root.20161228.025503.341576
Running step 1 of 1...
Streaming final output from /tmp/pv_hour.root.20161228.025503.341576/output...
"" 21158
"" 20958
"" 16080
"" 14194
"" 13114
"" 16898
"" 18870
"" 14067
"" 14053
"" 12683
"" 13185
"" 14785
"" 12449
"" 7364
"" 3628
"" 9074
"" 9317
"" 11887
"" 13492
"" 19564
"" 18390
"" 15697
"" 17518
"" 18785
Removing temp directory /tmp/pv_hour.root.20161228.025503.341576...
上一篇:RPC-dubbo基本使用


下一篇:常见的sql语句 注意点及用法【区分mysql 和Sqlserver】