任务：crm日志的查询条件 每次是哪几个字段查，有几种组合 ，统计每种组合查询的量
日志样例：
132.xxx.xx.x - - [-- ::] "GET /REST/HTableService?appId=crmyun&partition=2017&query=QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData&latnId=755&staffNo=GZTEST200×tamp=1506585708188&signature=D73E9B59E08EA7B1C2D0DDA72AC957E4 HTTP/1.1"
132.xxx.xx.x - - [-- ::] "GET /REST/HTableService?staffNo=xxTEST200&appId=crmyun&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData×tamp=1505871359000&signature=6743AE272C10BCC2261E11AF4CA5EA19&charset=UTF-8&partition=2017&query=STAFF_ID=1212100141|and|DATETIME>20170917000000000|and|DATETIME<20170919000000000 HTTP/1.1"
查询条件：query查询条件可以多个，用|and|分割。
步骤：
1、正则获取query查询条件组合
query=QUERY_TYPE1%3D%E6%8E%A5%E5%%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000
query=STAFF_ID=|and|DATETIME>|and|DATETIME<
2、截取query列表，得到查询条件组合；以"%7C|\|"分割，得到列表
['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'and', 'QUERY_VALUE1%3D17727955834', 'and', 'DATETIME%3E20170925000000000', 'and', 'DATETIME%3C20170928000000000']
['STAFF_ID=1212100141', 'and', 'DATETIME>20170917000000000', 'and', 'DATETIME<20170919000000000']
3、剔除 'and' 项(列表取[::2])得到新列表
['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'QUERY_VALUE1%3D17727955834', 'DATETIME%3E20170925000000000', 'DATETIME%3C20170928000000000']
['STAFF_ID=1212100141', 'DATETIME>20170917000000000', 'DATETIME<20170919000000000']
4、以'%3D|%3E|%3C|>|<|='分割，并将key放入set()中，得到去重后的结果
['QUERY_TYPE1', '%E6%8E%A5%E5%85%A5%E5%8F%B7']
['QUERY_VALUE1', '']
['DATETIME', '']
['DATETIME', ''] ['STAFF_ID', '']
['DATETIME', '']
['DATETIME', '']
5、将列表key值放入set()中，得到结果；参考代码如下
import sys
import time
import re def read_write():
with open("C:\\Users\\admin\\Desktop\\c5.log", 'r') as f1:
for line in f1.readlines():
pattern = re.compile(r'query=.*?\s')
results = re.search(pattern, line).group().split('&')
for result in results:
if result.startswith("query"):
temp = result[6: ]
list = re.split("%7C|\|",temp)[::2]
# print list
my_set = set()
for l in list:
arrya = re.split('%3D|%3E|%3C|>|<|=', l)
my_set.add(arrya[0])
print my_set
c = [i for i in my_set]
file = open("C:\\Users\\admin\\Desktop\\4.txt", 'a')
file.write(repr(c)+'\n')
file.close() if __name__ == '__main__':
start = time.time()
read_write()
stop = time.time()
print "running time is "+str(stop - start)