第一部分 英文文本分析词频
以Hamlet文本为例,文本下载链接: https://python123.io/resources/pye/hamlet.txt
#CalHamletV1.py #hamlet文本下载链接:https://python123.io/resources/pye/hamlet.txt def getText(): #对文本归一化处理(变为小写,特殊字符替换为空格) txt = open("hamlet.txt","r").read() txt = txt.lower() #所有字母变为小写 for ch in ‘!"#$%&()*+,-./:;<=>?@[\\]^_{|}`~‘: txt= txt.replace(ch," ") #用空格代替各种特殊字符 return txt hamletTxt=getText() words =hamletTxt.split() #根据空格分隔每一个字母 counts ={} for word in words: counts[word] = counts.get(word,0) + 1 #如果键不存在字典中,给出默认值 items = list(counts.items()) #变为列表类型,便于排序操作 items.sort(key=lambda x:x[1], reverse=True) #对第二个元素,从大到小倒排 #sort方法小知识:参数lambda用来指定列表中使用哪一个多元选项的列作为排序列,默认的排序是从小到大;reverse设为True,则排序从大到小 for i in range(10): #输出最多的10个单词 word, count = items[i] print("{0:<10}{1:>5}".format(word, count))
运行结果:
第二部分 中文文本分析词频
以《三国演义》文本为例,进行人物出场次数统计,文本下载链接:https://python123.io/resources/pye/threekingdoms.txt
分析:与英文词频分析不同,中文文本分词需要借助第三方库“jieba”,在cmd下输入“pip install jieba”即可安装该库。
#CalThreeKingdomsV1.py import jieba txt = open("threekingdoms.txt","r",encoding="utf-8").read() words = jieba.lcut(txt) counts = {} for word in words: if len(word) == 1: continue else: counts[word]=counts.get(word,0)+ 1 items = list(counts.items()) items.sort(key=lambda x:x[1],reverse=True) for i in range(15): word,count = items[i] print("{0:<10}{1:>5}".format(word,count))
运行结果:
分析结果:发现“孔明”和“孔明曰”是同一个人,分词时却分成两个词组;另外,“二人”、“却说”等并不是人名。因此,需要将词频与任务相关联,面向问题修改程序。
解决思路:1.排除词库;2.名称关联。通过不断显示结果进行优化程序。
#CalThreeKingdomsv2.py import jieba txt = open("threekingdoms.txt","r",encoding="utf-8").read() excludes = {"将军","却说","荆州","二人","不可","不能","如此"} words = jieba.lcut(txt) counts={} for word in words: if len(word) == 1: continue elif word =="诸葛亮" or word == "孔明曰": rword = "孔明" elif word =="关公"or word =="云长": rword ="关羽" elif word =="玄德" or word =="玄德曰": rword ="刘备" elif word == "孟德"or word =="丞相": rword = "曹操" else: rword = word counts[rword]= counts.get(rword,0)+ 1 for word in excludes: del counts[word] items = list(counts.items()) items.sort(key=lambda x:x[1],reverse=True) for i in range(10): word,count = items[i] print("{0:<10}{1:>5}".format(word,count))
运行结果:
接着,需要继续优化程序,把“商议”、“如何”等词语加到排除集合excludes中。
最终最终,可以得出《三国演义》人物出场统计结果为(前20):
曹操、孔明、刘备、关羽、张飞、吕布、赵云、孙权、司马懿、周瑜、袁绍、马超、魏延、黄忠、姜维、马岱、庞德、孟获、刘表、夏侯惇。