作业要求:https://edu.cnblogs.com/campus/gzcc/GZCC-16SE2/homework/2773
1. 下载一长篇中文小说。
来自红楼梦的一小章内容:
2. 从文件读取待分析文本。
text=open('123.txt','r',encoding='utf-8').read()
3. 安装并使用jieba进行中文分词。
pip install jieba
import jieba
jieba.lcut(text)
import jieba wordsls=jieba.lcut(text)
4. 更新词库,加入所分析对象的专业词汇。
jieba.add_word('天罡北斗阵') #逐个添加
jieba.load_userdict(word_dict) #词库文本文件
参考词库下载地址:https://pinyin.sogou.com/dict/
转换代码:scel_to_text
词库:
worddict1=[line.strip() for line in open('23.txt',encoding='utf-8').readlines()] jieba.load_userdict(worddict1)
5. 生成词频统计
wcdict={} for word in wordsls: if word not in worddict2:(7.排除语法型) if len(word)==1: continue else: wcdict[word]=wcdict.get(word,0)+1
6. 排序
wcls=list(wcdict.items()) wcls.sort(key=lambda x:x[1],reverse=True)
7. 排除语法型词汇,代词、冠词、连词等停用词。
文件:
stops
worddict2=[line.strip() for line in open('stops_chinese.txt',encoding='utf-8').readlines()]
8. 输出词频最大TOP20,把结果存放到文件里
import pandas as pd pd.DataFrame(data=word).to_csv('E:/1234.csv',encoding='utf-8')
9. 生成词云。
wl_split=" ".join(wordsls) from wordcloud import WordCloud import matplotlib.pyplot as plt mywc = WordCloud().generate(wl_split) plt.imshow(mywc) plt.axis("off") plt.show()
10.最总代码总和和截图:
import jieba text=open('D://123.txt','r',encoding='utf-8').read() worddict1=open('D://23.txt','r',encoding='utf-8').read() worddict2=open('D://stops_chinese.txt','r',encoding='utf-8').read() wordsls=jieba.lcut(text) wcdict={} for word in wordsls: if word not in worddict2: if len(word)==1: continue else: wcdict[word]=wcdict.get(word,0)+1 wcls=list(wcdict.items()) wcls.sort(key=lambda x:x[1],reverse=True) for i in range(25): print(wcls[i]) wl_split=" ".join(wordsls) from wordcloud import WordCloud import matplotlib.pyplot as plt mywc = WordCloud().generate(wl_split) plt.imshow(mywc) plt.axis("off") plt.show()