python笔记1--lxml.etree解析html

前言

lxml可以解析html页面,学过xpath定位的,可以立马上手

使用环境:

python3.7

lxml 4.3.3

lxml安装

pip install lxml,安装报错;指定版本为4.4.3时,安装成功

python笔记1--lxml.etree解析html

 

 python笔记1--lxml.etree解析html

pip show lxml查看版本号

python笔记1--lxml.etree解析html

html解析

这里用到etree.HTML方法把html的文本内容解析成html对象

要打印html内容,可以用etree.tostring方法,encoding="utf-8"参数可以正常输出html里面的中文内容。pretty_print=True是以标准格式输出

# coding:utf-8

from lxml import etree

htmldemo = '''
<meta charset="UTF-8"> <!-- for HTML5 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<html><head><title>yoyo ketang</title></head>
<body>
<b><!--Hey, this in comment!--></b>
<p class="title"><b>yoyoketang</b></p>
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>
<p class="story">...</p>
'''

# etree.HTML解析html内容
demo = etree.HTML(htmldemo)
# 打印解析内容str
t = etree.tostring(demo, encoding="utf-8", pretty_print=True)
print(t.decode("utf-8"))  

 运行结果

<html>
  <head><meta charset="UTF-8"/> <!-- for HTML5 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>yoyo ketang</title>
</head>
  <body>
<b><!--Hey, this in comment!--></b>
<p class="title"><b>yoyoketang</b></p>
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>
<p class="story">...</p>
</body>
</html>

 soupparser解析器

soupparser解析器比上面的etree.HTML容错性要好一点,因为其处理不规范的html能力比etree强太多。

import lxml.html.soupparser as soupparser
demo = soupparser.fromstring(htmldemo)
t = etree.tostring(demo, encoding="utf-8", pretty_print=True)

print(t.decode("utf-8"))

  xpath使用案例

使用html解析器,最终是想获取html上的某些元素属性和text文本内容,接下来看下,用最少的代码,简单高效的找出想要的内容。

比如要获取“这里是我的微信公众号:yoyoketang

# coding:utf-8

from lxml import etree

htmldemo = '''
<meta charset="UTF-8"> <!-- for HTML5 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<html><head><title>yoyo ketang</title></head>
<body>
<b><!--Hey, this in comment!--></b>
<p class="title"><b>yoyoketang</b></p>
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>
<p class="story">...</p>
'''
# etree.HTML解析html内容
demo = etree.HTML(htmldemo)
rs=demo.xpath('//p[@class="yoyo"]')
t=rs[0].text
print(t)  

 运行结果:

 python笔记1--lxml.etree解析html

 

 从代码量上看,简单的三行代码就能找到想要的内容了,rs是xpath定位获取到的一个list对象,会找出所有符合条件的元素对象。可以用for循环查看详情。

# coding:utf-8

from lxml import etree

htmldemo = '''
<meta charset="UTF-8"> <!-- for HTML5 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<html><head><title>yoyo ketang</title></head>
<body>
<b><!--Hey, this in comment!--></b>
<p class="title"><b>yoyoketang</b></p>
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>
<p class="story">...</p>
'''
# etree.HTML解析html内容
demo = etree.HTML(htmldemo)
rs=demo.xpath('//p[@class="yoyo"]')
print(rs) #list对象
for j in rs:
    #打印定位到的内容
    print(etree.tostring(j,encoding="utf-8",pretty_print=True).decode("utf-8"))
    print(j.attrib)  

 运行结果

[<Element p at 0x262b525a988>]
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>


{'class': 'yoyo'}

 二次查找

通过xpath定位语法//p[@class="yoyo"]定位到的是class="yoyo"这个元素以及它的所有子节点,如果想定位其中一个子节点,可以二次定位,继续xpath查看,如获取:python笔记

# coding:utf-8

from lxml import etree

htmldemo = '''
<meta charset="UTF-8"> <!-- for HTML5 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<html><head><title>yoyo ketang</title></head>
<body>
<b><!--Hey, this in comment!--></b>
<p class="title"><b>yoyoketang</b></p>
<p class="yoyo">这里是我的微信公众号:yoyoketang
<a href="http://www.cnblogs.com/yoyoketang/tag/fiddler/" class="sister" id="link1">fiddler教程</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/python/" class="sister" id="link2">python笔记</a>,
<a href="http://www.cnblogs.com/yoyoketang/tag/selenium/" class="sister" id="link3">selenium文档</a>;
快来关注吧!</p>
<p class="story">...</p>
'''
# etree.HTML解析html内容
demo = etree.HTML(htmldemo)
rs=demo.xpath('//p[@class="yoyo"]')
print(rs[0].text)
rs1=rs[0].xpath('//a[@id="link2"]')
print(rs1[0].text)
rs2=demo.xpath('//a[@id="link2"]')
print(rs2[0].text)  

 运行结果

python笔记1--lxml.etree解析html

 

上一篇:python测试开发django-83.Dockerfile部署django项目


下一篇:linux下snmp编译安装