BeautifulSoup使用手册（查询篇）

2023-11-26 08:03:34

开始使用呢
解析器
四种对象
tag对象
Css选择器
- select
- select_one

基本上是按官方文档所写 https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/，过完。·BeautifulSoup 内容太多了，用的时候查起来方便一点

开始使用呢

从一个soup对象开始，以下两种方式生成一个soup对象

from bs4 import BeautifulSoup
soup = BeautifulSoup(open("index.html"))    ##传入文件
soup = BeautifulSoup("<html>data</html>")   ##文本

构造soup对象时，可以传入解析器参数，如果不传入的话，会以最好的方式去解析

解析器

解析器有如下

html.parser
lxml
xml
html5lib

使用html.parser解析器

soup = BeautifulSoup("<html>data</html>","html.parser")

四种对象

Beautiful Soup将复杂HTML文档转换成一个复杂的树形结构,每个节点都是Python对象,所有对象可以归纳为4种 Tag , NavigableString , BeautifulSoup , Comment

tag对象

tag对象，同网页中的标签的意思

html标签

soup = BeautifulSoup("<html>data</html>","html.parser")
soup.html

a标签

soup = BeautifulSoup("<a >data</a>","html.parser")
soup.a

标签名(name)

标签拥有自己的名字，用可以直接使用标签调用

soup = BeautifulSoup("<html><a >data</a></html>","html.parser")
print(soup.a.name)

**结果为 a **

属性值(Attributes)

soup = BeautifulSoup("<html><a href='baidu.com'>data</a></html>","html.parser")
print(soup.a['href'])

**结果为： baidu.com **

多值属性

HTML 4定义了一系列可以包含多个值的属性.在HTML5中移除了一些,却增加更多.最常见的多值的属性是 class (一个tag可以有多个CSS的class). 还有一些属性 rel , rev , accept-charset , headers , accesskey . 在Beautiful Soup中多值属性的返回类型是list:

soup = BeautifulSoup("<html><a href='baidu.com' rev='ll' class='night'>data</a></html>","html.parser")
print(soup.a['class'])
print(soup.a['rev'])

**结果为： **

['night']
['ll']

非多值的情况下回返回字符串

id_soup = BeautifulSoup('<p id="my id"></p>')
id_soup.p['id']
# 'my id'

属性赋值，属性可以赋值，多值的情况如下，数组回以空格拼接上去

rel_soup = BeautifulSoup('<p>Back to the <a rel="index">homepage</a></p>')
rel_soup.a['rel']
# ['index']
rel_soup.a['rel'] = ['index', 'contents']
print(rel_soup.p)
# <p>Back to the <a rel="index contents">homepage</a></p>

内容

刚刚我们已经得到了tag，并且获取到了属性，现在我们来获取内容。很简单tag元素调用.string

from bs4 import BeautifulSoup
if __name__ == '__main__':
    soup = BeautifulSoup("<a >data</a>", "html.parser")
    print(soup.a.string)
    print(type(soup.a.string))
    print(str(soup.a.string))

**结果为： **

data
<class 'bs4.element.NavigableString'>
data

我们发现这样得到的内容并不是str型，但查看bs4.element.NavigableString的源码，发现NavigableString的源码继承了str类

class NavigableString(str, PageElement):
省略其他代码

官方手册中如下描述如果想在Beautiful Soup之外使用 NavigableString 对象,需要调用 unicode() 方法,将该对象转换成普通的Unicode字符串,否则就算Beautiful Soup已方法已经执行结束,该对象的输出也会带有对象的引用地址.这样会浪费内存.而python3中并没有unicode()，使用str()代替

Comment对象

这种情况下，会产生Comment对象

markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))

结果为：

Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>

我们可以看到这时候.string返回的对象不再是bs4.element.NavigableString，而是Comment

prettify()方法

prettify()方法返回html对象

markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))
print(soup.b.prettify())

结果为：

Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>
<b>
 <!--Hey, buddy. Want to buy a used parser?-->
</b>

soup = BeautifulSoup("<a href='baidu.com'>data</a>", "html.parser")
print(soup.a.prettify())

结果为：

<a href="baidu.com">
 data
</a>

find_all方法

返回所有<a> 标签列表

contents属性

返回元素的所有子元素也是标签列表

el = soup.head.contents

children属性

返回所有子元素的一个迭代器

from bs4 import BeautifulSoup
html='''<!DOCTYPE html>
<html><head><title>The Dormouse's story</title></head>
    <body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
    </body>
</html>'''
if __name__ == '__main__':
    soup = BeautifulSoup(html, "html.parser")
    al=soup.find_all('a')
    print(al)
    print(type(al[0]))
    el = soup.head.contents
    print(el)
    print(type(el[0]))
    chd=soup.head.children
    print(chd)
    for el in soup.head.children:
        print(type(el))

结果为：

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
<class 'bs4.element.Tag'>
[<title>The Dormouse's story</title>]
<class 'bs4.element.Tag'>
<list_iterator object at 0x0000021EABA5F2B0>
<class 'bs4.element.Tag'>

descendants属性

contents属性和children属性都是直接子元素，而descendants是所有子孙元素（这里看做孙元素）

from bs4 import BeautifulSoup
if __name__ == '__main__':
    soup = BeautifulSoup("<html><head><title>The Dormouse's story</title></head></html>","html.parser")
    head_tag = soup.head
    for child in head_tag.descendants:
        print(child)

结果为：

<title>The Dormouse's story</title>
The Dormouse's story

兄弟元素

from  bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>
</body>
</html>'''


if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html5lib")
    print(soup.p)
    print(soup.p.next_sibling)

下一个兄弟元素，兄弟元素的理解为同父元素，next_sibling下一个挨着的兄弟元素
结果为：

<p class="title"><b>The Dormouse's story</b></p>
    //空行

解释一下上面的结果，上面空行是因为第一个p标签后面有回车符，所以p的下一个兄弟元素并不是第二个p。修改代码为如下

next_sibling

from  bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p><p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>
</body>
</html>'''


if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html.parser")
    print(soup.p)
    print(soup.p.next_sibling)

结果为：

<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
    <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
    <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

previous_siblings

这个代表上一个兄弟元素，和next_sibling同理

next_element

next_element代表下一个元素，和下一个兄弟元素的 next_sibling相比，next_element不排除下一个不是兄弟元素的情况

last_a_tag = soup.find("a", id="link3")
last_a_tag
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

last_a_tag.next_sibling
# '; and they lived at the bottom of a well.'

last_a_tag.next_element
# u'Tillie'

previous_element

previous_element 代表上一个元素，和next_element同理

过滤器find_all(flter)

find_all 方法中的过滤器可以用来过滤tag中的name

字符串

soup.find_all('b')
# 结果为 [<b>The Dormouse's story</b>]

正则表达式

import re
for tag in soup.find_all(re.compile("^b")):
    print(tag.name)

匹配到了所有b为开头的标签

结果为：

body
b

for tag in soup.find_all(re.compile("t")):
    print(tag.name)
# 结果为下面两行，标签名包含t的标签
# html
# title

列表

soup.find_all(["a", "b"])
# [<b>The Dormouse's story</b>,
#  <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

True/False

没太大意义和使用价值

for tag in soup.find_all(True):
    print(tag.name)
# html
# head
# title
# body
# p
# b
# p
# a
# a
# a
# p

方法

定义了元素含有class属性，但不含id属性的过滤方法

def has_class_but_no_id(tag):
    return tag.has_attr('class') and not tag.has_attr('id')

soup.find_all(has_class_but_no_id)
# 结果如下三行
# [<p class="title"><b>The Dormouse's story</b></p>,
#  <p class="story">Once upon a time there were...</p>,
#  <p class="story">...</p>]

方法还能过滤属性

def not_lacie(href):
        return href and not re.compile("lacie").search(href)
soup.find_all(href=not_lacie)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

总概

soup.find_all("title")
# [<title>The Dormouse's story</title>]

soup.find_all("p", "title")
# [<p class="title"><b>The Dormouse's story</b></p>]

soup.find_all("a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.find_all(id="link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

import re
soup.find(string=re.compile("sisters"))
# u'Once upon a time there were three little sisters; and their names were\n'

有几个方法很相似,还有几个方法是新的,参数中的 string 和 id 是什么含义? 
为什么 find_all("p", "title") 返回的是CSS Class为”title”的<p>标签?

参数列表解读

find_all( name , attrs , recursive , string , **kwargs )

name 根据tag的name筛选
attrs 接受字典类型的属性比如 soup.find_all(attrs={"id": "title"})，阅读SoupStrainer类的源码可以发现如下

 if not isinstance(attrs, dict):
            # Treat a non-dict value for attrs as a search for the 'class'
            # attribute.
            kwargs['class'] = attrs
            attrs = None

这里可以看出当attrs不为字典，为普通字符实，会把attrs这个参数当做'class'属性

recursive 是否从当前位置递归往下查询，如果不递归，只会查询当前soup文档的子元素
string 这里是通过tag的内容来搜索，并且返回的是类容，而不是tag类型的元素
**kwargs 自动拆包接受属性值，所以才会有soup.find_all('a',id='title') ，id='title'为**kwargs自动拆包掺入

class_

按照CSS类名搜索tag的功能非常实用,但标识CSS类名的关键字 class 在Python中是保留字,使用 class 做参数会导致语法错误.从Beautiful Soup的4.1.1版本开始,可以通过 class_ 参数搜索有指定CSS类名的tag:

soup.find_all("a", class_="sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

搜索 class 属性时也可以通过CSS值完全匹配:

css_soup.find_all("p", class_="body strikeout")
# [<p class="body strikeout"></p>]

limit 参数

限制返回的条数

soup.find_all("a", limit=2)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

find

同find_all方法类似，唯一区别是find返回一个元素，find_all返回一个列表。方法参数和find_all相同find( name , attrs , recursive , string , **kwargs ) ，使用方式也相同

soup.find_all('title', limit=1)
# [<title>The Dormouse's story</title>]
soup.find('title')
# <title>The Dormouse's story</title>

除了find_all，find之外，还有很多方法类似，如下

find_parents

find_parents( name , attrs , recursive , string , **kwargs )

查找所有的父辈节点（包括多级的祖先），返回一个列表

find_parent

find_parent( name , attrs , recursive , string , **kwargs )

查找上一个父辈节点，会一直在文档树上面查找，不会只查找一级别，特别注意

from bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
    <div class='cover'> <div> <p></p>  </div>  </div>
</body>
</html>'''
if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html.parser")
    p=soup.find('p')
    print(p.find_parent("div"))
    print(p.find_parents("div"))

#结果
<div> <p></p> </div>    #文档树上一个查找的div
[<div> <p></p> </div>, <div class="cover"> <div> <p></p> </div> </div>]  #所有div

官方案例

a_string = soup.find(string="Lacie")
a_string
# u'Lacie'

a_string.find_parents("a")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

a_string.find_parent("p")
# <p class="story">Once upon a time there were three little sisters; and their names were
#  <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
#  and they lived at the bottom of a well.</p>

a_string.find_parents("p", class="title")
# []

find_next_siblings，find_next_sibling

find_next_siblings( name , attrs , recursive , string , **kwargs )
find_next_sibling( name , attrs , recursive , string , **kwargs )

这2个方法通过 .next_siblings 属性对当tag的所有后面解析 [5] 的兄弟tag节点进行迭代，find_next_siblings() 方法返回所有符合条件的后面的兄弟节点, find_next_sibling() 只返回符合条件的后面的第一个tag节点.

find_previous_siblings，find_previous_sibling

find_previous_siblings( name , attrs , recursive , string , **kwargs )
find_previous_sibling( name , attrs , recursive , string , **kwargs )

和find_next_siblings，find_next_sibling 类似，只是查找前面的兄弟元素

find_all_next，find_next

find_all_next( name , attrs , recursive , string , **kwargs )
find_next( name , attrs , recursive , string , **kwargs )

这2个方法通过 .next_elements 属性对当前tag的之后的 [5] tag和字符串进行迭代, find_all_next() 方法返回所有符合条件的节点, find_next() 方法返回第一个符合条件的节点

find_all_previous，find_previous

find_all_previous( name , attrs , recursive , string , **kwargs )
find_previous( name , attrs , recursive , string , **kwargs )

和find_all_next，find_next类似,向前查找。这2个方法通过 .previous_elements 属性对当前节点前面 [5] 的tag和字符串进行迭代。

Css选择器

select

css选择器的方法为select(css_selector)
目前支持的选择器如下案例，css选择器可以参考https://www.w3school.com.cn/cssref/css_selectors.ASP

soup.select("title")
# [<title>The Dormouse's story</title>]

soup.select("p nth-of-type(3)")
# [<p class="story">...</p>]

soup.select("body a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("html head title")
# [<title>The Dormouse's story</title>]


soup.select("head > title")
# [<title>The Dormouse's story</title>]

soup.select("p > a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("p > a:nth-of-type(2)")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

soup.select("p > #link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("body > a")

soup.select("#link1 ~ .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie"  id="link3">Tillie</a>]

soup.select("#link1 + .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select(".sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("[class~=sister]")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("#link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("a#link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select("#link1,#link2")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select('a[href]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href="http://example.com/elsie"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select('a[href^="http://example.com/"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href$="tillie"]')
# [<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href*=".com/el"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

multilingual_markup = """
 <p lang="en">Hello</p>
 <p lang="en-us">Howdy, y'all</p>
 <p lang="en-gb">Pip-pip, old fruit</p>
 <p lang="fr">Bonjour mes amis</p>
"""
multilingual_soup = BeautifulSoup(multilingual_markup)
multilingual_soup.select('p[lang|=en]')
# [<p lang="en">Hello</p>,
#  <p lang="en-us">Howdy, y'all</p>,
#  <p lang="en-gb">Pip-pip, old fruit</p>]

select_one

返回查找到的元素的第一个

码农公寓

开始使用呢

解析器

四种对象

tag对象

标签名(name)

属性值(Attributes)

多值属性

内容

Comment对象

prettify()方法

find_all方法

contents属性

children属性

descendants属性

兄弟元素

next_sibling

previous_siblings

next_element

previous_element

过滤器find_all(flter)

字符串

正则表达式

列表

True/False

方法

总概

参数列表解读

class_

limit 参数

find

find_parents

find_parent

find_next_siblings，find_next_sibling

find_previous_siblings，find_previous_sibling

find_all_next，find_next

find_all_previous，find_previous

Css选择器

select

select_one

相关文章