Python的Web编程[0] -> Web客户端[1] -> Web 页面解析

2022-10-23 16:07:36

Web页面解析 / Web page parsing

1 HTMLParser解析

下面介绍一种基本的Web页面HTML解析的方式，主要是利用Python自带的html.parser模块进行解析。其主要步骤为:

创建一个新的Parser类，继承HTMLParser类;
重载handler_starttag等方法，实现指定功能;
实例化新的Parser并将HTML文本feed给类实例。

完整代码

 from html.parser import HTMLParser

 # An HTMLParser instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are encountered

 # Subclass HTMLParser and override its methods to implement the desired behavior

 class MyHTMLParser(HTMLParser):

     # attrs is the attributes set in HTML start tag

     def handle_starttag(self, tag, attrs):

         print('Encountered a start tag:', tag)

         for attr in attrs:

             print('     attr:', attr)

     def handle_endtag(self, tag):

         print('Encountered an end tag :', tag)

     def handle_data(self, data):

         print('Encountered some data  :', data)

 parser = MyHTMLParser()

 parser.feed('<html><head><title>Test</title></head>'

             '<body><h1>Parse me!</h1></body></html>'

             '<img src="python-logo.png" alt="The Python logo">')

代码中首先对模块进行导入，派生一个新的 Parser 类，随后重载方法，当遇到起始tag时，输出并判断是否有定义属性，有则输出，遇到终止tag与数据时同样输出。

Note: handle_starttag()函数的attrs为由该起始tag属性组成的元组元素列表，即列表中包含元组，元组中第一个参数为属性名，第二个参数为属性值。

输出结果

Encountered a start tag: html

Encountered a start tag: head

Encountered a start tag: title

Encountered some data  : Test

Encountered an end tag : title

Encountered an end tag : head

Encountered a start tag: body

Encountered a start tag: h1

Encountered some data  : Parse me!

Encountered an end tag : h1

Encountered an end tag : body

Encountered an end tag : html

Encountered a start tag: img

     attr: ('src', 'python-logo.png')

     attr: ('alt', 'The Python logo')

从输出中可以看到，解析器将HTML文本进行了解析，并且输出了tag中包含的属性。

2 BeautifulSoup解析

接下来介绍一种第三方的HTML页面解析包BeautifulSoup，同时与HTMLParser进行对比。

首先需要进行BeautifulSoup的安装，安装方式如下，

pip install beautifulsoup4

完整代码

 from html.parser import HTMLParser

 from io import StringIO

 from urllib import request

 from bs4 import BeautifulSoup, SoupStrainer

 from html5lib import parse, treebuilders

 URLs = ('http://python.org',

         'http://www.baidu.com')

 def output(x):

     print('\n'.join(sorted(set(x))))

 def simple_beau_soup(url, f):

     'simple_beau_soup() - use BeautifulSoup to parse all tags to get anchors'

     # BeautifulSoup returns a BeautifulSoup instance

     # find_all function returns a bs4.element.ResultSet instance,

     # which contains bs4.element.Tag instances,

     # use tag['attr'] to get attribute of tag

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib').find_all('a'))

 def faster_beau_soup(url, f):

     'faster_beau_soup() - use BeautifulSoup to parse only anchor tags'

     # Add find_all('a') function

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib', parse_only=SoupStrainer('a')).find_all('a'))

 def htmlparser(url, f):

     'htmlparser() - use HTMLParser to parse anchor tags'

     class AnchorParser(HTMLParser):

         def handle_starttag(self, tag, attrs):

             if tag != 'a':

                 return

             if not hasattr(self, 'data'):

                 self.data = []

             for attr in attrs:

                 if attr[0] == 'href':

                     self.data.append(attr[1])

     parser = AnchorParser()

     parser.feed(f.read())

     output(request.urljoin(url, x) for x in parser.data)

     print('DONE')

 def html5libparse(url, f):

     'html5libparse() - use html5lib to parser anchor tags'

     #output(request.urljoin(url, x.attributes['href']) for x in parse(f) if isinstance(x, treebuilders.etree.Element) and x.name == 'a')

 def process(url, data):

     print('\n*** simple BeauSoupParser')

     simple_beau_soup(url, data)

     data.seek(0)

     print('\n*** faster BeauSoupParser')

     faster_beau_soup(url, data)

     data.seek(0)

     print('\n*** HTMLParser')

     htmlparser(url, data)

     data.seek(0)

     print('\n*** HTML5lib')

     html5libparse(url, data)

     data.seek(0)

 if __name__=='__main__':

     for url in URLs:

         f = request.urlopen(url)

         data = StringIO(f.read().decode())

         f.close()

         process(url, data)

分段解释

首先将所需模块进行导入，其中StringIO模块用来实现字符串缓存容器，

 from html.parser import HTMLParser

 from io import StringIO

 from urllib import request

 from bs4 import BeautifulSoup, SoupStrainer

 from html5lib import parse, treebuilders

 URLs = ('http://python.org',

         'http://www.baidu.com')

接着定义一个输出函数，利用集合消除重复参数同时进行换行分离。

 def output(x):

     print('\n'.join(sorted(set(x))))

此处定义一个简单的bs解析函数，首先利用BeautifulSoup类传入HTML文本以及features(新版提示使用‘html5lib’)，生成一个BeautifulSoup实例，再利用find_all()函数返回所有tag为‘a’的链接锚集合类(bs4.element.Tag)，通过Tag获取href属性，最后利用urljoin函数生成链接并输出。

 def simple_beau_soup(url, f):

     'simple_beau_soup() - use BeautifulSoup to parse all tags to get anchors'

     # BeautifulSoup returns a BeautifulSoup instance

     # find_all function returns a bs4.element.ResultSet instance,

     # which contains bs4.element.Tag instances,

     # use tag['attr'] to get attribute of tag

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib').find_all('a'))

接着定义一个新的解析函数，这个函数可以通过参数传入parse_only来设置需要解析的锚标签，从而加快解析的速度。

Note: 这部分存在一个问题，当使用‘html5lib’特性时，是不支持parse_only参数的，因此会对整个标签进行搜索。有待解决。

 def faster_beau_soup(url, f):

     'faster_beau_soup() - use BeautifulSoup to parse only anchor tags'

     # Add find_all('a') function

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib', parse_only=SoupStrainer('a')).find_all('a'))

再定义一个用html方式进行解析的函数，可参见前节使用方式，首先建立一个锚解析的类，在遇到起始标签时，判断是否为‘a’锚，在进入时判断是否有data属性，没有的话初始化属性为空，随后对attrs参数遍历，获取href参数。最后生成实例并feed数据。

 def htmlparser(url, f):

     'htmlparser() - use HTMLParser to parse anchor tags'

     class AnchorParser(HTMLParser):

         def handle_starttag(self, tag, attrs):

             if tag != 'a':

                 return

             if not hasattr(self, 'data'):

                 self.data = []

             for attr in attrs:

                 if attr[0] == 'href':

                     self.data.append(attr[1])

     parser = AnchorParser()

     parser.feed(f.read())

     output(request.urljoin(url, x) for x in parser.data)

     print('DONE')

最后定义一个process函数，对于传入的data，每次使用完后都需要seek(0)将光标移回初始。

 def process(url, data):

     print('\n*** simple BeauSoupParser')

     simple_beau_soup(url, data)

     data.seek(0)

     print('\n*** faster BeauSoupParser')

     faster_beau_soup(url, data)

     data.seek(0)

     print('\n*** HTMLParser')

     htmlparser(url, data)

     data.seek(0)

     print('\n*** HTML5lib')

     html5libparse(url, data)

     data.seek(0)

最终解析的结果为网页内所有的链接。

 if __name__=='__main__':

     for url in URLs:

         f = request.urlopen(url)

         data = StringIO(f.read().decode())

         f.close()

         process(url, data)

运行输出结果

*** simple BeauSoupParser

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

*** faster BeauSoupParser

Warning (from warnings module):

  File "C:\Python35\lib\site-packages\bs4\builder\_html5lib.py", line 63

    warnings.warn("You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.")

UserWarning: You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

*** HTMLParser

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

DONE

*** HTML5lib

*** simple BeauSoupParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

*** faster BeauSoupParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

*** HTMLParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

DONE

*** HTML5lib

参考链接

《Python 核心编程第3版》

码农公寓

相关文章