Scrapy

https://docs.scrapy.org/en/latest/intro/overview.html

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

安装

1、 安装python 2.7

2、安装 pip

https://jingyan.baidu.com/article/e73e26c0d94e0524adb6a7ff.html

进入命令行，然后把目录切换到python的安装目录下的Script文件夹下，运行 easy_inatall pip

3、 安装scrpay

https://docs.scrapy.org/en/latest/intro/install.html

pip install Scrapy

如果安装不成功，多尝试两次

运行报错：

ImportError： No module named win32api

解决办法：

https://blog.csdn.net/jianhong1990/article/details/46406499

下载安装： http://sourceforge.net/projects/pywin32/files%2Fpywin32/

运行

保存如下脚本为

quotes_spider.py

import scrapy

class QuotesSpider(scrapy.Spider):

    name = "quotes"

    start_urls = [

        'http://quotes.toscrape.com/tag/humor/',

    ]

    def parse(self, response):

        for quote in response.css('div.quote'):

            yield {

                'text': quote.css('span.text::text').extract_first(),

                'author': quote.xpath('span/small/text()').extract_first(),

            }

        next_page = response.css('li.next a::attr("href")').extract_first()

        if next_page is not None:

            yield response.follow(next_page, self.parse)

在此目录下，运行

scrapy runspider quotes_spider.py -o quotes.json

相关知识

CSS选择器

http://www.w3school.com.cn/cssref/css_selectors.asp

XPATH

http://www.cnblogs.com/fdszlzl/archive/2009/06/02/1494836.html

Python yield

https://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/