以下命令都是在CMD中运行,首先把路径定位到项目文件夹
------------------------------------------------------------------------------------------
1. 创建一个scrapy project
scrapy startproject project_name
--------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
2. 自动生成spider代码命令
首先进入项目目录,然后敲入如下命令
scrapy genspider spider_name start_url
------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
3. 列出项目中所有spider
scrapy list
返回项目中所有爬虫的名字
-------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
4. 运行一个scrapy project,并且保存结果到csv文件
scrapy crawl spider_name -o file.csv
会通过yield命令生成iteration generator,把数据保存到file.csv文件中
---------------------------------------------------------------------------------------------
English Version
The Below commands will run at CMD which included python path in system env path.
Before run the commands, navigate to the directory where you want to create Scrapy project.
------------------------------------------------------------------------------------------
1. Createa a Scrapy project
scrapy startproject project_name
--------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
2. Create a spider. It will create a spider sub-class in your source code based on template.
Navigete to project directory, then key in below command
scrapy genspider spider_name start_url
------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
3. List all spiders in your project
scrapy list
It will return all spiders with spider name
-------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
4. Run a Scrapy project and save the result set into a CSV file.
scrapy crawl spider_name -o file.csv -t csv
Scrapy framework will make a iteration generator thru yield command in code and save your data into file.csv
---------------------------------------------------------------------------------------------