利用python scrapy 框架抓取豆瓣小组数据

2021-08-14 19:08:10

因为最近在找房子在豆瓣小组-上海租房上找，发现搜索困难，于是想利用爬虫将数据抓取. 顺便熟悉一下Python.

这边有scrapy 入门教程出处：http://www.cnblogs.com/txw1958/archive/2012/07/16/scrapy-tutorial.html

差不多跟教程说的一样，问题技术难点是转码，上述教程并未详细指出. 我还是把代码贴出来，请供参考.

E:\tutorial>tree /f

Folder PATH listing for volume 文档

Volume serial number is -BBB3

E:.

│  scrapy.cfg

│

└─tutorial

    │  items.py

    │  items.pyc

    │  pipelines.py

    │  pipelines.pyc

    │  settings.py

    │  settings.pyc

    │  __init__.py

    │  __init__.pyc

    │

    └─spiders

            douban_spider.py

            douban_spider.pyc

            __init__.py

            __init__.pyc

item.py: 这有一篇很好介绍ITEM的文章（http://blog.csdn.net/iloveyin/article/details/41309609）

from scrapy.item import Item, Field

class DoubanItem(Item):

    title = Field()

    link = Field()

    #resp = Field()

    #dateT = Field()

pipelines.py #定义你自己的PipeLine方式，详细中文转码可在此处解决

# -*- coding: utf-8 -*-

# Don't forget to add your pipeline to the ITEM_PIPELINES setting

# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html

import json

import codecs  

class TutorialPipeline(object):

    def __init__(self):

        self.file = codecs.open('items.json', 'wb', encoding='gbk')

    def process_item(self, item, spider):

        line = json.dumps(dict(item)) + '\n'

        print line

        self.file.write(line.decode("unicode_escape"))

        return item

在setting.py 加入相应的 ITEM_PIPELINES 属性（红色字体为新加部分）

# -*- coding: utf-8 -*-

# Scrapy settings for tutorial project

#

# For simplicity, this file contains only the most important settings by

# default. All the other settings are documented here:

#

#     http://doc.scrapy.org/en/latest/topics/settings.html

#

BOT_NAME = 'tutorial'

SPIDER_MODULES = ['tutorial.spiders']

NEWSPIDER_MODULE = 'tutorial.spiders'

ITEM_PIPELINES = {

    'tutorial.pipelines.TutorialPipeline':300

}  

# Crawl responsibly by identifying yourself (and your website) on the user-agent

#USER_AGENT = 'tutorial (+http://www.yourdomain.com)'

接下来是spider.py

from scrapy.spider import BaseSpider

from scrapy.selector import HtmlXPathSelector

from scrapy.http import Request

from tutorial.items import DoubanItem

class DoubanSpider(BaseSpider):

    name = "douban"

    allowed_domains = ["douban.com"]

    start_urls = [

        "http://www.douban.com/group/shanghaizufang/discussion?start=0",

        "http://www.douban.com/group/shanghaizufang/discussion?start=25",

        "http://www.douban.com/group/shanghaizufang/discussion?start=50",

        "http://www.douban.com/group/shanghaizufang/discussion?start=75",

        "http://www.douban.com/group/shanghaizufang/discussion?start=100",

        "http://www.douban.com/group/shanghaizufang/discussion?start=125",

        "http://www.douban.com/group/shanghaizufang/discussion?start=150",

        "http://www.douban.com/group/shanghaizufang/discussion?start=175",

        "http://www.douban.com/group/shanghaizufang/discussion?start=200"

]

    def parse(self, response):

        hxs = HtmlXPathSelector(response)

        sites = hxs.xpath('//tr/td')

        items=[]

        for site in sites:

            item = DoubanItem()

            item['title'] =site.xpath('a/@title').extract()

            item['link'] = site.xpath('a/@href').extract()

           # item['resp'] = site.xpath('text()').extract()

           # item['dateT'] = site.xpath('text()').extract()

            items.append(item)

        return items

用JSON数据方式导出：

scrapy crawl douban -o items.json -t json

这有个JSON 转成CSV工具的网站，可以帮助转换：

https://json-csv.com/

结果效果展示,这样方便检索和过滤

码农公寓

相关文章