使用scrapy框架爬取图片网全站图片(二十多万张)，并打包成exe可执行文件

2022-09-27 07:32:15

目标网站：https://www.mn52.com/

本文代码已上传至git和百度网盘，链接分享在文末

网站概览

目标，使用scrapy框架抓取全部图片并分类保存到本地。

1.创建scrapy项目

scrapy startproject images

2.创建spider

cd images

scrapy genspider mn52 www.mn52.com

创建后结构目录如下

3.定义item定义爬取字段

# -*- coding: utf-8 -*-

# Define here the models for your scraped items

#

# See documentation in:

# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy

class ImagesItem(scrapy.Item):

    # define the fields for your item here like:

    # name = scrapy.Field()

    image_url = scrapy.Field()

    field_name = scrapy.Field()

    detail_name = scrapy.Field()

这里定义三个字段，分别为图片地址，图片类别名称和图片详细类别名称

4.编写spider

# -*- coding: utf-8 -*-

import scrapy

import requests

from lxml import etree

from images.items import ImagesItem

class Mn52Spider(scrapy.Spider):

    name = 'mn52'

    # allowed_domains = ['www.mn52.com']

    # start_urls = ['http://www.mn52.com/']

    def start_requests(self):

        response = requests.get('https://www.mn52.com/')

        result = etree.HTML(response.text)

        li_lists = result.xpath('//*[@id="bs-example-navbar-collapse-2"]/div/ul/li')

        for li in li_lists:

            url = li.xpath('./a/@href')[0]

            field_name = li.xpath('./a/text()')[0]

            print('https://www.mn52.com' + url,field_name)

            yield scrapy.Request('https://www.mn52.com' + url,meta={'field_name':field_name},callback=self.parse)

    def parse(self, response):

        field_name = response.meta['field_name']

        div_lists = response.xpath('/html/body/section/div/div[1]/div[2]/div')

        for div_list in div_lists:

            detail_urls = div_list.xpath('./div/a/@href').extract_first()

            detail_name = div_list.xpath('./div/a/@title').extract_first()

            yield scrapy.Request(url='https:' + detail_urls,callback=self.get_image,meta={'detail_name':detail_name,'field_name':field_name})

        url_lists = response.xpath('/html/body/section/div/div[3]/div/div/nav/ul/li')

        for url_list in url_lists:

            next_url = url_list.xpath('./a/@href').extract_first()

            if next_url:

                yield scrapy.Request(url='https:' + next_url,callback=self.parse,meta=response.meta)

    def get_image(self,response):

        field_name = response.meta['field_name']

        image_urls = response.xpath('//*[@id="originalpic"]/img/@src').extract()

        for image_url in image_urls:

            item = ImagesItem()

            item['image_url'] = 'https:' + image_url

            item['field_name'] = field_name

            item['detail_name'] = response.meta['detail_name']

            # print(item['image_url'],item['field_name'],item['detail_name'])

            yield item

逻辑思路：spider中重写start_requests获取起始路由，这里起始路由设为图片的一级分类如性感美女，清纯美女，萌宠图片等的地址，使用requests库请求mn52.com来解析获取。然后把url和一级分类名称传递给parse，在parse中解析获取图片的二级分类地址和二级分类名称，并判断是否含有下一页信息，这里使用scrapy框架自动去重原理，所以不用再写去重逻辑。获取到的数据传递给get_image，在get_image中解析出图片路由并赋值到item字段中然后yield item.

5.pipelines管道文件下载图片

# -*- coding: utf-8 -*-

# Define your item pipelines here

#

# Don't forget to add your pipeline to the ITEM_PIPELINES setting

# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

import scrapy

import os

# 导入scrapy 框架里的 管道文件的里的图像 图像处理的专用管道文件

from scrapy.pipelines.images import ImagesPipeline

# 导入图片路径名称

from images.settings import IMAGES_STORE as images_store

class Image_down(ImagesPipeline):

    def get_media_requests(self, item, info):

        yield scrapy.Request(url=item['image_url'])

    def item_completed(self, results, item, info):

        print(results)

        image_url = item['image_url']

        image_name = image_url.split('/')[-1]

        old_name_list = [x['path'] for t, x in results if t]

        # 真正的原图片的存储路径

        old_name = images_store + old_name_list[0]

        image_path = images_store + item['field_name'] + '/' + item['detail_name'] + '/'

        # 判断图片存放的目录是否存在

        if not os.path.exists(image_path):

            # 根据当前页码创建对应的目录

            os.makedirs(image_path)

        # 新名称

        new_name = image_path + image_name

        # 重命名

        os.rename(old_name, new_name)

        return item

分析：scrapy内置的ImagesPipeline自己生成经过md5加密的图片名称，这里导入os模块来使图片下载到自定义的文件夹和名称，需要继承ImagesPipeline类。

6.settings中设置所需参数（部分）

#USER_AGENT = 'images (+http://www.yourdomain.com)'

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'

#指定输出某一种日志信息

LOG_LEVEL = 'ERROR'

#将日志信息存储到指定文件中，不在终端输出

LOG_FILE = 'log.txt'

# Obey robots.txt rules

ROBOTSTXT_OBEY = False

IMAGES_STORE = './mn52/'

ITEM_PIPELINES = {

   # 'images.pipelines.ImagesPipeline': 300,

   'images.pipelines.Image_down': 301,

}

说明：settings中主要定义了图片的存储路径，user_agent信息，打开下载管道。

7.创建crawl.py文件，运行爬虫

from scrapy.crawler import CrawlerProcess

from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

# mn52是爬虫名

process.crawl('mn52')

process.start()

这里没有使用cmdline方法运行程序，因为涉及到后面的打包问题，使用cmdline方法无法打包。

8.至此，scrapy爬虫已编写完毕，右键运行crawl文件启动爬虫，图片就会下载。

控制台输出：

打开文件夹查看图片

全站下载完成的话大概有22万张图片

9.打包scrapy爬虫。

scrapy爬虫作为一个框架爬虫也是可以被打包成exe文件的，是不是很神奇/

码农公寓

相关文章