python scrapy 报错 raise ValueError(f'Missing scheme in request url:

class ImgSpider(scrapy.Spider):
    name = 'img'
    # allowed_domains = ['https://sc.chinaz.com/tupian/']
    start_urls = ['https://sc.chinaz.com/tupian/']

    def parse(self, response):
        res_list = response.xpath('//div[@id="container"]/div')
        for i in res_list:
            res_item = i.xpath('./div/a/img/@src2').extract_first()
            item = ImgsproItem()
            item['src'] = res_item
            yield item

错误提示如下

 raise ValueError(f'Missing scheme in request url: {self._url}')

原因分析:

  res_item得到的地址缺少域名,是不完整的url,为确保请求地址正确,可以使用urljoin()。

参见 https://*.com/questions/42026244/scrapy-valueerrormissing-scheme-in-request-url-s-self-url

代码修改如下后可正确执行

    def parse(self, response):
        res_list = response.xpath('//div[@id="container"]/div')
        for i in res_list:
            res_item = i.xpath('./div/a/img/@src2').extract_first()
            item = ImgsproItem()
            # item['src'] = res_item
            item['src'] = response.urljoin(res_item)
            yield item

 

上一篇:Address 2 OpenCV errors (raise.c and GTK) by reinstalling and rebuilding


下一篇:自动化维护任务 – Automated Maintenance Task (转)