分析Ajax请求并抓取今日头条街拍美图

2022-03-20 23:51:40

项目说明

本项目以今日头条为例，通过分析Ajax请求来抓取网页数据。

有些网页请求得到的HTML代码里面并没有我们在浏览器中看到的内容。这是因为这些信息是通过Ajax加载并且通过JavaScript渲染生成的，这时候就需要我们分析网页请求。

准备工作

python3、requests、Beautiful Soup、MongoDB、pymongo

抓取分析

在抓取之前首先分析抓取的逻辑，打开今日头条的首页https://www.toutiao.com/如图

右上角有一个搜索入口，这里尝试抓取街拍美图，所有输入“街拍”二字，搜索一下，结果如下图所示：

这时打开发者工具，查看所有网络请求，首先打开第一个网络请求，这个请求的URL就是当前的链接：https://www.toutiao.com/search/?keyword=街拍，

刷新一下界面，查看响应结果，如下没有找到页面上的内容

切换到XHR查看，找到了我们需要的信息

article_url就是内容详细的链接

再看一下Headers,这是我们需要构造的请求参数

进入内容详细页查看响应信息，找到了每个图片链接的页面是在Doc，这种形式的url就不能通过Beautiful Soup这样的解析库来解析，可以使用正则表达式re模块。

流程框架

1.抓取索引页内容

利用requests请求目标站点，得到索引页的HTML代码，返回结果。

2.抓取详情页内容

解析返回结果，得到详情页链接，并进一步抓取详情页内容。

3.下载图片与保存数据库

将图片下载至本地，并把页面信息以及图片URL保存至MongoDB。

4.开启循环及多线程

对多页面内容遍历，开启多线程提高爬取速度。

实战演练

第一次请求

第二次请求

刚才分析了Ajax请求的逻辑，下面就用程序来实现美图下载吧。

首先，实现方法get_page()来加载单个ajax请求的结果。其中可变化的参数就是offset和keyword，所以我们将它们当作参数传递，请求索引页0实现如下：

import requests

from urllib.parse import urlencode

from requests.exceptions import RequestException

def get_page_index(offset, keyword):

    parameters = {

        'offset': offset,

        'format': 'json',

        'keyword': keyword,

        'autoload': 'true',

        'count': '20',

        'cur_tab': '1',

        'from': 'search_tab'

    }

    url = 'https://www.toutiao.com/search_content/?' + urlencode(parameters)

    try:

        response = requests.get(url)

        if response.status_code == 200:

            return response.text

        return None

    except RequestException:

        print('请求索引页出错')

        return None

def main():

    html = get_page_index(0, '街拍')

    print(html)

if __name__ == '__main__':

    main()

这里我们用urlencode()方法构造请求的GRT参数，然后用requests请求这个链接，如果返回状态码为200，返回html结果

 {"count": , "action_label": "click_search", "return_count": , "has_more": , "page_id": "/search/", "request_id": "2018042818492217201614709945825F", "cur_tab": , "tab": {"tab_list": [{"tab_name": "\u7efc\u5408", "tab_id": , "tab_code": "news"}, {"tab_name": "\u89c6\u9891", "tab_id": , "tab_code": "video"}, {"tab_name": "\u56fe\u96c6", "tab_id": , "tab_code": "gallery"}, {"tab_name": "\u7528\u6237", "tab_id": , "tab_code": "pgc"}, {"tab_name": "\u95ee\u7b54", "tab_id": , "tab_code": "wenda"}], "cur_tab": }, "offset": , "action_label_web": "click_search", "show_tabs": , "data": [{"cell_type": , "key_info": {}, "keyinfo": {}, "display": {"status": , "term": "\u8857\u62cd", "search_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F", "title": "\u8def\u4eba\u8857\u62cd, \u97f5\u5473\u5341\u8db3\u7684\u5c11\u5987\u548c\u6e05\u7eaf\u53ef\u7231\u7684\u59b9\u5b50\u4f60\u66f4\u559c\u6b22\u54ea\u4e2a? \u65f6\u5c1a\u8fa3\u5988\u9ed1\u8272\u7d27\u8eab\u88e4\uff0c\u5e26\u7740\u592a\u9633\u5e3d\u9876\u7740\u70c8\u65e5\u51fa\u6765\u8fd0\u52a8\uff0c\u8fd9\u8eab\u88c5\u626e\u79d2\u6740\u4f17\u591a\u8def\u4eba \u5f02\u5e38\u4e30\u6ee1\u7684\u8fa3\u5988\uff0c\u628a\u4e0a\u8863\u6491\u5f97\u6ee1\u6ee1\u7684\uff0c\u770b\u4f3c\u4e0d\u4f11\u95f2\u7684\u88e4\u5b50\u540c\u6837\u7d27\u7d27\u7684\u3002 \u8857\u62cd\u65f6\u5c1a\uff1a\u6027\u611f\u70ed\u88e4\u7f8e\u5973\u6237\u5916\u73a9\u8857\u62cd \u6444\u5f71\u8857\u62cd-\u91cd\u5e86\u8857\u62cd\uff01", "total_count": , "queryLabel": "", "results": [{"original_page_url": "http://www.toutiao.com/a6446609483727667470", "img": "https://p3.pstatp.com/w640/31c600009c0969c4b980", "text": "\u8def\u4eba\u8857\u62cd, \u97f5\u5473\u5341\u8db3\u7684\u5c11\u5987\u548c\u6e05\u7eaf\u53ef\u7231\u7684\u59b9\u5b50\u4f60\u66f4\u559c\u6b22\u54ea\u4e2a?", "original_image_url": "", "width": , "score": 23.700184, "height": , "img_url": "https://p3.pstatp.com/w320/31c600009c0969c4b980", "id": "31c600009c0969c4b980", "page_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F&from_img_id=31c600009c0969c4b980"}, {"original_page_url": "http://www.toutiao.com/a6428929553491034369", "img": "https://p3.pstatp.com/w640/243800026fcf510fc0b7", "text": "\u65f6\u5c1a\u8fa3\u5988\u9ed1\u8272\u7d27\u8eab\u88e4\uff0c\u5e26\u7740\u592a\u9633\u5e3d\u9876\u7740\u70c8\u65e5\u51fa\u6765\u8fd0\u52a8\uff0c\u8fd9\u8eab\u88c5\u626e\u79d2\u6740\u4f17\u591a\u8def\u4eba", "original_image_url": "", "width": , "score": 28.470556, "height": , "img_url": "https://p3.pstatp.com/w320/243800026fcf510fc0b7", "id": "243800026fcf510fc0b7", "page_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F&from_img_id=243800026fcf510fc0b7"}, {"original_page_url": "http://www.toutiao.com/a6451333778619056397", "img": "https://p3.pstatp.com/w640/31e90003b9e4ffcca0e4", "text": "\u5f02\u5e38\u4e30\u6ee1\u7684\u8fa3\u5988\uff0c\u628a\u4e0a\u8863\u6491\u5f97\u6ee1\u6ee1\u7684\uff0c\u770b\u4f3c\u4e0d\u4f11\u95f2\u7684\u88e4\u5b50\u540c\u6837\u7d27\u7d27\u7684\u3002", "original_image_url": "", "width": , "score": 30.017776, "height": , "img_url": "https://p3.pstatp.com/w320/31e90003b9e4ffcca0e4", "id": "31e90003b9e4ffcca0e4", "page_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F&from_img_id=31e90003b9e4ffcca0e4"}, {"original_page_url": "http://www.toutiao.com/a6391287308302401793", "img": "https://p3.pstatp.com/w640/17f300012568f7cabf1a", "text": "\u8857\u62cd\u65f6\u5c1a\uff1a\u6027\u611f\u70ed\u88e4\u7f8e\u5973\u6237\u5916\u73a9\u8857\u62cd", "original_image_url": "", "width": , "score": 23.275711, "height": , "img_url": "https://p3.pstatp.com/w320/17f300012568f7cabf1a", "id": "17f300012568f7cabf1a", "page_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F&from_img_id=17f300012568f7cabf1a"}, {"original_page_url": "http://www.toutiao.com/a6455485530577568269", "img": "https://p3.pstatp.com/w640/320c0004a4736257757a", "text": "\u6444\u5f71\u8857\u62cd-\u91cd\u5e86\u8857\u62cd\uff01", "original_image_url": "", "width": , "score": 31.582335, "height": , "img_url": "https://p3.pstatp.com/w320/320c0004a4736257757a", "id": "320c0004a4736257757a", "page_url": "https://image.toutiao.com/toutiao/search?trigger_all=1&term=%E8%A1%97%E6%8B%8D&search_from=toutiaosearch&search_id=E6E40D0F15A429E53FF741BBE105D31F&from_img_id=320c0004a4736257757a"}], "queryLabelID": -, "message": ""}, "tokens": ["\u8857\u62cd"], "id_str": "", "ala_src": "image", "id": }, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6549135496977580551%2C%22source%22%3A%22%E8%80%83%E7%A0%94%E8%B7%AF%E4%B8%8A%E5%A5%8B%E6%96%97%E7%9A%84%E4%BA%8C%E7%8B%97%E5%AD%90%22%7D&gd_label=click_search&groupid=6549135496977580551&item_id=6549135496977580551", "media_name": "\u8003\u7814\u8def\u4e0a\u594b\u6597\u7684\u4e8c\u72d7\u5b50", "show_play_effective_count": , "item_source_url": "/group/6549135496977580551/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/152483930041467a89bbd8e"}, {"url": "//p3.pstatp.com/list/pgc-image/15248393003721597ed0a60"}, {"url": "//p3.pstatp.com/list/pgc-image/1524839300384b0afa5f3f1"}, {"url": "//p9.pstatp.com/list/pgc-image/1524839300323a289ed6c8d"}], "datetime": "2018-04-27 22:30:33", "more_mode": true, "create_time": "", "has_gallery": true, "id": "", "user_id": , "title": "\u4e0a\u6d77\u8857\u62cd\u56fe", "has_video": false, "share_url": "http://toutiao.com/group/6549135496977580551/", "source": "\u8003\u7814\u8def\u4e0a\u594b\u6597\u7684\u4e8c\u72d7\u5b50", "comment_count": , "article_url": "http://toutiao.com/group/6549135496977580551/", "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m1584754949706766/", "media_avatar_url": "//p3.pstatp.com/medium/54ea0000638f8d64f984", "middle_mode": false, "gallary_image_count": , "media_creator_id": , "tag_id": , "source_url": "/group/6549135496977580551/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6549135496977580551/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_fashion", "image_url": "//p1.pstatp.com/large/pgc-image/152483930041467a89bbd8e", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6549091673454936590%2C%22source%22%3A%22%E6%B5%B7%E7%8E%B2%E6%97%B6%E5%B0%9A%22%7D&gd_label=click_search&groupid=6549091673454936590&item_id=6549091673454936590", "media_name": "\u6d77\u73b2\u65f6\u5c1a", "show_play_effective_count": , "item_source_url": "/group/6549091673454936590/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1524829212175be45859842"}, {"url": "//p9.pstatp.com/list/pgc-image/1524829212466ebb4f98cd7"}, {"url": "//p3.pstatp.com/list/pgc-image/15248292128376adb9d7f4e"}, {"url": "//p3.pstatp.com/list/pgc-image/15248292131756d5985c6da"}], "datetime": "2018-04-27 19:40:29", "more_mode": true, "create_time": "", "has_gallery": true, "id": "", "user_id": , "title": "\u6b66\u6c49\u8857\u62cd\uff0c\u5929\u84dd\u8272\u725b\u4ed4\u886c\u8863\u642d\u914d\u7834\u6d1e\u725b\u4ed4\u88e4\uff0c\u7b80\u6d01\u5927\u65b9\u53c8\u767e\u642d\uff01", "has_video": false, "share_url": "http://toutiao.com/group/6549091673454936590/", "source": "\u6d77\u73b2\u65f6\u5c1a", "comment_count": , "article_url": "http://toutiao.com/group/6549091673454936590/", "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m1562530317664257/", "media_avatar_url": "//p3.pstatp.com/medium/382f0005b0e7dbc47d8d", "middle_mode": false, "gallary_image_count": , "media_creator_id": , "tag_id": , "source_url": "/group/6549091673454936590/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6549091673454936590/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_fashion", "image_url": "//p1.pstatp.com/large/pgc-image/1524829212175be45859842", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6549099779656253960%2C%22source%22%3A%22%E6%B5%B7%E7%8E%B2%E6%97%B6%E5%B0%9A%22%7D&gd_label=click_search&groupid=6549099779656253960&item_id=6549099779656253960", "media_name": "\u6d77\u73b2\u65f6\u5c1a", "show_play_effective_count": , "single_mode": true, "item_source_url": "/group/6549099779656253960/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/15248311067351840c1569b"}, {"url": "//p3.pstatp.com/list/pgc-image/1524831106800649768a475"}, {"url": "//p3.pstatp.com/list/pgc-image/15248311068265c8ee8baa9"}], "article_url": "http://toutiao.com/group/6549099779656253960/", "datetime": "2018-04-27 20:11:57", "more_mode": true, "create_time": "", "has_gallery": false, "id": "", "user_id": , "title": "\u5317\u4eac\u8857\u62cd\uff0c\u590f\u65e5\u4f11\u95f2\u7a7f\u642d\u98ce\u683c\uff0c\u4f60\u4eec\u66f4\u559c\u6b22\u54ea\u4e00\u6b3e\uff1f", "has_video": false, "share_url": "http://toutiao.com/group/6549099779656253960/", "source": "\u6d77\u73b2\u65f6\u5c1a", "comment_count": , "media_creator_id": , "comments_count": , "large_mode": false, "abstract": "\u817f\u578b\u5f88\u6f02\u4eae\uff0c\u7ef7\u5e26\u5143\u7d20\u7684\u52a0\u5165\u66f4\u663e\u7ea4\u957f\u6f02\u4eae\u7684\u5c0f\u59d0\u59d0\uff0c\u54ea\u4e2a\u89d2\u5ea6\u90fd\u597d\u770b\u6536\u8170\u7684\u8bbe\u8ba1\u66f4\u663e\u8170\u8eab\u5341\u8db3\u7684\u9ad8\u8170\u534a\u88d9\u6d6a\u6f2b\u53c8\u4f18\u96c5\u6d45\u8272\u7cfb\u7684\u540a\u5e26\u6de1\u96c5\u6e05\u65b0\uff0c\u5f20\u671b\u7684\u5154\u5b50\u66f4\u663e\u7075\u52a8\u9542\u7a7a\u534a\u88d9\u66f4\u5177\u8f7b\u76c8\u611f\uff0c\u7b11\u5bb9\u6cbb\u6108\u529b\u6ee1\u5206\u68a6\u5e7b\u98d8\u9038\u7684\u4e2d\u88d9\uff0c\u6210\u529f\u52fe\u52d2\u51fa\u66f2\u7ebf\u683c\u5b50\u886c\u886b\uff0c\u725b\u4ed4\u88e4\uff0c\u5c0f\u767d\u978b\uff0c\u9633\u5149\u707f\u70c2\u7f8e\u5c11\u5973\u957f\u53d1\u98d8\u98d8\u4e0d\u8bf4\uff0c\u8fd8\u662f\u4e00\u5934\u67d4\u987a\u7684\u76f4\u53d1", "media_url": "http://toutiao.com/m1562530317664257/", "media_avatar_url": "//p3.pstatp.com/medium/382f0005b0e7dbc47d8d", "middle_mode": false, "tag_id": , "source_url": "/group/6549099779656253960/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6549099779656253960/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_fashion", "image_url": "//p3.pstatp.com/large/pgc-image/15248311067351840c1569b", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"cell_type": , "key_info": {"keyword": "\u8857\u62cd"}, "keyinfo": {"keyword": "\u8857\u62cd"}, "display": {"items": [{"reason": "\u7f8e\u56fd\u65b0\u751f\u4ee3\u6a21\u7279", "name": "gigi hadid", "img": "http://p1.pstatp.com/large/5e3c001cc5114318f04e"}, {"reason": "90\u540e\u65b0\u9510\u65f6\u5c1a\u9020\u578b\u5e08", "name": "\u989c\u601d\u8bd7", "img": "http://p1.pstatp.com/large/5e3d0019e9b77a3b1894"}, {"reason": "\u827e\u6ecb\u5973\u4e8b\u4ef6\u5f53\u4e8b\u4eba", "name": "\u95eb\u5fb7\u5229", "img": "http://p1.pstatp.com/large/5e400003edc5a9661321"}, {"reason": "\u5185\u5730\u65b0\u56db\u5c0f\u82b1\u65e6\u4e4b\u4e00", "name": "\u502a\u59ae", "img": "http://p1.pstatp.com/large/5e3700181a5e1b2d1d2d"}, {"reason": "\u7ef4\u79d8\u5927\u79c0\u7684\u4e1c\u65b9\u8d85\u6a21", "name": "\u4f55\u7a57", "img": "http://p1.pstatp.com/large/5e43000444218771d3b6"}, {"reason": "\u56db\u5927\u65f6\u88c5\u5468\u7684\u660e\u661f", "name": "nick wooster", "img": "http://p1.pstatp.com/large/5e36001581e3bb2192da"}, {"reason": "\u7c73\u5170\u65f6\u5c1a\u535a\u4e3b", "name": "chiara ferragni", "img": "http://p1.pstatp.com/large/5e41000661e67cbab87f"}, {"reason": "90\u540e\u5308\u7259\u5229\u6a21\u7279\u6f14\u5458", "name": "\u82ad\u82ad\u62c9\u00b7\u5e15\u5c14\u6587", "img": "http://p1.pstatp.com/large/5e410000605c0acde5b0"}, {"reason": "\u66fe\u662f\u5168\u7403\u8d85\u7ea7\u540d\u6a21\u4e4b\u4e00", "name": "\u7c73\u5170\u8fbe\u00b7\u53ef\u513f", "img": "http://p1.pstatp.com/large/5e3c0006a46d74739566"}, {"reason": "\u6700\u96be\u88ab\u5fd8\u8bb0\u521d\u604b\u5973\u5076\u50cf", "name": "\u6797\u5141\u513f", "img": "http://p1.pstatp.com/large/5e420001e907713d2f36"}], "rank": , "title": "\u8857\u62cd_\u76f8\u5173\u4eba\u7269"}, "tokens": ["\u8857\u62cd"], "id_str": "", "ala_src": "knowledge_graph", "id": }, {"cell_type": , "key_info": {"keyword": "\u8857\u62cd"}, "keyinfo": {"keyword": "\u8857\u62cd"}, "display": {"hits": [{"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=a66300de8776429e8c2235e42bd7c611&line=0&app_id=13", "title": "\u563b\u563b \u4f60\u4eec\u5e73\u65f6\u5403\u65e9\u996d\u5417", "detail_schema": "sslocal://awemevideo?group_id=6529696611210104068&group_source=19&item_id=6529696611210104068&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=0", "digg_count": , "large_thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/6b06000de45d3ab5b19c~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/6b06000de45d3ab5b19c~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=33d41427a7df49a8b9923266ba663430&line=0&app_id=13", "title": "\u4f60\u7684\u7537\u670b\u53cb\u662f\u76d6\u4e16\u82f1\u96c4\u5417\uff1f", "detail_schema": "sslocal://awemevideo?group_id=6510541773645810948&group_source=19&item_id=6510541773645810948&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=1", "digg_count": , "large_thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/57ea0012c267a4463ad6~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/57ea0012c267a4463ad6~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=a61270a4633a45889d2291077d9fbac1&line=0&app_id=13", "title": "\u6bcf\u5929\u76f8\u7ea6\u592a\u53e4\u91cc\uff0c\u5c0f\u59d0\u59d0\u4eec\u9047\u5230\u597d\u5929\u6c14\u5c31\u5168\u90e8\u51fa\u95e8\u4e86\u3002@\u6296\u97f3\u5c0f\u52a9\u624b", "detail_schema": "sslocal://awemevideo?group_id=6519329479372836104&group_source=19&item_id=6519329479372836104&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=2", "digg_count": , "large_thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/60790000391f68072257~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/60790000391f68072257~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=65cc5a509b974e66ab7aeb7f6eb2d644&line=0&app_id=13", "title": "\u6f6e\u642d\u4e3a\u4f60\u7684\u5e05\u6c14\u548c\u7f8e\u4e3d\u52a0\u5206", "detail_schema": "sslocal://awemevideo?group_id=6527050972009598212&group_source=19&item_id=6527050972009598212&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=3", "digg_count": , "large_thumbnail_url": "http://sf6-ttcdn-tos.pstatp.com/img/mosaic-legacy/68fa0000fc1f7883a591~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf6-ttcdn-tos.pstatp.com/img/mosaic-legacy/68fa0000fc1f7883a591~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=dc3d16452a94414bbf5350d7ff11cb63&line=0&app_id=13", "title": "\u53c8\u662f\u6ee1\u6ee1\u7684\u72d7\u72fc \u5404\u4f4d\u4e0d\u8981\u4ecb\u610f\u5566 \u6bcf\u65e5\u5206\u4eab\u6f6e\u642d \u559c\u6b22\u70b9\u7231\u5fc3 \u8c22\u8c22", "detail_schema": "sslocal://awemevideo?group_id=6527871799374384392&group_source=19&item_id=6527871799374384392&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=4", "digg_count": , "large_thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/69cb000fde1f5ba5fe59~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/69cb000fde1f5ba5fe59~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=0d8df51ae9ad4cb699074a9383375ee2&line=0&app_id=13", "title": "\u6293\u7d27\u65f6\u95f4\u8c08\u4e00\u573a\u604b\u7231\u5427\uff01\u522b\u4eba\u90fd\u7a7f\u60c5\u4fa3\u88c5\u4e86~\u6bcf\u5929\u6492\u4e00\u6ce2\u72d7\u7cae", "detail_schema": "sslocal://awemevideo?group_id=6531270527519034631&group_source=19&item_id=6531270527519034631&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=5", "digg_count": , "large_thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/6d1a000d1610d1d853a8~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/6d1a000d1610d1d853a8~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=f0509c06fac742e2bb842963084254d0&line=0&app_id=13", "title": "2018\u6210\u90fd\u521d\u6625\u8857\u5934\u98ce \u8ddf\u95fa\u871c\u4e00\u8d77\u53bb\u5237\u8857\u5427", "detail_schema": "sslocal://awemevideo?group_id=6527869452183145742&group_source=19&item_id=6527869452183145742&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=6", "digg_count": , "large_thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/69cd000455374acb15a2~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/69cd000455374acb15a2~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=10d9be1b509c4e58aae9f874bb584d90&line=0&app_id=13", "title": "\u519c\u6751\u7248\u201c\u8857\u62cd\u201d\u3001\u54ea\u91cc\u4e0d\u4e00\u6837\ud83d\ude02\ud83d\ude02", "detail_schema": "sslocal://awemevideo?group_id=6525925958908644615&group_source=19&item_id=6525925958908644615&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=7", "digg_count": , "large_thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/683d000587b39f5eceea~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf3-ttcdn-tos.pstatp.com/img/mosaic-legacy/683d000587b39f5eceea~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=c43538ad1aaa44638ac32613b772269e&line=0&app_id=13", "title": "\u65e5\u7528\u54c1\u5355\u8bcd\u6765\u4e00\u6ce2\uff0c\u901b\u5b8c\u8d85\u5e02\u6211\u8981\u53bb\u5403\u996d\u5566@\u6296\u97f3\u5c0f\u52a9\u624b", "detail_schema": "sslocal://awemevideo?group_id=6528703733793033480&group_source=19&item_id=6528703733793033480&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=8", "digg_count": , "large_thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/6a750001c0a8f2ceec69~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf1-ttcdn-tos.pstatp.com/img/mosaic-legacy/6a750001c0a8f2ceec69~240x240.jpeg", "group_id": ""}, {"video_url": "https://aweme.snssdk.com/aweme/v1/play/?video_id=7fefcbd0bdea4f34a16055e21cf8a5eb&line=0&app_id=13", "title": "", "detail_schema": "sslocal://awemevideo?group_id=6523386823488572686&group_source=19&item_id=6523386823488572686&request_id=1524912562.52&log_pb=%7B%22impr_id%22%3A%221524912562.52%22%7D&gd_ext_json=%7B%22log_pb%22%3A%7B%22impr_id%22%3A%221524912562.52%22%7D%7D&category_name=__search__&load_more=7&keyword=%E8%A1%97%E6%8B%8D&size=10&offset=9", "digg_count": , "large_thumbnail_url": "http://sf6-ttcdn-tos.pstatp.com/img/mosaic-legacy/647e000de1e838e7291a~375x375.jpeg", "play_count": , "thumbnail_url": "http://sf6-ttcdn-tos.pstatp.com/img/mosaic-legacy/647e000de1e838e7291a~240x240.jpeg", "group_id": ""}], "abtest": -, "fromIndex": , "headers": {}, "statusCodeValue": , "hitCount": , "query": "\u8857\u62cd", "total": , "statusCode": "OK"}, "tokens": ["\u8857\u62cd"], "id_str": "", "ala_src": "hotsoon_video", "id": }, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%E5%8C%97%E4%BA%AC%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6335650371411902721%2C%22source%22%3A%22%E5%BE%AE%E8%A1%97%E6%8B%8D%22%7D&gd_label=click_search&groupid=6335650371411902721&item_id=6335653890435842562", "media_name": "\u5fae\u8857\u62cd", "show_play_effective_count": , "single_mode": true, "item_source_url": "/group/6335650371411902721/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/d290017ceeeed09d37e"}, {"url": "//p9.pstatp.com/list/ef500000fb4bee3f3fc"}, {"url": "//p1.pstatp.com/list/ef300002c052dbcdf55"}], "article_url": "http://toutiao.com/group/6335650371411902721/", "datetime": "2016-09-29 15:33:06", "more_mode": true, "create_time": "", "has_gallery": false, "id": "", "user_id": , "title": "\u8857\u62cd\uff1a\u5317\u4eac\u8857\u62cd\u7684\u7f8e\u5973\u4eec", "has_video": false, "share_url": "http://toutiao.com/group/6335650371411902721/", "source": "\u5fae\u8857\u62cd", "comment_count": , "media_creator_id": , "comments_count": , "large_mode": false, "abstract": "\u6709\u559c\u6b22\u7684\u5c31\u70b9\u4e2a\u8d5e\u54df", "media_url": "http://toutiao.com/m50923451971/", "middle_mode": false, "tag_id": , "source_url": "/group/6335650371411902721/", "item_id": "", "seo_url": "/group/6335650371411902721/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_photography", "image_url": "//p3.pstatp.com/large/d290017ceeeed09d37e", "has_image": true, "highlight": {"source": [[, ]], "abstract": [], "title": [[, ], [, ]]}, "group_id": ""}, {"merge_user": [{"screen_name": "\u7ca4B\u8857\u62cd", "verified_content": "", "media_id": "", "description": "\u5927\u5e05\u6bd4\u548c\u5c0f\u7f8e\u817b\u306e\u805a\u96c6\u5730\u3002", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "user_auth_info": {"auth_type": "", "other_auth": {"pgc": "\u5934\u6761\u53f7\u65f6\u5c1a\u539f\u521b\u4f5c\u8005"}, "auth_info": "\u5934\u6761\u53f7\u65f6\u5c1a\u539f\u521b\u4f5c\u8005"}, "avatar_url": "http://p3.pstatp.com/medium/2b7002107206ebc4a7e", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/6203908847/", "verify_content": "\u5934\u6761\u53f7\u65f6\u5c1a\u539f\u521b\u4f5c\u8005", "id": "", "name": "\u7ca4B\u8857\u62cd"}, {"screen_name": "\u7687\u57ce\u6839\u4e94\u7237\u539f\u521b\u8857\u62cd", "verified_content": "", "media_id": "", "description": "\u4eba\u50cf\u6349\u5f71 \u4eba\u6587\u7eaa\u5b9e \u65c5\u6e38\u98ce\u5149 \u9876\u7ea7\u539f\u521b\u8857\u62cd", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "user_auth_info": {"auth_type": "", "auth_info": "\u8857\u62cd\u8fbe\u4eba"}, "avatar_url": "http://p3.pstatp.com/medium/6cb000fd8de5eec5814", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/6685177416/", "verify_content": "\u8857\u62cd\u8fbe\u4eba", "id": "", "name": "\u7687\u57ce\u6839\u4e94\u7237\u539f\u521b\u8857\u62cd"}, {"screen_name": "3A\u8857\u62cd", "verified_content": "", "media_id": "", "description": "", "user_id": "", "gender": "", "follow_count": , "no_display_pgc_icon": "", "user_type": "", "is_pgc": , "user_auth_info": {"auth_type": "", "auth_info": "\u6444\u5f71\u8fbe\u4eba"}, "avatar_url": "http://p1.pstatp.com/medium/289a001a1b2868d356ea", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/62376452948/", "verify_content": "\u6444\u5f71\u8fbe\u4eba", "id": "", "user_verified": "", "name": "3A\u8857\u62cd"}, {"screen_name": "\u66b4\u8d70\u8857\u62cd", "verified_content": "", "media_id": "", "description": "\u66b4\u8d70\u8857\u62cd\u4e3b\u8981\u4ee5\u91c7\u8bbf\u9ad8\u989c\u503c\u5bf9\u8c61\u4e3a\u4e3b\uff0c\u98ce\u683c\u9752\u6625\u5411\u4e0a\uff0c\u8bdd\u9898\u6709\u8da3\u8d34\u8fd1\u751f\u6d3b\u3002", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "user_auth_info": {"auth_type": "", "auth_info": "\u641e\u7b11\u81ea\u5a92\u4f53"}, "avatar_url": "http://p1.pstatp.com/medium/10687/2312895944", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/5826291222/", "verify_content": "\u641e\u7b11\u81ea\u5a92\u4f53", "id": "", "name": "\u66b4\u8d70\u8857\u62cd"}, {"screen_name": "\u8857\u62cd\u7f8e\u5973", "media_id": "", "description": "\u7231\u8857\u62cd\uff0c\u5c55\u73b0\u65f6\u5c1a\u81ea\u7136\u7f8e\uff01", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "verified_content": "", "avatar_url": "http://p3.pstatp.com/medium/97e0016cb071b0c0037", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/50018626654/", "id": "", "name": "\u8857\u62cd\u7f8e\u5973"}, {"screen_name": "\u9f99\u54e5\u8857\u62cd", "media_id": "", "description": "\u7231\u7f8e\u4e4b\u5fc3\uff0c\u4eba\u7686\u6709\u4e4b\uff01", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "verified_content": "", "avatar_url": "http://p3.pstatp.com/medium/e5800033c20c4063fb9", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/20355086918/", "id": "", "name": "\u9f99\u54e5\u8857\u62cd"}, {"screen_name": "\u8857\u62cd\u7f8e\u7709", "media_id": "", "description": "\u5206\u4eab\u8857\u4e0a\u7684\u65f6\u5c1a\u7f8e\u5973\u5199\u771f", "user_id": "", "gender": "", "follow_count": , "no_display_pgc_icon": "", "user_type": "", "is_pgc": , "verified_content": "", "avatar_url": "http://p3.pstatp.com/medium/d29001188d400c269d9", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/6681524201/", "id": "", "user_verified": "", "name": "\u8857\u62cd\u7f8e\u7709"}, {"screen_name": "\u5c0f\u8d75\u8857\u62cd", "media_id": "", "description": "\u6700\u7f8e\u597d\u7684\u8857\u62cd\uff0c\u62cd\u6700\u7f8e\u7684\u57ce\u5e02", "user_id": "", "gender": "", "follow_count": , "user_verified": "", "user_type": "", "is_pgc": , "verified_content": "", "avatar_url": "http://p3.pstatp.com/medium/3e5d0000ccca9f91e2ea", "create_time": "", "user_decoration": {}, "highlight": {"name": [[, ]]}, "source_url": "http://www.toutiao.com/c/user/71172384643/", "id": "", "name": "\u5c0f\u8d75\u8857\u62cd"}], "cell_type": , "id": "", "keyword": "\u8857\u62cd"}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6549108208798007822%2C%22source%22%3A%22%E6%B5%B7%E7%8E%B2%E6%97%B6%E5%B0%9A%22%7D&gd_label=click_search&groupid=6549108208798007822&item_id=6549108208798007822", "media_name": "\u6d77\u73b2\u65f6\u5c1a", "show_play_effective_count": , "item_source_url": "/group/6549108208798007822/", "labels": [], "image_list": [{"url": "//p9.pstatp.com/list/pgc-image/1524833057239acf2788501"}, {"url": "//p3.pstatp.com/list/pgc-image/1524833057568fcdf4e9ccc"}, {"url": "//p3.pstatp.com/list/pgc-image/1524833057962e03b6c8077"}, {"url": "//p3.pstatp.com/list/pgc-image/152483305827851f7196167"}], "datetime": "2018-04-27 20:44:39", "more_mode": true, "create_time": "", "has_gallery": true, "id": "", "user_id": , "title": "\u6210\u90fd\u8857\u62cd\uff0c\u5c0f\u897f\u88c5\u5916\u5957\u5185\u642d\u7eb1\u7eb1\u8fde\u8863\u88d9\uff0c\u7a7f\u51fa\u6070\u5230\u597d\u5904\u7684\u5973\u4eba\u5473", "has_video": false, "share_url": "http://toutiao.com/group/6549108208798007822/", "source": "\u6d77\u73b2\u65f6\u5c1a", "comment_count": , "article_url": "http://toutiao.com/group/6549108208798007822/", "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m1562530317664257/", "media_avatar_url": "//p3.pstatp.com/medium/382f0005b0e7dbc47d8d", "middle_mode": false, "gallary_image_count": , "media_creator_id": , "tag_id": , "source_url": "/group/6549108208798007822/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6549108208798007822/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_fashion", "image_url": "//p3.pstatp.com/large/pgc-image/1524833057239acf2788501", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"cell_type": , "queries": [{"url": "/search/keyword=%E8%A1%97%E6%8B%8D+%E8%87%80", "text": "\u8857\u62cd \u81c0"}, {"url": "/search/keyword=%E8%A1%97%E6%8B%8D%E4%BD%8E%E8%85%B0%E8%A3%A4", "text": "\u8857\u62cd\u4f4e\u8170\u88e4"}, {"url": "/search/keyword=%E4%B8%89%E9%87%8C%E5%B1%AF%E8%A1%97%E6%8B%8D", "text": "\u4e09\u91cc\u5c6f\u8857\u62cd"}, {"url": "/search/keyword=%E8%B7%AF%E4%BA%BA%E8%A1%97%E6%8B%8D", "text": "\u8def\u4eba\u8857\u62cd"}, {"url": "/search/keyword=%E6%9D%AD%E5%B7%9E%E8%A1%97%E6%8B%8D", "text": "\u676d\u5dde\u8857\u62cd"}, {"url": "/search/keyword=%E4%B8%9D%E8%A2%9C%E7%BE%8E%E8%85%BF%E8%A1%97%E6%8B%8D", "text": "\u4e1d\u889c\u7f8e\u817f\u8857\u62cd"}, {"url": "/search/keyword=%E5%B0%8F%E8%9B%8B%E8%A1%97%E6%8B%8D", "text": "\u5c0f\u86cb\u8857\u62cd"}, {"url": "/search/keyword=%E8%A1%97%E6%8B%8D%E7%89%9B%E4%BB%94%E8%A3%A4", "text": "\u8857\u62cd\u725b\u4ed4\u88e4"}]}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6548772710850232835%2C%22source%22%3A%22%E6%B5%B7%E7%8E%B2%E6%97%B6%E5%B0%9A%22%7D&gd_label=click_search&groupid=6548772710850232835&item_id=6548772710850232835", "media_name": "\u6d77\u73b2\u65f6\u5c1a", "show_play_effective_count": , "item_source_url": "/group/6548772710850232835/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/15247549365224e90e72eeb"}, {"url": "//p1.pstatp.com/list/pgc-image/152475493712465f1647064"}, {"url": "//p3.pstatp.com/list/pgc-image/1524754937610124bb20c36"}, {"url": "//p1.pstatp.com/list/pgc-image/1524754938102349f2d2167"}], "datetime": "2018-04-26 23:02:45", "more_mode": true, "create_time": "", "has_gallery": true, "id": "", "user_id": , "title": "\u6210\u90fd\u8857\u62cd\uff0c\u4e00\u5b57\u80a9\u683c\u5b50\u4e0a\u8863\u642d\u914d\u725b\u4ed4\u534a\u8eab\u88d9\uff0c\u4f11\u95f2\u9752\u6625\u8303", "has_video": false, "share_url": "http://toutiao.com/group/6548772710850232835/", "source": "\u6d77\u73b2\u65f6\u5c1a", "comment_count": , "article_url": "http://toutiao.com/group/6548772710850232835/", "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m1562530317664257/", "media_avatar_url": "//p3.pstatp.com/medium/382f0005b0e7dbc47d8d", "middle_mode": false, "gallary_image_count": , "media_creator_id": , "tag_id": , "source_url": "/group/6548772710850232835/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6548772710850232835/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news", "image_url": "//p1.pstatp.com/large/pgc-image/15247549365224e90e72eeb", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6548322295658578435%2C%22source%22%3A%22%E8%80%83%E7%A0%94%E8%B7%AF%E4%B8%8A%E5%A5%8B%E6%96%97%E7%9A%84%E4%BA%8C%E7%8B%97%E5%AD%90%22%7D&gd_label=click_search&groupid=6548322295658578435&item_id=6548322295658578435", "media_name": "\u8003\u7814\u8def\u4e0a\u594b\u6597\u7684\u4e8c\u72d7\u5b50", "show_play_effective_count": , "item_source_url": "/group/6548322295658578435/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1524649871083e4d5656205"}, {"url": "//p3.pstatp.com/list/pgc-image/1524649871165e619bd74af"}, {"url": "//p3.pstatp.com/list/pgc-image/152464987107476f35dc15c"}, {"url": "//p3.pstatp.com/list/pgc-image/152464987108793a139aa07"}], "datetime": "2018-04-25 17:54:54", "more_mode": true, "create_time": "", "has_gallery": true, "id": "", "user_id": , "title": "\u5317\u4eac\u5de5\u4f53\u65c1\u8857\u62cd", "has_video": false, "share_url": "http://toutiao.com/group/6548322295658578435/", "source": "\u8003\u7814\u8def\u4e0a\u594b\u6597\u7684\u4e8c\u72d7\u5b50", "comment_count": , "article_url": "http://toutiao.com/group/6548322295658578435/", "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m1584754949706766/", "media_avatar_url": "//p3.pstatp.com/medium/54ea0000638f8d64f984", "middle_mode": false, "gallary_image_count": , "media_creator_id": , "tag_id": , "source_url": "/group/6548322295658578435/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6548322295658578435/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_photography", "image_url": "//p1.pstatp.com/large/pgc-image/1524649871083e4d5656205", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6548780299155669507%2C%22source%22%3A%22%E6%B5%B7%E7%8E%B2%E6%97%B6%E5%B0%9A%22%7D&gd_label=click_search&groupid=6548780299155669507&item_id=6548780299155669507", "media_name": "\u6d77\u73b2\u65f6\u5c1a", "show_play_effective_count": , "single_mode": true, "item_source_url": "/group/6548780299155669507/", "labels": [], "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/152475672212610f6143723"}, {"url": "//p3.pstatp.com/list/pgc-image/15247567221608eac6fdfbb"}, {"url": "//p9.pstatp.com/list/pgc-image/1524756722129cf741b0682"}], "article_url": "http://toutiao.com/group/6548780299155669507/", "datetime": "2018-04-26 23:32:12", "more_mode": true, "create_time": "", "has_gallery": false, "id": "", "user_id": , "title": "\u91cd\u5e86\u8857\u62cd\uff0c\u6ce2\u70b9\u96ea\u7eba\u886c\u886b\u642d\u914d\u9c7c\u5c3e\u88d9\uff0c\u5973\u4eba\u5473\u5341\u8db3\uff01", "has_video": false, "share_url": "http://toutiao.com/group/6548780299155669507/", "source": "\u6d77\u73b2\u65f6\u5c1a", "comment_count": , "media_creator_id": , "comments_count": , "large_mode": false, "abstract": "\u6ce2\u70b9\u96ea\u7eba\u886c\u886b\u642d\u914d\u9c7c\u5c3e\u88d9\uff0c\u5305\u5305\u4e0e\u9ad8\u8ddf\u978b\u76f8\u547c\u5e94\uff0c\u770b\u6765\u7ea2\u8272\u662f\u91cd\u70b9\u5462\uff0c\u590d\u53e4\u7684\u58a8\u955c\uff0c\u4f7f\u6574\u4f53\u98ce\u683c\u5f88\u660e\u663e\uff0c\u6000\u65e7\u98ce\u3002", "media_url": "http://toutiao.com/m1562530317664257/", "media_avatar_url": "//p3.pstatp.com/medium/382f0005b0e7dbc47d8d", "middle_mode": false, "tag_id": , "source_url": "/group/6548780299155669507/", "item_id": "", "user_auth_info": {}, "seo_url": "/group/6548780299155669507/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "news_fashion", "image_url": "//p1.pstatp.com/large/pgc-image/152475672212610f6143723", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ]]}, "group_id": ""}, {"open_url": "sslocal://detail?aggr_type=0&article_type=0&gd_ext_json=%7B%22city%22%3A%22%E9%87%8D%E5%BA%86%22%2C%22log_pb%22%3A%7B%22impr_id%22%3A%222018042818492217201614709945825F%22%7D%2C%22query%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22search_result_id%22%3A6340143356450242818%2C%22source%22%3A%22%E4%BC%98%E9%9B%85%E4%B8%8D%E5%A4%B1%E9%9F%B5%E5%91%B3%22%7D&gd_label=click_search&groupid=6340143356450242818&item_id=6339411492097491457", "media_name": "\u4f18\u96c5\u4e0d\u5931\u97f5\u5473", "show_play_effective_count": , "single_mode": true, "item_source_url": "/group/6340143356450242818/", "labels": [], "image_list": [{"url": "//p9.pstatp.com/list/e9500067e99c75d3e94"}, {"url": "//p1.pstatp.com/list/efc0004827df0ab76f4"}, {"url": "//p3.pstatp.com/list/e9500067e9ae31992e9"}], "article_url": "http://toutiao.com/group/6340143356450242818/", "datetime": "2016-10-11 18:08:55", "more_mode": true, "create_time": "", "has_gallery": false, "id": "", "user_id": , "title": "\u8857\u62cd\uff1a\u91cd\u5e86\u7f8e\u5973\u8857\u62cd\u7f8e\u817f\u5982\u6797", "has_video": false, "share_url": "http://toutiao.com/group/6340143356450242818/", "source": "\u4f18\u96c5\u4e0d\u5931\u97f5\u5473", "comment_count": , "media_creator_id": , "comments_count": , "large_mode": false, "abstract": "", "media_url": "http://toutiao.com/m50251835214/", "middle_mode": false, "tag_id": , "source_url": "/group/6340143356450242818/", "item_id": "", "seo_url": "/group/6340143356450242818/", "keyword": "\u8857\u62cd", "behot_time": "", "tag": "", "image_url": "//p3.pstatp.com/large/e9500067e99c75d3e94", "has_image": true, "highlight": {"source": [], "abstract": [], "title": [[, ], [, ]]}, "group_id": ""}, {"cell_type": , "key_info": {}, "keyinfo": {}, "display": {"url": "https://baike.sogou.com/m/fullLemma?ch=jrtt.search.item&cid=xm.click&lid=126224", "source": "\u641c\u72d7\u767e\u79d1", "pic": "http://pic.baike.soso.com/ugc/baikepic2/16309/20170825180937-1646773836_jpg_500_755_38306.jpg/300", "lemmaTitle": "\u8857\u62cd", "picAbs": "\u8857\u62cd\u662f\u4e00\u79cd\u6e90\u4e8e\u6b27\u7f8e\u56fd\u5bb6\u7684\u6587\u5316\uff0c\u6700\u65e9\u662f\u6e90\u4e8e\u65f6\u5c1a\u6742\u5fd7\u7684\u9700\u6c42\uff0c\u7528\u76f8\u673a\u6355\u6349\u8857\u4e0a\u7684\u65f6\u5c1a\u5143\u7d20\uff0c\u8fd8\u8981\u4f20\u9012\u6765\u81ea\u6c11\u95f4\u7684\u6d41\u884c\u4fe1\u606f\uff0c\u4e8e\u662f\u6240\u8c13\u201c\u8857\u5934\u79c0\u201d\u5c31\u5e94\u8fd0\u800c\u751f\u3002\u4e00\u822c\u6765\u8bf4\uff0c\u4e00\u6b21\u4e13\u4e1a\u7684\u8857\u62cd\uff0c\u5fc5\u987b\u6355\u6349\u5230\u88ab\u62cd\u8005\u8863\u670d\u7684\u7ec6\u8282\u3001\u642d\u914d\u548c\u724c\u5b50\u3002\u5982\u4eca\uff0c\u201c\u8857\u62cd\u201d\u6d3b\u52a8\u6b63\u9010\u6b65\u6210\u4e3a\u56fd\u5185\u5e74\u8f7b\u4eba\u4e00\u9879\u65b0\u7684\u8857\u5934\u6587\u5316\u6d3b\u52a8\u3002"}, "tokens": ["\u8857\u62cd"], "id_str": "", "ala_src": "sogou_baike", "id": }, {"cell_type": , "queries": [{"url": "/search/keyword=%E8%A1%97%E6%8B%8D%E4%BD%8E%E8%85%B0%E8%A3%A4", "text": "\u8857\u62cd\u4f4e\u8170\u88e4"}, {"url": "/search/keyword=%E4%B8%89%E9%87%8C%E5%B1%AF%E8%A1%97%E6%8B%8D", "text": "\u4e09\u91cc\u5c6f\u8857\u62cd"}, {"url": "/search/keyword=%E8%B7%AF%E4%BA%BA%E8%A1%97%E6%8B%8D", "text": "\u8def\u4eba\u8857\u62cd"}, {"url": "/search/keyword=%E6%9D%AD%E5%B7%9E%E8%A1%97%E6%8B%8D", "text": "\u676d\u5dde\u8857\u62cd"}, {"url": "/search/keyword=%E4%B8%9D%E8%A2%9C%E7%BE%8E%E8%85%BF%E8%A1%97%E6%8B%8D", "text": "\u4e1d\u889c\u7f8e\u817f\u8857\u62cd"}, {"url": "/search/keyword=%E5%B0%8F%E8%9B%8B%E8%A1%97%E6%8B%8D", "text": "\u5c0f\u86cb\u8857\u62cd"}, {"url": "/search/keyword=%E8%A1%97%E6%8B%8D%E7%89%9B%E4%BB%94%E8%A3%A4", "text": "\u8857\u62cd\u725b\u4ed4\u88e4"}, {"url": "/search/keyword=%E8%A1%97%E6%8B%8D%E9%AB%98%E8%B7%9F%E9%9E%8B", "text": "\u8857\u62cd\u9ad8\u8ddf\u978b"}]}], "message": "success", "action_label_pgc": "click_search"}

返回结果

可以看到，返回的结果是json的格式，需要使用python的json模块序列化然后解析出每个详情页url

def parse_page_index(html):

    data = json.loads(html)

    if data and 'data' in data.keys():

        for item in data.get('data'):

            yield item.get('article_url')

下一步请求每个详情页

def get_page_detail(url):

    try:

        response = requests.get(url)

        if response.status_code == 200:

            return response.text

        return None

    except RequestException:

        print('请求详情页出错')

        return None

然后解析每个详情页中的内容

则调用response的json()方法将结果转为JSON格式，然后返回。

接下来，再实现一个解析方法，：提取每条数据的image_detail字段中的每一张图片链接，将图片链接和图片所属的标题一并返回，此时可以构造一个生成器，实现代码如下：

码农公寓

项目说明

准备工作

抓取分析

流程框架

实战演练

相关文章