## 前言 最近，我使用爬虫技术，爬取了美国航空航天局，也就是你电影里经常见到的 `NASA`, 火星探索的相关图片，有 `10000` 张吧。 嗯嗯，小事情，小事情。 完事儿之后，有点小激动，于是就有了这篇文章，将有以下内容： 1. 我为什么要爬取NASA的图片 2. 我是如何爬取NASA图片的（超详细） 3. 我得到了什么(高清大图) 3. 我发现了什么秘密（超劲爆） ## 我为什么要爬NASA的图片 我已经超过35了，瑟瑟发抖啥时候被开了。 天天想着万一哪天失业了干点啥，想着玩个自媒体吧，天天给大家瞎白话。白话点历史谜团，宇宙奥秘啥的，于是我就盯上了 `NASA`。 `NASA` 有各种宇宙探索任务，并且有相关的文章，访谈，图片，视频公开，这是不可多得的资源库。 ## 我是如何爬取NASA图片的（超详细） `NASA` 的网站是可以公开访问的，地址是 ```url https://www.nasa.gov/ ```  打开以后，它首页长这样，可以看到各种内容。右上角还有个搜索框，我们输入 `Mars` 也就是 `火星` 稍等片刻，会展示出与 `Mars` 相关的各种内容，其中有一项 `Mars Exploration` 也就是 `火星探索` 点开之后，就到了一个新页面，然后找到 `Images` 图片，就到了我们爬取的目标页 ```url https://www.nasa.gov/mission_pages/mars/images/index.html ```  页面下拉，你将会看到有个大大的按钮，上面写着 `MORE IMAGES`， 点击试试你就发现： 页面的内容，并不是页面直接加载的，而是通过 `api` 请求后，异步渲染的 F12, 打开浏览器开发者模式,重新执行刚才的步骤，观察请求信息，发现都会有如下的情况  看上去这个`url`地址非常重要，我们先看他的请求地址： ``` https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi ``` 注意看里面的参数 ``` size=24&from=24 ``` 很显然，`size` 就是每次请求图片的数量，`from` 经过试验，是查询初始位置，我们可以改它来获取其他内容 我们再来看看它的返回信息： ```json { "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 659, "max_score": null, "hits": [{ "_index": "nasa-public", "_type": "ubernode", "_id": "450040", "_score": null, "_source": { "image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ", "topics": ["3140", "3152"], "nid": "450040", "title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "type": "ubernode", "uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision", "collections": ["4525", "5246"], "link-or-attachment": "link", "missions": ["6336"], "primary-tag": "6336", "cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "promo-date-time": "2019-08-05T17:49:00-04:00", "secondary-tag": "3140", "master-image": { "fid": "603128", "alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "width": "1600", "id": "603128", "title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "uri": "public://thumbnails/image/pia23314-16.jpg", "height": "900" }, "ubernode-type": "image" }, "sort": [1565041740000] }, { "_index": "nasa-public", "_type": "ubernode", "_id": "433172", "_score": null, "_source": { "image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.", "topics": ["3152"], "nid": "433172", "title": "Opportunity Emerges in a Dusty Picture", "type": "ubernode", "uri": "/image-feature/opportunity-emerges-in-a-dusty-picture", "collections": ["7628"], "link-or-attachment": "link", "missions": ["3639"], "primary-tag": "3152", "cardfeed-title": "Opportunity Emerges in a Dusty Picture", "promo-date-time": "2018-09-26T12:39:00-04:00", "secondary-tag": "7628", "master-image": { "fid": "584263", "alt": "NASA's Opportunity rover appears as a blip in the center of this square", "width": "1400", "id": "584263", "title": "NASA's Opportunity rover appears as a blip in the center of this square", "uri": "public://thumbnails/image/pia22549-16.jpg", "height": "788" }, "ubernode-type": "image" }, "sort": [1537979940000] }] } } ``` 上面的 `json` 内容太长，我删除了一些重复的，实际上 `hits` 这个数组，也是24个，和页面上显示的图片数量是一样的。那基本可以断定，页面上的信息，就是从这个数组来的。 进而对比发现，`master-image`字段下，就是我们需要的信息，包括 `图片地址`, `图片尺寸`, `图片标题`。 下面就上代码，组装请求URL, 获取内容，下载图片 三步走 我使用了 `Dart` 语言，你们随意 ```dart import 'dart:convert'; import 'package:dio/dio.dart'; main() async { // 每页数量是固定24个,改变初始值即可 for (int from = 0; from < 24 * 100; from = from + 24) { await getPage(from); } } //获取每一页的信息并且下载 Future getPage(int from) async { String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' + from.toString() + '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi'; //获取到内容 var res = await Dio().get(url); var map = jsonDecode(res.toString()); (map['hits']['hits'] as List).forEach((element) async { Uri fileUri = Uri.parse(getUri(element)); String savePath = getSavePath(element); await Dio().downloadUri(fileUri, savePath); print('已下载: ' + savePath); }); } //获取图片下载地址 String getUri(dynamic element) { String uri = element['_source']['master-image']['uri'].toString(); uri = uri.replaceAll('public://', 'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/'); return uri; } //处理信息，并且返回图片保存地址 String getSavePath(dynamic element) { String id = element['_id']; String fid = element['_source']['master-image']['fid'].toString(); String title = element['_source']['master-image']['title'].toString(); String uri = element['_source']['master-image']['uri'].toString(); String savePath = id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last; savePath = savePath.replaceAll('/', ''); savePath = savePath.replaceAll('\\', ''); savePath = savePath.replaceAll('"', ''); savePath = 'images/' + savePath; return savePath; } ``` 上面的代码，还是很简单的，有经验的同学应该一看就懂。 走起来吧。 ``` 已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg 已下载: images/452007_605784_Danielson Crater on Mars.jpg 已下载: images/458478_615132_Gullies on Mars.jpg 已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg 已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg 已下载: images/470381_643473_CME.jpg 已下载: images/458813_615896_Mars.jpg 已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg 已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg 已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg 已下载: images/463659_626874_Avalanche on Mars.jpg 已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg 已下载: images/468636_639726_Mars' Jezero Crater.jpg ``` ## 我得到了什么 这些图片  还有这些  图片，图片标题都有了, 够看一个月了我估计。 ## 我发现了什么秘密  这张图，我最喜欢了。一个如此清晰，一个如此浑浊，这是为什么？ 火星人的裂缝产生器？ 好吧，真正的秘密是： `NASA` 的网站竟然没有防采集，不信你也试试。。。