前言
今天的“受害者”为【猫耳FM】，一个音频网站
地址：https://www.missevan.com/sound/m/110
对于本篇文章有疑问的同学可以加【资料白嫖、解答交流群：1039649593】
知识点：
- requests
- time
- re
- concurrent.futures
开发环境：
- 版 本：anaconda5.2.0（python3.6.5）
- 编辑器：pycharm
【付费VIP完整版】只要看了就能学会的教程，80集Python基础入门视频教学
导入模块
import time import requests import concurrent.futures import re
通过函数式编程，实现各个功能模块
发送请求
def get_html(url):
response = requests.get(url)
return response
第一次解析
def parse(response):
mp3_ids = re.findall('<a target="_player" href="/sound/(.*?)" title=".*?">', response.text)
return mp3_ids
第二次解析
def parse_2(response):
json_data = response.json()
title = json_data['info']['sound']['soundstr']
soundurl = json_data['info']['sound']['soundurl']
return title, soundurl
保存数据
def save(title, mp3_data):
with open('mp3\\' + title + '.mp3', mode='wb') as f:
f.write(mp3_data)
print(title, '下载完成!!!')
修改标题
def change_title(title):
new_title = re.sub(r'[\//|:?<>"*]', '_', title)
return new_title
主函数，调用里面包含的整体连贯
# 1. 发送请求
response = get_html(url)
# 2. 解析数据 soundid
mp3_ids = parse(response)
for mp3_id in mp3_ids:
# 3. 请求另外详情页 地址拼接 https://www.missevan.com/sound/getsound?soundid=3922170
mp3_url = 'https://www.missevan.com/sound/getsound?soundid=' + mp3_id
resp_2 = get_html(mp3_url)
# 4. 解析音频url地址 音频标题
title, soundurl = parse_2(resp_2)
# 修改标题
title = change_title(title)
# 5. 请求音频url地址 音频 二进制数据 content
mp3_data = get_html(soundurl).content
# 6. 下载保存 到本地
save(title, mp3_data)
翻页
start_time = time.time()
for page in range(1, 5):
print(f'----------正在爬取第{page}页-------------')
run(f'https://www.missevan.com/sound/m?id=110&p={page}')
print('一共花费了:', time.time()-start_time)
多线程
if __name__ == '__main__':
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=1000) as executor:
for page in range(1, 5):
url = f'https://www.missevan.com/sound/m?id=110&p={page}'
executor.submit(run, url)
print('一共花费了:', time.time()-start_time)