给出了一个包含5万个网站网址的列表,我的任务是找出哪些网站是可以访问的.我们的想法是向每个URL发送HEAD请求并查看状态响应.从我听到的异步方法是要走的路,现在我正在使用asyncio和aiohttp.

我提出了以下代码,但速度非常糟糕.在我的10mbit连接上,1000个URL大约需要200秒.我不知道速度有多快,但我不熟悉Python中的异步编程,所以我觉得我在某处出错了.正如您所看到的,我已尝试将允许的同时连接数增加到1000(默认值为100),并且DNS解析的持续时间保留在缓存中;也没有任何重大影响.环境有Python 3.6和aiohttp 3.5.4.

与此问题无关的代码审查也值得赞赏.

import asyncio
import time
from socket import gaierror
from typing import List, Tuple

import aiohttp
from aiohttp.client_exceptions import TooManyRedirects

# Using a non-default user-agent seems to avoid lots of 403 (Forbidden) errors
HEADERS = {
    'user-agent': ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) '
                   'AppleWebKit/537.36 (KHTML, like Gecko) '
                   'Chrome/45.0.2454.101 Safari/537.36'),
}


async def get_status_code(session: aiohttp.ClientSession, url: str) -> Tuple[int, str]:
    try:
        # A HEAD request is quicker than a GET request
        resp = await session.head(url, allow_redirects=True, ssl=False, headers=HEADERS)
        async with resp:
            status = resp.status
            reason = resp.reason
        if status == 405:
            # HEAD request not allowed, fall back on GET
            resp = await session.get(
                url, allow_redirects=True, ssl=False, headers=HEADERS)
            async with resp:
                status = resp.status
                reason = resp.reason
        return (status, reason)
    except aiohttp.InvalidURL as e:
        return (900, str(e))
    except aiohttp.ClientConnectorError:
        return (901, "Unreachable")
    except gaierror as e:
        return (902, str(e))
    except aiohttp.ServerDisconnectedError as e:
        return (903, str(e))
    except aiohttp.ClientOSError as e:
        return (904, str(e))
    except TooManyRedirects as e:
        return (905, str(e))
    except aiohttp.ClientResponseError as e:
        return (906, str(e))
    except aiohttp.ServerTimeoutError:
        return (907, "Connection timeout")
    except asyncio.TimeoutError:
        return (908, "Connection timeout")


async def get_status_codes(loop: asyncio.events.AbstractEventLoop, urls: List[str],
                           timeout: int) -> List[Tuple[int, str]]:
    conn = aiohttp.TCPConnector(limit=1000, ttl_dns_cache=300)
    client_timeout = aiohttp.ClientTimeout(connect=timeout)
    async with aiohttp.ClientSession(
            loop=loop, timeout=client_timeout, connector=conn) as session:
        codes = await asyncio.gather(*(get_status_code(session, url) for url in urls))
        return codes


def poll_urls(urls: List[str], timeout=20) -> List[Tuple[int, str]]:
    """
    :param timeout: in seconds
    """
    print("Started polling")
    time1 = time.time()
    loop = asyncio.get_event_loop()
    codes = loop.run_until_complete(get_status_codes(loop, urls, timeout))
    time2 = time.time()
    dt = time2 - time1
    print(f"Polled {len(urls)} websites in {dt:.1f} seconds "
          f"at {len(urls)/dt:.3f} URLs/sec")
    return codes

解决方法:

现在,您立即启动所有请求.因此,可能瓶颈出现在某处.为避免这种情况,可以使用semaphore：

# code

sem = asyncio.Semaphore(200)


async def get_status_code(session: aiohttp.ClientSession, url: str) -> Tuple[int, str]:
    try:
        async with sem:
            resp = await session.head(url, allow_redirects=True, ssl=False, headers=HEADERS)
            # code

我按照以下方式测试了它：

poll_urls([
    'http://httpbin.org/delay/1' 
    for _ 
    in range(2000)
])

得到了：

Started polling
Polled 2000 websites in 13.2 seconds at 151.300 URLs/sec

虽然它请求单个主机,但它表明异步方法可以完成这项工作：13秒. &LT 2000秒

还有几件事可以做：

>您应该发挥信号量值以获得更好的性能
为您的具体环境和任务.
>尝试将超时从20降低到,比方说,5
秒：因为你只是在做头部请求所以不应该花太多时间
时间.如果请求挂起5秒,则很有可能不会
一点都没有.
>在脚本运行时监视系统资源(网络/ CPU / RAM)
可以帮助找出瓶颈是否仍然存在.
>顺便问一下,你安装了aiodns(如doc所示)？
> disabling ssl有什么改变吗？
>尝试启用logging的调试级别,以查看是否有任何有用的信息
>尝试设置client tracing,特别是测量每个请求步骤的时间,以查看哪些需要花费大部分时间

没有完全可重现的情况,很难说更多.