python ThreadPoolExecutor使用记录

ThreadPoolExecutor 是python的一个线程池模块
官方文档:https://docs.python.org/3/library/concurrent.futures.html

from concurrent.futures import ThreadPoolExecutor, as_completed
通过submit | map 均可将执行函数提交到线程池中,map返回的是有序的结果
max_workers 定义线程数
python  ThreadPoolExecutor使用记录
默认是cpu + 4,最大32,可自定义

submit方式:
with ThreadPoolExecutor(max_works=16) as executor: futures = {executor.submit(func, arg):arg for arg in args} for future in as_completed(futures): future.result()

map方式:
with ThreadPoolExecutor(max_works=16) as executor: futures = executor.map(func, args) for future in futures: print(future)

submit&map均可添加多个参数,map需要联合repeat使用
from itertools import repeat

with ThreadPoolExecutor(max_workers=16) as executor:
futures = executor.map(func, args, repeat(arg1), repeat(arg2))
for index, future in enumerate(futures):
print(future)


待更新


上一篇:在线程池里面执行


下一篇:–spark submit的调优以及参数解析