Python中的并行编程速度

  这里主要想记录下今天碰到的一个小知识点:Python中的并行编程速率如何?

  我想把AutoTool做一个并行化改造,主要目的当然是想提高多任务的执行速度。第一反应就是想到用多线程执行不同模块任务,但是在我收集Python多线程编程资料的时候发现一个非常奇怪的信息,那就是Python的多线程并不是真正的多线程,因为有一个GIL的存在(可以参考这篇文章讲解《Python最难的问题》)导致Python实际上默认(CPython解释器)只能是单线程执行。

  这里我写了一个例子可以看看:

 #!/usr/bin/env python
# -*- coding: utf-8 -*-
# @File : batch_swig_runner.py
# @Time : 2019/7/8 18:09
# @Author : KuLiuheng
# @Email : liuheng.klh@alibaba-inc.com from swig_runner import SwigRunner import time
import logging
from threading import Thread
from multiprocessing import Pool class TestRunner(Thread):
def __init__(self, name, path):
super(TestRunner, self).__init__()
self.name = name
self.path = path def run(self):
logging.warning("Message from the thread-%s START" % self.name)
for i in range(10000000): # 耗时操作模拟
j = int(i) * 10.1
# time.sleep(1)
logging.warning("Message from the thread-%s END" % self.name)
return self.path def multi_process(mname, mpath):
logging.warning("Message from the thread-%s START" % mname)
for i in range(10000000): # 耗时操作模拟
j = int(i) * 10.1
# time.sleep(1)
logging.warning("Message from the thread-%s END" % mname) class BatchSwigRunner(object):
def __init__(self, modules=None):
"""
用模块信息字典(工程名: 工程路径)来初始化
:param modules: {工程名: 工程路径}
"""
if modules is not None:
self._modules = modules
else:
self._modules = dict() def add_module_info(self, name, path):
self._modules[name] = path def start(self):
"""
启动批量任务执行,并返回执行过程中的错误信息
:return: list(工程序号,工程名称) 出错的工程信息列表
"""
runners = list()
for (project_name, project_path) in self._modules.items():
# logging.warning('BatchSwigRunner.start() [%s][%s]' % (project_name, project_path))
sub_runner = TestRunner(project_name, project_path)
sub_runner.daemon = True
sub_runner.start()
runners.append(sub_runner) for runner in runners:
runner.join() if __name__ == '__main__':
batch_runner = BatchSwigRunner()
batch_runner.add_module_info('name1', 'path1')
batch_runner.add_module_info('name2', 'path2')
batch_runner.add_module_info('name3', 'path3')
batch_runner.add_module_info('name4', 'path4')
start_time = time.time()
batch_runner.start() print 'Total time comsumed = %.2fs' % (time.time() - start_time) print('========================================')
start_time = time.time() for index in range(4):
logging.warning("Message from the times-%d START" % index)
for i in range(10000000): # 耗时操作模拟
j = int(i) * 10.1
# time.sleep(1)
logging.warning("Message from the times-%d END" % index) print '>>Total time comsumed = %.2fs' % (time.time() - start_time) print('----------------------------------------------')
start_time = time.time() pool = Pool(processes=4)
for i in range(4):
pool.apply_async(multi_process, ('name++%d' % i, 'path++%d' % i))
pool.close()
pool.join()
print '>>>> Total time comsumed = %.2fs' % (time.time() - start_time)

  看结果就发现很神奇的结论:

C:\Python27\python.exe E:/VirtualShare/gitLab/GBL-310/GBL/AutoJNI/autoTool/common/batch_swig_runner.py
WARNING:root:Message from the thread-name4 START
WARNING:root:Message from the thread-name2 START
WARNING:root:Message from the thread-name3 START
WARNING:root:Message from the thread-name1 START
WARNING:root:Message from the thread-name2 END
WARNING:root:Message from the thread-name4 END
WARNING:root:Message from the thread-name3 END
Total time comsumed = 15.92s
========================================
WARNING:root:Message from the thread-name1 END
WARNING:root:Message from the times-0 START
WARNING:root:Message from the times-0 END
WARNING:root:Message from the times-1 START
WARNING:root:Message from the times-1 END
WARNING:root:Message from the times-2 START
WARNING:root:Message from the times-2 END
WARNING:root:Message from the times-3 START
WARNING:root:Message from the times-3 END
>>Total time comsumed = 11.59s
----------------------------------------------
WARNING:root:Message from the thread-name++0 START
WARNING:root:Message from the thread-name++1 START
WARNING:root:Message from the thread-name++2 START
WARNING:root:Message from the thread-name++3 START
WARNING:root:Message from the thread-name++1 END
WARNING:root:Message from the thread-name++0 END
WARNING:root:Message from the thread-name++2 END
WARNING:root:Message from the thread-name++3 END
>>>> Total time comsumed = 5.69s Process finished with exit code 0

  其运行速度是(计算密集型):multiprocessing > normal > threading.Thread

  请注意这里用的是持续计算来模拟耗时操作:

for i in range(10000000):   # 耗时操作模拟
j = int(i) * 10.1

  如果用空等待(time.sleep(1)类似IO等待)来模拟耗时操作,那么结果就是(IO等待型):threading.Thread > multiprocessing > normal

C:\Python27\python.exe E:/VirtualShare/gitLab/GBL-310/GBL/AutoJNI/autoTool/common/batch_swig_runner.py
WARNING:root:Message from the thread-name4 START
WARNING:root:Message from the thread-name2 START
WARNING:root:Message from the thread-name3 START
WARNING:root:Message from the thread-name1 START
WARNING:root:Message from the thread-name3 END
WARNING:root:Message from the thread-name4 END
WARNING:root:Message from the thread-name2 END
WARNING:root:Message from the thread-name1 END
WARNING:root:Message from the times-0 START
Total time comsumed = 1.01s
========================================
WARNING:root:Message from the times-0 END
WARNING:root:Message from the times-1 START
WARNING:root:Message from the times-1 END
WARNING:root:Message from the times-2 START
WARNING:root:Message from the times-2 END
WARNING:root:Message from the times-3 START
WARNING:root:Message from the times-3 END
>>Total time comsumed = 4.00s
----------------------------------------------
WARNING:root:Message from the thread-name++0 START
WARNING:root:Message from the thread-name++1 START
WARNING:root:Message from the thread-name++2 START
WARNING:root:Message from the thread-name++3 START
WARNING:root:Message from the thread-name++0 END
WARNING:root:Message from the thread-name++1 END
WARNING:root:Message from the thread-name++2 END
WARNING:root:Message from the thread-name++3 END
>>>> Total time comsumed = 1.73s Process finished with exit code 0

  为何会有这样的结果呢?

(1)threading机制中因为GIL的存在,实际上是一把全局锁让多线程变成了CPU线性执行,只可能用到一颗CPU计算。当sleep这样是释放CPU操作发生时,可以迅速切换线程,切换速度可以接受(比multiprocessing快),比normal(阻塞等待)当然快的多;

(2)这里用了多进程Pool,可以真正意义上使用多CPU,对于CPU计算密集型的操作(上面的for循环计算)那么肯定是多核比单核快。所以就出现了第一种测试场景的结果。

上一篇:fir.im Weekly - 94 个 iOS 开发资源推荐


下一篇:【Leetcode】二叉树简单路径最大和问题