python并行多个线程和进程

工作站配置了20核CPU，平时运行程序只让一个CPU进行运转，浪费了很多时间。下面介绍同时启动多个CPU运行一个程序的方法：

一个进程（CPU）包含多个线程，线程并行的python库为threading，进程并行的库为multiprocessing。

父进程（主函数）运行结束后，如果子进程（子函数）还没有运行结束，需要使用join方法让父进程等待全部结束后再结束。

下面程序中，父进程每隔30s检查子进程的存留情况，等全部子进程运行结束后通过join结束整个程序。

并行多个线程的例子

 1 from threading import Thread #并行线程
 2 from threading import active_count
 3 from time import time, sleep
 4 
 5 def process(times,data):
 6     print("now is",times,"th process")
 7     sleep(15)
 8     f   = open(times,"w")
 9     f.write(data)
10     f.close()
11     print("OK!",times,"is finished")
12 
13 if __name__ == '__main__':
14     tstart    = time()
15     threads   = []
16     for i in range(5): #一个进程里并行5个线程
17         name='testdata{:02d}.txt'.format(i)
18         na  ='{:6.4f}'.format(i)
19         print(name)
20         t = Thread(target=process,args=(name,na))
21         threads.append(t)
22         t.setDaemon(True)
23         t.start()
24         sleep(2)
25     th_no = active_count()
26     while (th_no>1): #主线程监视间隔30s
27         sleep(30)
28         th_no = active_count()
29         trun = time()
30         print("now there are",th_no,"threads, time=",trun-tstart,"sec")
31     for t in threads:
32         t.join()
33     tend      = time()
34     print("sum up cost",tend-tstart,"sec")

 

并行多个进程的例子

CPU占用较低的小程序，并行多线程效率更高，CPU占用100%的大程序，并行多个进程效率更高，并行多个线程反而会使效率降低。

 1 from multiprocessing import Process #不同点1
 2 from multiprocessing import active_children #不同点2
 3 from time import time, sleep
 4 
 5 def process(times,data):
 6     print("now is",times,"th process")
 7     sleep(15)
 8     f   = open(times,"w")
 9     f.write(data)
10     f.close()
11     print("OK!",times,"is finished")
12 
13 if __name__ == '__main__':
14     tstart    = time()
15     threads   = []
16     for i in range(10): #数量要小于CPU核数
17         name='testdata{:02d}.txt'.format(i)
18         na  ='{:6.4f}'.format(i)
19         print(name)
20         t = Process(target=process,args=(name,na)) #与threading.Thread使用方法相似
21         threads.append(t)
22         t.start()
23         sleep(1.0)
24     pro_no = len(active_children()) #每隔30s父进程监视子进程留存数量
25     while (pro_no>1):
26         sleep(30)
27         pro_no = len(active_children())
28         trun     = time()
29         print("Running CPU NO=",pro_no," Running time=",trun-tstart,"sec")
30     for t in threads: #等待子进程全部结束开始join
31         t.join()
32     tend      = time()
33     print("sum up cost",tend-tstart,"sec")

参考官方文档：

并行多个进程 https://docs.python.org/zh-cn/3/library/multiprocessing.html

并行多个线程 https://docs.python.org/zh-cn/3/library/threading.html