Python:multiprocessing
lastmod 2022-06-13created 2019-03-0617 edits
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
The multiprocessing module also introduces APIs which do not have analogs in the threading module. A prime example of this is the Pool object, which offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism). The following example demonstrates the common practice of defining such functions in a module so that child processes can successfully import that module.
Example
This basic example of data parallelism uses Pool:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
will print to standard output
Shared Memory
See Python:multiprocessing.shared_memory.
Capture, Standard output
from multiprocessing import Process
import os
import sys
def info(title):
print title
print 'module name:', __name__
print 'parent process:', os.getppid()
print 'process id:', os.getpid()
def f(name):
sys.stdout = open(str(os.getpid()) + ".out", "w")
info('function f')
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
q = Process(target=f, args=('fred',))
q.start()
p.join()
q.join()
Locking example
from multiprocessing import Process, Lock, Manager
class Worker:
def __init__(self):
self.cache = Manager().list()
def pop_data(self, lock, redis):
lock.acquire()
while True:
item = redis.pop_data()
if item == None: # 큐가 비워졌으면 끝
item = None
break
if item in self.cache:
print 'already finished! dump it = {}'.format(item)
continue
if len(self.cache) > 10:
self.cache.pop(0)
self.cache.append(item)
break
lock.release()
return each_item
def working(self, lock, redis):
while True:
data = self.pop_data(lock, redis)
if not data:
break
~~~~~ # 어떤 작업
def main(argv):
redis = ~~~~
process_list = []
lock = Lock()
process_count = 5
for i in xrange(process_count):
worker = Worker()
ps = Process(target=worker.working, args=(lock, redis, i))
process_list.append(ps)
ps.start()
for i in xrange(len(process_list)):
process_list[i].join()
if '__name__' == '__main__':
main()
Troubleshooting
Deadlock at the end of the program
When using multiprocessing.Queue, if the program deadlocks on exit, it is a good idea to check whether Queue.close() is used.
UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
The following error occurs:
/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 10 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/psm_e27e5f9e': [Errno 2] No such file or directory: '/psm_e27e5f9e'
warnings.warn('resource_tracker: %r: %s' % (name, e))
/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/psm_2cf099ac': [Errno 2] No such file or directory: '/psm_2cf099ac'
<8 more similar messages omitted>
... Not solved yet.
See also
Favorite site
- multiprocessing --- Process-based parallelism
- [Recommended] HAMA blog - Python concurrency programming - (4) multiprocessing
- (Python) Multiprocessing (Thread vs Process)
- (Python) Python multi-thread and multi-process
- Python Multi-Threading, Multi-Processing
- [Recommended] On Sharing Large Arrays When Using Python's Multiprocessing 1
References
-
On_Sharing_Large_Arrays_When_Using_Pythons_Multiprocessing.pdf ↩