Skip to content
Search

Python:multiprocessing

lastmod 2022-06-13created 2019-03-0617 edits

multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

The multiprocessing module also introduces APIs which do not have analogs in the threading module. A prime example of this is the Pool object, which offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism). The following example demonstrates the common practice of defining such functions in a module so that child processes can successfully import that module.

Example

This basic example of data parallelism uses Pool:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

will print to standard output

[1, 4, 9]

Shared Memory

See Python:multiprocessing.shared_memory.

Capture, Standard output

from multiprocessing import Process
import os
import sys

def info(title):
    print title
    print 'module name:', __name__
    print 'parent process:', os.getppid()
    print 'process id:', os.getpid()

def f(name):
    sys.stdout = open(str(os.getpid()) + ".out", "w")
    info('function f')
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    q = Process(target=f, args=('fred',))
    q.start()
    p.join()
    q.join()

Locking example

from multiprocessing import Process, Lock, Manager

class Worker:
    def __init__(self):
        self.cache = Manager().list()

    def pop_data(self, lock, redis):
        lock.acquire()
        while True:
            item = redis.pop_data()
            if item == None:  # 큐가 비워졌으면 끝
                item = None
                break
            if item in self.cache:
                print 'already finished! dump it = {}'.format(item)
                continue
            if len(self.cache) > 10:
                self.cache.pop(0)
            self.cache.append(item)
            break
        lock.release()
        return each_item

    def working(self, lock, redis):
        while True:
            data = self.pop_data(lock, redis)
            if not data:
                break
            ~~~~~  # 어떤 작업

def main(argv):
    redis = ~~~~
    process_list = []
    lock = Lock()
    process_count = 5
    for i in xrange(process_count):
        worker = Worker()
        ps = Process(target=worker.working, args=(lock, redis, i))
        process_list.append(ps)
        ps.start()

    for i in xrange(len(process_list)):
        process_list[i].join()

if '__name__' == '__main__':
    main()

Troubleshooting

Deadlock at the end of the program

When using multiprocessing.Queue, if the program deadlocks on exit, it is a good idea to check whether Queue.close() is used.

UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown

The following error occurs:

/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 10 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/psm_e27e5f9e': [Errno 2] No such file or directory: '/psm_e27e5f9e'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/usr/local/lib/python3.8/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/psm_2cf099ac': [Errno 2] No such file or directory: '/psm_2cf099ac'
<8 more similar messages omitted>

... Not solved yet.

See also

Favorite site

References


  1. On_Sharing_Large_Arrays_When_Using_Pythons_Multiprocessing.pdf ↩

author Zer0 sitemap