Python:multiprocessing.shared memory

lastmod 2024-07-05created 2022-06-1315 edits

Provides shared memory for direct access across processes.

Usage

Producer Process:

from multiprocessing import shared_memory

server_sm = shared_memory.SharedMemory(create=True, size=10)
# use `server_sm.buf`
server_sm.close()
server_sm.unlink()

Consumer Process:

from multiprocessing import shared_memory

client_sm = shared_memory.SharedMemory(sm.name)
# use `client_sm.buf`
client_sm.close()

SharedMemory

Creates a new shared memory block or attaches to an existing shared memory block.

Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name, and a different process can attach to that same shared memory block using that same name.

As a resource for sharing data across processes, shared memory blocks may outlive the original process that created them. When one process no longer needs access to a shared memory block that might still be needed by other processes, the close() method should be called.

When a shared memory block is no longer needed by any process, the unlink() method should be called to ensure proper cleanup.

Simple Example

The following example demonstrates low-level use of SharedMemory instances:

>>> from multiprocessing import shared_memory
>>> shm_a = shared_memory.SharedMemory(create=True, size=10)
>>> type(shm_a.buf)
<class 'memoryview'>
>>> buffer = shm_a.buf
>>> len(buffer)
10
>>> buffer[:4] = bytearray([22, 33, 44, 55])  # Modify multiple at once
>>> buffer[4] = 100                           # Modify single byte at a time
>>> # Attach to an existing shared memory block
>>> shm_b = shared_memory.SharedMemory(shm_a.name)
>>> import array
>>> array.array('b', shm_b.buf[:5])  # Copy the data into a new array.array
array('b', [22, 33, 44, 55, 100])
>>> shm_b.buf[:5] = b'howdy'  # Modify via shm_b using bytes
>>> bytes(shm_a.buf[:5])      # Access via shm_a
b'howdy'
>>> shm_b.close()   # Close each SharedMemory instance
>>> shm_a.close()
>>> shm_a.unlink()  # Call unlink only once to release the shared memory

Numpy Shared Memory Example

The following example demonstrates a practical use of the SharedMemory class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells:

>>> # In the first Python interactive shell
>>> import numpy as np
>>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
>>> from multiprocessing import shared_memory
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> # Now create a NumPy array backed by shared memory
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]  # Copy the original data into shared memory
>>> b
array([1, 1, 2, 3, 5, 8])
>>> type(b)
<class 'numpy.ndarray'>
>>> type(a)
<class 'numpy.ndarray'>
>>> shm.name  # We did not specify a name so one was chosen for us
'psm_21467_46075'

>>> # In either the same shell or a new Python shell on the same machine
>>> import numpy as np
>>> from multiprocessing import shared_memory
>>> # Attach to the existing shared memory block
>>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
>>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
>>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
>>> c
array([1, 1, 2, 3, 5, 8])
>>> c[-1] = 888
>>> c
array([  1,   1,   2,   3,   5, 888])

>>> # Back in the first Python interactive shell, b reflects this change
>>> b
array([  1,   1,   2,   3,   5, 888])

>>> # Clean up from within the second Python shell
>>> del c  # Unnecessary; merely emphasizing the array is no longer used
>>> existing_shm.close()

>>> # Clean up from within the first Python shell
>>> del b  # Unnecessary; merely emphasizing the array is no longer used
>>> shm.close()
>>> shm.unlink()  # Free and release the shared memory block at the very end

Troubleshooting

There appear to be 1 leaked shared_memory objects to clean up at shutdown

The following UserWarning is printed when the program exits.

~: /usr/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

The problem situation is as follows.

The process that creates the shared memory uses code like the following.

server_sm = SharedMemory(create=True, size=10)  # register tracker 
# use `server_sm.buf`
server_sm.close()
server_sm.unlink()  # unregister tracker

Creating a SharedMemory registers a Tracker, and unlink() unregisters the Tracker.

However, the side that uses the shared memory (Consumer) does not call unlink(),

client_sm = SharedMemory(sm.name)  # register tracker 
client_sm.close()

so the Tracker is registered but there is no code that unregisters it, which causes the problem.

To solve this, either copy the original code as suggested here and register only when create=True in the constructor, or

(in my case) use a utility function like the following.

# -*- coding: utf-8 -*-

import os
from multiprocessing.shared_memory import SharedMemory


def _unregister_shared_memory_tracker(sm: SharedMemory) -> None:
    # https://bugs.python.org/issue39959
    if os.name != "nt":
        from multiprocessing.resource_tracker import unregister  # noqa

        unregister(getattr(sm, "_name"), "shared_memory")


class _AttachSharedMemoryContext:
    def __init__(self, name: str):
        self.name = name

    def __enter__(self) -> SharedMemory:
        self.sm = SharedMemory(name=self.name)
        return self.sm

    def __exit__(self, exc_type, exc_val, exc_tb):
        _unregister_shared_memory_tracker(self.sm)
        self.sm.close()


def attach_shared_memory(name: str):
    return _AttachSharedMemoryContext(name)

On the Consumer side, use it as follows:

with attach_shared_memory(name) as sm:
    # use `sm.buf`

Favorite site

multiprocessing.shared_memory — Provides shared memory for direct access across processes — Python 3.10.0 documentation

author Zer0 sitemap