Skip to content
Search

CuPy

lastmod 2024-08-13created 2024-07-1142 edits

CuPy is a NumPy/SciPy-compatible array library for GPU-accelerated computing with Python.

CuPy acts as a drop-in replacement to run existing NumPy/SciPy code on NVIDIA CUDA or AMD ROCm platforms.

Categories

Install

Requirements

NVIDIA CUDA GPU with the Compute Capability 3.0 or larger.

  • CUDA Toolkit: v11.2 / v11.3 / v11.4 / v11.5 / v11.6 / v11.7 / v11.8 / v12.0 / v12.1 / v12.2 / v12.3 / v12.4
    • If you have multiple versions of CUDA Toolkit installed, CuPy will automatically choose one of the CUDA installations. See Working with Custom CUDA Installation for details.
    • This requirement is optional if you install CuPy from conda-forge. However, you still need to have a compatible driver installed for your GPU. See Installing CuPy from Conda-Forge for details.
  • Python: v3.9 / v3.10 / v3.11 / v3.12

NOTE

Currently, CuPy is tested against Ubuntu 20.04 LTS / 22.04 LTS (x86_64), CentOS 7 / 8 (x86_64) and Windows Server 2016 (x86_64).

Python Dependencies

NumPy/SciPy-compatible API in CuPy v13 is based on NumPy 1.26 and SciPy 1.11, and has been tested against the following versions:

NOTE

SciPy and Optuna are optional dependencies and will not be installed automatically.

NOTE

Before installing CuPy, we recommend you to upgrade setuptools and pip: python -m pip install -U setuptools pip

Additional CUDA Libraries

Part of the CUDA features in CuPy will be activated only when the corresponding libraries are installed.

  • cuTENSOR: v2.0
  • NCCL: v2.16 / v2.17
    • The library to perform collective multi-GPU / multi-node computations.
  • cuDNN: v8.8
    • The library to accelerate deep neural network computations.
  • cuSPARSELt: v0.2.0
    • The library to accelerate sparse matrix-matrix multiplication.

Installing CuPy from PyPI

Wheels (precompiled binary packages) are available for Linux and Windows. Package names are different depending on your CUDA Toolkit version.

CUDA

Command

v11.2 ~ 11.8 (x86_64 / aarch64)

pip install cupy-cuda11x

v12.x (x86_64 / aarch64)

pip install cupy-cuda12x

To enable features provided by additional CUDA libraries (cuTENSOR / NCCL / cuDNN), you need to install them manually. If you installed CuPy via wheels, you can use the installer command below to setup these libraries in case you don’t have a previous installation: 

$ python -m cupyx.tools.install_library --cuda 11.x --library cutensor

NOTE

Append --pre -U -fhttps://pip.cupy.dev/pre options to install pre-releases (e.g., pip install cupy-cuda11x --pre -U -fhttps://pip.cupy.dev/pre).

Building CuPy for ROCm From Source

To build CuPy from source, set the CUPY_INSTALL_USE_HIP, ROCM_HOME, and HCC_AMDGPU_TARGET environment variables. (HCC_AMDGPU_TARGET is the ISA name supported by your GPU. Run rocminfo and use the value displayed in Name: line (e.g., gfx900). You can specify a comma-separated list of ISAs if you have multiple GPUs of different architectures.) 

$ export CUPY_INSTALL_USE_HIP=1
$ export ROCM_HOME=/opt/rocm
$ export HCC_AMDGPU_TARGET=gfx906
$ pip install cupy

NOTE

If you don’t specify the HCC_AMDGPU_TARGET environment variable, CuPy will be built for the GPU architectures available on the build host. This behavior is specific to ROCm builds; when building CuPy for NVIDIA CUDA, the build result is not affected by the host configuration.

Device Pointer 

import cupy as cp

# GPU 배열 생성
array = cp.arange(1000000).reshape(1000, 1000)

# Device pointer 얻기
dev_ptr = array.data.ptr

# Device pointer 출력
print(f'Device pointer: {dev_ptr}')

Checking Device Overlap

How to check the Device Overlap property to see whether CUDA:Stream is supported: 

cupy.cuda.runtime.getDeviceProperties(0)['deviceOverlap']

Asynchronous operations

In any API that has a blocking parameter, that parameter makes it use an asynchronous function.

  • cupy.ndarray
  • cupy.array
  • cupy.asarray
  • cupy.asnumpy 
* blocking (bool) - False로 설정하면 복사본이 지정된 (제공된 경우) 또는 현재 스트림에서 비동기적으로 실행되며 사용자는 스트림 순서를 확인할 책임이 있습니다.

cudaGetDeviceCount

Gets the number of CUDA devices. 

import cupy as cp

num_devices = cp.cuda.runtime.getDeviceCount()

print(f"CUDA 장치 수: {num_devices}")

Selecting a Device 

with cp.cuda.Device(1):
   x_on_gpu1 = cp.array([1, 2, 3, 4, 5])
x_on_gpu0 = cp.array([1, 2, 3, 4, 5])

If the array's device and the current device do not match, CuPy functions try to set up Peer-to-Peer Memory Access (P2P) between the two devices so that the current device can read the array directly from the other device. P2P is available only when the topology permits it. If P2P is unavailable, such an attempt fails with a ValueError exception.

The N-dimensional array (ndarray)

cupy.ndarray

cupy.array

cupy.asarray

cupy.asnumpy

cupy.get_array_module

cupyx.scipy.get_array_module

Data transfer

Moving arrays to a device

cupy.asarray() can be used to move a numpy.ndarray, a list, or any object that can be passed to numpy.array() to the current device: 

x_cpu = np.array([1, 2, 3])
x_gpu = cp.asarray(x_cpu)  # move the data to the current device.

cupy.asarray() can accept cupy.ndarray, which means we can transfer the array between devices with this function. 

with cp.cuda.Device(0):
    x_gpu_0 = cp.ndarray([1, 2, 3])  # create an array in GPU 0

with cp.cuda.Device(1):
    x_gpu_1 = cp.asarray(x_gpu_0)  # move the array to GPU 1

Note

cupy.asarray() does not copy the input array when possible. So if you pass an array on the current device, the input object itself is returned.

If you need to copy the array in this situation, set copy=True among the arguments of cupy.array(). In fact, cupy.asarray() is equivalent to cupy.array(arr, dtype, copy=False).

Moving arrays from a device to the host

Moving a device array to the host can be done by cupy.asnumpy() as follows: 

x_gpu = cp.array([1, 2, 3])  # create an array in the current device

x_cpu = cp.asnumpy(x_gpu)  # move the array to the host.

We can also use cupy.ndarray.get(): 

x_cpu = x_gpu.get()

Note

If you are using Chainer, you can use to_cpu() and <codee>to_gpu()</code> to move arrays back and forth between a device and the host, or between devices.

to_gpu() has a device option that specifies the device the array is transferred to.

Translator's note: Chainer is a framework that supports dynamic computational graphs, and CuPy was split off from Chainer.

Writing CPU/GPU-agnostic code

Thanks to the compatibility between CuPy and NumPy, you can write code that works on both CPU and GPU. The cupy.get_array_module() function makes this easy. It returns the numpy or cupy module based on its argument. A CPU/GPU-agnostic function is defined and used as follows. 

# log(1 + exp(x))의 안정적인 구현
def softplus(x):
    xp = cp.get_array_module(x)
    return xp.maximum(0, x) + xp.log1p(xp.exp(-abs(x)))

Sometimes you may need to convert explicitly to a host or device array.

cupy.asarray() and cupy.asnumpy() can be used in generic implementations to convert between CuPY and NumPy arrays. 

x_cpu = np.array([1, 2, 3])
y_cpu = np.array([4, 5, 6])
x_cpu + y_cpu  # array([5, 7, 9])
x_gpu + y_cpu
"""
Traceback (most recent call last):
...
TypeError: Unsupported type <class 'numpy.ndarray'>
"""

cp.asnumpy(x_gpu) + y_cpu  # array([5, 7, 9])
cp.asnumpy(x_gpu) + cp.asnumpy(y_cpu)  # array([5, 7, 9])
x_gpu + cp.asarray(y_cpu)  # array([5, 7, 9])
cp.asarray(x_gpu) + cp.asarray(y_cpu)  # array([5, 7, 9])

How to allocate a CUDA MemoryPointer 

import cupy as cp
import numpy as np

size = 1024  # 배열의 크기
memory = cp.cuda.alloc(size * cp.float32().itemsize)
array = cp.ndarray((size,), dtype=cp.float32, memptr=memory)

How to directly assign a CUDA low-level device pointer (GPU) 

mem = cp.cuda.UnownedMemory(ptr, size, owning_obj)
mem_ptr = cp.cuda.MemoryPointer(mem, 0)
arr = cp.ndarray(..., memptr= mem_ptr, ...)  # make sure you interpret the array shape/dtype/strides correctly

Pinned Memory

How to synchronize pinned memory 

def pinned_array(array):
    # first constructing pinned memory
    mem = cupy.cuda.alloc_pinned_memory(array.nbytes)
    src = numpy.frombuffer(
                mem, array.dtype, array.size).reshape(array.shape)
    src[...] = array
    return src

a_cpu = np.ones((10000, 10000), dtype=np.float32)
b_cpu = np.ones((10000, 10000), dtype=np.float32)
# np.ndarray with pinned memory
a_cpu = pinned_array(a_cpu)
b_cpu = pinned_array(b_cpu)

a_stream = cp.cuda.Stream(non_blocking=True)
b_stream = cp.cuda.Stream(non_blocking=True)

a_gpu = cp.empty_like(a_cpu)
b_gpu = cp.empty_like(b_cpu)

a_gpu.set(a_cpu, stream=a_stream)
b_gpu.set(b_cpu, stream=b_stream)

# wait until a_cpu is copied in a_gpu
a_stream.synchronize()
# This line runs parallel to b_gpu.set()
a_gpu *= 2

Why the allocated memory size does not exactly match

Because memory is allocated to fit the memory alignment: 

# Round up the memory size to fit memory alignment of cudaHostAlloc 
unit = self._allocation_unit_size 
size = internal.clp2(((size + unit - 1) // unit) * unit)

Memory alignment code 

# cudaMalloc() is aligned to at least 512 bytes
# cf. https://gist.github.com/sonots/41daaa6432b1c8b27ef782cd14064269
DEF ALLOCATION_UNIT_SIZE = 512
# for test
_allocation_unit_size = ALLOCATION_UNIT_SIZE


cpdef size_t _round_size(size_t size):
    """Rounds up the memory size to fit memory alignment of cudaMalloc."""
    # avoid 0 div checking
    size = (size + ALLOCATION_UNIT_SIZE - 1) // ALLOCATION_UNIT_SIZE
    return size * ALLOCATION_UNIT_SIZE

cpdef size_t _bin_index_from_size(size_t size):
    """Returns appropriate bins index from the memory size."""
    # avoid 0 div checking
    return (size - 1) // ALLOCATION_UNIT_SIZE

Benchmarking

Use cupyx.profiler.benchmark. This function runs a few warm-up runs to reduce timing fluctuation and to exclude the overhead of the first call.

Synopsis: 

cupyx.profiler.benchmark(func, args=(), kwargs={}, n_repeat=10000, *, name=None, n_warmup=10, max_duration=inf, devices=None)

Usage example: 

from cupyx.profiler import benchmark

def my_func(a):
    return cp.sqrt(cp.sum(a**2, axis=-1))

a = cp.random.random((256, 1024))
print(benchmark(my_func, (a,), n_repeat=20))

The output looks like this: 

my_func             :    CPU:   44.407 us   +/- 2.428 (min:   42.516 / max:   53.098) us     GPU-0:  181.565 us   +/- 1.853 (min:  180.288 / max:  188.608) us

Example of a manual implementation: 

import time
start_gpu = cp.cuda.Event()
end_gpu = cp.cuda.Event()

start_gpu.record()
start_cpu = time.perf_counter()
out = my_func(a)
end_cpu = time.perf_counter()
end_gpu.record()
end_gpu.synchronize()
t_gpu = cp.cuda.get_elapsed_time(start_gpu, end_gpu)
t_cpu = end_cpu - start_cpu

Basics

Low-level CUDA support

cupy.cuda.Event

A CUDA event, which is a synchronization point of CUDA streams.

This class handles the CUDA event handle in RAII way, i.e., when an Event instance is destroyed by the GC, its handle is also destroyed.

  • class cupy.cuda.Event(block=False, disable_timing=False, interprocess=False)
    • Parameters:
      • block (bool) – If True, the event blocks on the .synchronize() method.
      • disable_timing (bool) – If True, the event does not prepare the timing data.
      • interprocess (bool) – If True, the event can be passed to other processes.
    • .synchronize() Method
      • Synchronizes all device work to the event.
      • If the event is created as a blocking event, it also blocks the CPU thread until the event is done.

IPC

  • cupy.cuda.runtime.ipcGetMemHandle(intptr_t devPtr)
  • cupy.cuda.runtime.ipcOpenMemHandle(bytes handle, unsigned int flags=cudaIpcMemLazyEnablePeerAccess)
  • cupy.cuda.runtime.ipcCloseMemHandle(intptr_t devPtr)
  • cupy.cuda.runtime.ipcGetEventHandle(intptr_t event)
  • cupy.cuda.runtime.ipcOpenEventHandle(bytes handle)

See CuPy:IPC for an implementation example.

Memory Pool 

import cupy
import numpy

mempool = cupy.get_default_memory_pool()
pinned_mempool = cupy.get_default_pinned_memory_pool()

# Create an array on CPU.
# NumPy allocates 400 bytes in CPU (not managed by CuPy memory pool).
a_cpu = numpy.ndarray(100, dtype=numpy.float32)
print(a_cpu.nbytes)                      # 400

# You can access statistics of these memory pools.
print(mempool.used_bytes())              # 0
print(mempool.total_bytes())             # 0
print(pinned_mempool.n_free_blocks())    # 0

# Transfer the array from CPU to GPU.
# This allocates 400 bytes from the device memory pool, and another 400
# bytes from the pinned memory pool.  The allocated pinned memory will be
# released just after the transfer is complete.  Note that the actual
# allocation size may be rounded to larger value than the requested size
# for performance.
a = cupy.array(a_cpu)
print(a.nbytes)                          # 400
print(mempool.used_bytes())              # 512
print(mempool.total_bytes())             # 512
print(pinned_mempool.n_free_blocks())    # 1

# When the array goes out of scope, the allocated device memory is released
# and kept in the pool for future reuse.
a = None  # (or `del a`)
print(mempool.used_bytes())              # 0
print(mempool.total_bytes())             # 512
print(pinned_mempool.n_free_blocks())    # 1

# You can clear the memory pool by calling `free_all_blocks`.
mempool.free_all_blocks()
pinned_mempool.free_all_blocks()
print(mempool.used_bytes())              # 0
print(mempool.total_bytes())             # 0
print(pinned_mempool.n_free_blocks())    # 0

Interoperability

NumPy

cupy.ndarray implements array_ufunc interface.

Numba

Numba is a Python JIT compiler with NumPy support.

cupy.ndarray implements cuda_array_interface, the CUDA array interchange interface compatible with Numba v0.39.0 or later (see CUDA Array Interface for details).

This means you can pass CuPy arrays to kernels JITed with Numba. The following is a simple example code borrowed from numba/numba#2860. 

import cupy
from numba import cuda

@cuda.jit
def add(x, y, out):
        start = cuda.grid(1)
        stride = cuda.gridsize(1)
        for i in range(start, x.shape[0], stride):
                out[i] = x[i] + y[i]

a = cupy.arange(10)
b = a * 2
out = cupy.zeros_like(a)

print(out)  # => [0 0 0 0 0 0 0 0 0 0]

add[1, 32](a, b, out)

print(out)  # => [ 0  3  6  9 12 15 18 21 24 27]

In addition, cupy.asarray() supports zero-copy conversion from a Numba CUDA array to a CuPy array. 

import numpy
import numba
import cupy

x = numpy.arange(10)  # type: numpy.ndarray
x_numba = numba.cuda.to_device(x)  # type: numba.cuda.cudadrv.devicearray.DeviceNDArray
x_cupy = cupy.asarray(x_numba)  # type: cupy.ndarray

mpi4py

PyTorch

PyTorch is a machine learning framework that provides high-performance, differentiable tensor operations.

PyTorch also supports cuda_array_interface, so zero-copy data exchange between CuPy and PyTorch can be done at no cost.

The only caveat is that PyTorch by default creates CPU tensors, which do not have the cuda_array_interface property defined, so users need to make sure the tensor is already on the GPU before exchanging. 

import cupy as cp
import torch

import cupy as cp
import torch

# convert a torch tensor to a cupy array
a = torch.rand((4, 4), device='cuda')
b = cp.asarray(a)
b *= b

print(b)
"""
array([[0.8215962 , 0.82399917, 0.65607935, 0.30354425],
       [0.422695  , 0.8367199 , 0.00208597, 0.18545236],
       [0.00226746, 0.46201342, 0.6833052 , 0.47549972],
       [0.5208748 , 0.6059282 , 0.1909013 , 0.5148635 ]], dtype=float32)
"""

print(a)
"""
tensor([[0.8216, 0.8240, 0.6561, 0.3035],
        [0.4227, 0.8367, 0.0021, 0.1855],
        [0.0023, 0.4620, 0.6833, 0.4755],
        [0.5209, 0.6059, 0.1909, 0.5149]], device='cuda:0')
"""

# check the underlying memory pointer is the same
assert a.__cuda_array_interface__['data'][0] == b.__cuda_array_interface__['data'][0]

# convert a cupy array to a torch tensor
a = cp.arange(10)
b = torch.as_tensor(a, device='cuda')
b += 3

print(b)
"""tensor([ 3,  4,  5,  6,  7,  8,  9, 10, 11, 12], device='cuda:0')"""

print(a)
"""array([ 3,  4,  5,  6,  7,  8,  9, 10, 11, 12])"""

assert a.__cuda_array_interface__['data'][0] == b.__cuda_array_interface__['data'][0]

PyTorch also supports zero-copy data exchange through DLPack (see #DLPack below). 

import cupy
import torch

# Create a PyTorch tensor.
tx1 = torch.randn(1, 2, 3, 4).cuda()

# Convert it into a CuPy array.
cx = cupy.from_dlpack(tx1)

# Convert it back to a PyTorch tensor.
tx2 = torch.from_dlpack(cx)

Feeding a torch.Tensor directly to cupy.from_dlpack() is only supported in the (new) DLPack data exchange protocol added in CuPy v10+ and PyTorch 1.10+.

For earlier versions, you need to wrap the Tensor with torch.utils.dlpack.to_dlpack() as shown in the example above.

RMM

DLPack

DLPack is a specification of tensor structure for sharing tensors among frameworks.

CuPy supports importing from and exporting to the DLPack data structure (cupy.from_dlpack() and cupy.ndarray.toDlpack()). 

import cupy

# Create a CuPy array.
cx1 = cupy.random.randn(1, 2, 3, 4).astype(cupy.float32)

# Convert it into a DLPack tensor.
dx = cx1.toDlpack()

# Convert it back to a CuPy array.
cx2 = cupy.from_dlpack(dx)

See also

Favorite site

author Zer0 sitemap