Skip to content
Search

PyTorch

lastmod 2026-03-11created 2018-12-0465 edits

Categories

  • torch.autograd - tape-based automatic differentiation library that supports all differentiable Tensor operations in Torch
  • torch.device - related to switching between CPU/GPU
  • torch.nn - neural network library deeply integrated with the automatic graph, designed for maximum flexibility - https://pytorch.org/docs/stable/nn.html
  • torch.legacy (.nn/optim) - legacy code ported over from Torch for backward compatibility
  • torch.optim - optimization package used with torch.nn, with standard optimization methods such as SGD, RMSProp, LBFGS, and Adam
  • torch.utils - DataLoader, Trainer, and other utility functions for convenience
  • torch.multiprocessing - Python multiprocessing, but with magical memory sharing of Torch Tensors across processes. Useful for data loading and Hogwild training

Libraries

  • torchaudio
  • torchtext
  • torchvision
  • TorchElastic
  • TorchServe - TorchServe is a flexible and easy-to-use tool for serving PyTorch models.
  • PyTorch on XLA Devices
  • PyTorch vs Tensorflow

ETC

  • Stanford CRFM - AI-generated CUDA kernels outperform PyTorch-optimized code

Projects

CppRl - PyTorch C++ Reinforcement Learning
PyTorch C++ Reinforcement Learning
https://github.com/Omegastick/pytorch-cpp-rl
Reddit - CppRl: A C++ reinforcement learning library using the new PyTorch C++ frontend : MachineLearning
higher
higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.
https://github.com/facebookresearch/higher
Kornia
Open Source Differentiable Computer Vision Library for PyTorch
https://github.com/arraiyopensource/kornia
pytorch-struct
https://github.com/harvardnlp/pytorch-struct
Fast, general, and tested differentiable structured prediction in PyTorch
A library of tested, GPU implementations of core structured prediction algorithms for deep learning applications. (or an implementation of Inside-Outside and Forward-Backward Algorithms Are Just Backprop"1)
cvxpylayers
https://github.com/cvxgrp/cvxpylayers
Differentiable convex optimization layers
cvxpylayers is a Python library for constructing differentiable convex optimization layers in PyTorch and TensorFlow using CVXPY. A convex optimization layer solves a parametrized convex optimization problem in the forward pass to produce a solution. It computes the derivative of the solution with respect to the parameters in the backward pass.
pycls
https://github.com/facebookresearch/pycls
Codebase for Image Classification Research, written in PyTorch.
pycl is an image classification codebase, written in PyTorch. The codebase was originally developed for a project that led to the On Network Design Spaces for Visual Recognition work. pycls has since matured into a general image classification codebase that has been adopted by a number representation learning projects at Facebook AI Research.

PyTorch Visualization

Distributed Applications

Download and Requirements

For example, when installing v2.3.0: 

# ROCM 6.0 (Linux only)
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/rocm6.0
# CUDA 11.8
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu118
# CUDA 12.1
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121
# CPU only
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cpu

Once you have chosen one of these, create a requirements.txt file as follows: 

--index-url https://download.pytorch.org/whl/cu117
torch==2.0.1+cu117
torchvision==0.15.2+cu117
torchaudio==2.0.2+cu117

torch.no_grad() 

with torch.no_grad()
    ...

Wrapping code in a no_grad() with statement like this makes Pytorch turn off the autograd engine. This means gradients are no longer tracked automatically. You might then wonder: if we are not running backpropagation via loss.backward(), does it really matter whether gradients are computed or not?

That's right. The main purpose of torch.no_grad() is to reduce memory usage and speed up computation by turning off autograd. After all, why compute gradients during inference that you won't use anyway?

So just remember that inference is usually wrapped in a torch.no_grad() with statement.

model.eval() 

model.eval()

Going back to the original question, you might think: isn't the torch.no_grad() above enough? We're not computing gradients anymore. That's true, but model.eval() plays a slightly different role. At present (2019), some layers behave differently during training and inference. For example, a Dropout layer should be active during training but not during inference. The same goes for BatchNorm.

In effect, model.eval() is used to switch the behavior of such layers to inference (eval) mode. Therefore, to get the model behavior we usually want, we should use both of the above.

Device Pointer

(Naturally) you can get a pointer with tensor.data_ptr() only if the tensor is allocated in CUDA memory with device='cuda'. 

import torch

# CUDA tensor 생성
tensor = torch.arange(1000000, device='cuda').reshape(1000, 1000)

# Device pointer 얻기
dev_ptr = tensor.data_ptr()

# Device pointer 출력
print(f'Device pointer: {dev_ptr}')

Deep Reinforcement Learning Algorithms with PyTorch

  1. Deep Q Learning (DQN) (Mnih et al. 2013)
  2. DQN with Fixed Q Targets (Mnih et al. 2013)
  3. Double DQN (DDQN) (Hado van Hasselt et al. 2015)
  4. DDQN with Prioritised Experience Replay (Schaul et al. 2016)
  5. Dueling DDQN (Wang et al. 2016)
  6. REINFORCE (Williams et al. 1992)
  7. Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al. 2016 )
  8. Twin Delayed Deep Deterministic Policy Gradients (TD3) (Fujimoto et al. 2018)
  9. Soft Actor-Critic (SAC & SAC-Discrete) (Haarnoja et al. 2018)
  10. Asynchronous Advantage Actor Critic (A3C) (Mnih et al. 2016)
  11. Syncrhonous Advantage Actor Critic (A2C)
  12. Proximal Policy Optimisation (PPO) (Schulman et al. 2017)
  13. DQN with Hindsight Experience Replay (DQN-HER) (Andrychowicz et al. 2018)
  14. DDPG with Hindsight Experience Replay (DDPG-HER) (Andrychowicz et al. 2018 )
  15. Hierarchical-DQN (h-DQN) (Kulkarni et al. 2016)
  16. Stochastic NNs for Hierarchical Reinforcement Learning (SNN-HRL) (Florensa et al. 2017)
  17. Diversity Is All You Need (DIAYN) (Eyensbach et al. 2018)

Memory management

PyTorch uses a caching memory allocator to speed up memory allocations. This allows fast memory deallocation without device synchronizations. However, the unused memory managed by the allocator will still show as if used in nvidia-smi. You can use memory_allocated() and max_memory_allocated() to monitor memory occupied by tensors, and use memory_reserved() and max_memory_reserved() to monitor the total amount of memory managed by the caching allocator. Calling empty_cache() releases all unused cached memory from PyTorch so that those can be used by other GPU applications. However, the occupied GPU memory by tensors will not be freed so it can not increase the amount of GPU memory available for PyTorch.

For more advanced users, we offer more comprehensive memory benchmarking via memory_stats(). We also offer the capability to capture a complete snapshot of the memory allocator state via memory_snapshot(), which can help you understand the underlying allocation patterns produced by your code.

Use of a caching allocator can interfere with memory checking tools such as cuda-memcheck. To debug memory errors using cuda-memcheck, set PYTORCH_NO_CUDA_MEMORY_CACHING=1 in your environment to disable caching.

The behavior of caching allocator can be controlled via environment variable PYTORCH_CUDA_ALLOC_CONF. The format is PYTORCH_CUDA_ALLOC_CONF=<option>:<value>,<option2>:<value2>... Available options:

  • max_split_size_mb prevents the allocator from splitting blocks larger than this size (in MB). This can help prevent fragmentation and may allow some borderline workloads to complete without running out of memory. Performance cost can range from ‘zero’ to ‘substatial’ depending on allocation patterns. Default value is unlimited, i.e. all blocks can be split. The memory_stats() and memory_summary() methods are useful for tuning. This option should be used as a last resort for a workload that is aborting due to ‘out of memory’ and showing a large amount of inactive split blocks.
  • roundup_power2_divisions helps with rounding the requested allocation size to nearest power-2 division and making better use of the blocks. In the current CUDACachingAllocator, the sizes are rounded up in multiple of blocks size of 512, so this works fine for smaller sizes. However, this can be inefficient for large near-by allocations as each will go to different size of blocks and re-use of those blocks are minimized. This might create lots of unused blocks and will waste GPU memory capacity. This option enables the rounding of allocation size to nearest power-2 division. For example, if we need to round-up size of 1200 and if number of divisions is 4, the size 1200 lies between 1024 and 2048 and if we do 4 divisions between them, the values are 1024, 1280, 1536, and 1792. So, allocation size of 1200 will be rounded to 1280 as the nearest ceiling of power-2 division.
  • roundup_bypass_threshold_mb bypass rounding the requested allocation size, for allocation requests larger than the threshold value (in MB). This can help reduce the memory footprint when making large allocations that are expected to be persistent or have a large lifetime.
  • garbage_collection_threshold helps actively reclaiming unused GPU memory to avoid triggering expensive sync-and-reclaim-all operation (release_cached_blocks), which can be unfavorable to latency-critical GPU applications (e.g., servers). Upon setting this threshold (e.g., 0.8), the allocator will start reclaiming GPU memory blocks if the GPU memory capacity usage exceeds the threshold (i.e., 80% of the total memory allocated to the GPU application). The algorithm prefers to free old & unused blocks first to avoid freeing blocks that are actively being reused. The threshold value should be between greater than 0.0 and less than 1.0.

Troubleshooting

PyTorch multiprocessing

Using PyTorch with the Python:multiprocessing package may cause the following error. 

`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

As stated in pytorch documentation the best practice to handle multiprocessing is to use torch.multiprocessing instead of multiprocessing.

Be aware that sharing CUDA tensors between processes is supported only in Python 3, either with spawn or forkserver as start method.

Without touching your code, a workaround for the error you got is replacing 

from multiprocessing import Process, Pool

with: 

from torch.multiprocessing import Pool, Process, set_start_method
try:
     set_start_method('spawn')
except RuntimeError:
    pass

Library not loaded libomp.dylib 

Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    import torch
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/__init__.py", line 79, in <module>
    from torch._C import *
ImportError: dlopen(/usr/local/c2core/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so, 9): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /usr/local/c2core/lib/python3.7/site-packages/torch/lib/libshm.dylib
  Reason: image not found

Install OpenMP. On macOS, use brew install libomp.

Found no NVIDIA driver on your system

When building the CUDA parts of PyTorch, the following error may occur. 

Traceback (most recent call last):
  File "setup.py", line 68, in <module>
    cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension},
  File "/usr/local/c2core/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/c2core/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/c2core/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/c2core/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/c2core/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/local/c2core/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/c2core/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/c2core/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
    _build_ext.run(self)
  File "/usr/local/c2core/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/local/c2core/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 372, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/local/c2core/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/usr/local/c2core/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/c2core/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/c2core/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 199, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/local/c2core/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/usr/local/c2core/lib/python3.7/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 288, in unix_wrap_compile
    "'-fPIC'"] + cflags + _get_cuda_arch_flags(cflags)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1013, in _get_cuda_arch_flags
    capability = torch.cuda.get_device_capability()
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/cuda/__init__.py", line 320, in get_device_capability
    prop = get_device_properties(device)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/cuda/__init__.py", line 325, in get_device_properties
    _lazy_init()  # will define _get_device_properties and _CudaDeviceProperties
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/cuda/__init__.py", line 196, in _lazy_init
    _check_driver()
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/cuda/__init__.py", line 101, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

This is caused by pytorch's runtime check. To pass options directly to nvcc, add a ;-separated list of CUDA architectures to the TORCH_CUDA_ARCH_LIST environment variable. 

TORCH_CUDA_ARCH_LIST="Pascal;Volta;Turing" FORCE_CUDA=1 /usr/local/c2core/bin/python3.7 setup.py build

See maskrcnn-benchmark#Force cuda build.

Why results keep changing

INFORMATION

One-line summary: the most important thing is to call model.eval() when predicting.

When making predictions (predict) with a model in PyTorch, results may keep changing for several reasons:

The model is in training mode

  • In PyTorch, a model has a train() mode and an eval() mode. In train() mode, layers such as dropout and batch normalization are active and can produce different results each time.
  • In eval() mode, on the other hand, these layers are disabled and predictions become deterministic.
  • Therefore, when making predictions, call model.eval() to switch to evaluation mode. 
model.eval()
with torch.no_grad():
    predictions = model(input_data)

Random Number Generator (RNG)

Random numbers are used in model initialization and data sampling. If you have not set a random seed, you may get different results on every run. Fixing the seed gives the same results. 

import torch
torch.manual_seed(42)

Dropout

Dropout is a technique that prevents overfitting by randomly deactivating certain neurons during training. If dropout is active in training mode, it can produce different predictions each time. Switching to model.eval() disables dropout.

Data sampling

If the data loader (DataLoader) samples randomly from the dataset, different data may be fed into the model each time. Predictions can differ in this case as well. To fix this, fix the random seed in the data sampling process.

Multiprocessing

When using multiprocessing, subtle differences between tasks can also arise and make prediction results differ.

Check whether a scaler is used

Data scaling is needed in many situations:

  • When variables have very different scales during analysis - the scale of the numbers differs greatly because of differences in the variables' units
  • When training a neural network, if the dataset values are uneven or very large, the cost diverges and training does not proceed normally

A scaler is what is used in these cases; if you used one, remove it. 

from numpy.typing import NDArray
from pandas import DataFrame
from sklearn.preprocessing import StandardScaler

# ...

def rescale_standard(
    df: DataFrame,
    *,
    copy=True,
    with_mean=True,
    with_std=True,
) -> NDArray:
    scaler = StandardScaler(copy=copy, with_mean=with_mean, with_std=with_std)
    return scaler.fit_transform(df)

# ...

df_without_rr = drop_rate(df)

if args.use_scaler:
    scaled_features = rescale_standard(df_without_rr)
    if args.debug and args.verbose >= VERBOSE_LEVEL_2:
        logger.debug(f"Scaled features:\n{scaled_features}")
else:
    scaled_features = df_without_rr.to_numpy().astype(float)

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Use Model().cuda() or Model().to(device).

If the above problem occurs after using load_state_dict on the Optimizer: 

optim_state = torch.load({CHECKPOINT_PATH})["optimizer"]
optim.load_state_dict(optim_state)
for state in optim.state.values():
    for k, v in state.items():
        if torch.is_tensor(v):
            state[k] = v.to(device)

WARNING

But it didn't work for me... Need to check what the problem is later

NumPy 2.x version issue 

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Switch to 1.26.4, the latest version in the NumPy 1.x series.

Incompatible Device Warning 

NVIDIA GeForce RTX 5080 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA GeForce RTX 5080 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

Remove the old torch version and install a newer one: 

python -m pip uninstall torch torchvision torchaudio
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

AssertionError: Size mismatch between tensors 

2026-03-11 13:27:41.484 114103/126720013196928 vrs ERROR Size mismatch between tensors
Traceback (most recent call last):
  File "/home/zer0/Project/ktfms_i/vrs/apps/__init__.py", line 30, in run_app
    app(args)
  File "/home/zer0/Project/ktfms_i/vrs/apps/train.py", line 440, in main
    TrainContext.from_namespace(args).run()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zer0/Project/ktfms_i/vrs/apps/train.py", line 110, in from_namespace
    return cls(
           ^^^^
  File "/home/zer0/Project/ktfms_i/vrs/apps/train.py", line 81, in __init__
    self._train_loader = DataLoader(TensorDataset())
                                    ^^^^^^^^^^^^^^^
  File "/home/zer0/Project/ktfms_i/.venv/lib/python3.11/site-packages/torch/utils/data/dataset.py", line 202, in __init__
    raise AssertionError("Size mismatch between tensors")
AssertionError: Size mismatch between tensors

Calling TensorDataset() without arguments raises an error in newer PyTorch versions. Change the initial value to None.

See also

Favorite site

Tutorials

Article

Guide

Example

Awesome

Article

References

  1. Eisner.spnlp16.pdf ↩

author Zer0 sitemap