Skip to content
Search

CUDA:Troubleshooting

lastmod 2026-04-17created 2018-02-0122 edits

Notes on how to solve CUDA-related problems.

C++11 support

When using --std c++11, which NVCC supports, the following error may occur. 

[your@server down]$ nvcc --std c++11 main.cpp 
nvcc warning : The -c++11 flag is not supported with the configured host compiler. Flag will be ignored.
In file included from /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:35,
                 from main.cpp:1:
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/c++0x_warning.h:31:2: error:
    #error This file requires compiler and library support for the upcoming ISO C++ standard,
    C++0x. This support is currently experimental, and must be enabled with the -std=c++0x or -std=gnu++0x compiler options.

In this case, the option must be passed directly to the compiler, as follows. 

nvcc -Xcompiler -std=c++0x main.cu

Explicitly specify the language

When a cpp file is passed, it may not be recognized as CUDA. In this case, you can force it to be recognized as CUDA as follows. 

nvcc --x cu test.cpp

Compiling CUDA in QT with MinGW

CUDA support for MinGw is not available. CUDA is only supported by Visual C++ environment in Windows. Using CUDA with QT under MinGw was not an option for me after this.

CUDA unavailable over Remote Desktop

GeForce-series devices cannot be detected properly over Remote Desktop. You therefore have to check directly on the machine itself.

Note that in this case, cudaGetDeviceCount() returns cudaErrorNoDevice.

invalid permissions

The following error message may be printed at runtime. 

memory access violation at address: 0x703cc0000: invalid permissions

This error occurs when Host code and Device code are not used in the right places. A typical example is accessing memory from the Host after cudaMalloc.

Note that in such a case, you can copy Host data to Device data using cudaMemcpy or similar.

Undefined __float128

The following error may occur while compiling CUDA. 

/usr/include/c++/4.9/type_traits(279): error: identifier "__float128" is undefined

If you added -std=gnu++11 to the NVCC compiler options, change it to -std=c++11. (For reference, the test environment was Ubuntu 14.04 GCC 4.9 NVCC CUDA 7.5.17.)

If you used Boost, check BOOST Ticket 11852 and apply the patch. The relevant header is boost/config/compiler/gcc.hpp, and the patch is as follows. 

 include/boost/config/compiler/gcc.hpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/boost/config/compiler/gcc.hpp b/include/boost/config/compiler/gcc.hpp
index d9dd59d..48605eb 100644

Index: include/boost/config/compiler/gcc.hpp
===================================================================
--- a/include/boost/config/compiler/gcc.hpp
+++ b/include/boost/config/compiler/gcc.hpp
@@ -153,7 +153,7 @@
 #else
 #include <stddef.h>
 #endif
-#if defined(_GLIBCXX_USE_FLOAT128) && !defined(__STRICT_ANSI__)
+#if defined(_GLIBCXX_USE_FLOAT128) && !defined(__STRICT_ANSI__) && !defined(__CUDACC__)
 # define BOOST_HAS_FLOAT128
 #endif

Without Monitor

Using CUDA without a monitor connected does not work properly. On Linux, this can be solved by installing the xserver-xorg-video-dummy driver for the X Window System.

Alternatively, switch between WDDM/TCC modes.

Stop on reboot

Notes on the issue where it stops working after a reboot.

  • Aug 22, 2018 - Don't use the .run installer; install the graphics driver from a PPA, then install from the .deb file

Compiling dynamic parallelism

Add -rdc=true to the compile options, and add the cudadevrt library when linking.

CUDA dynamic parallelism debugging is not supported in preemption mode

Right click the monitor's tray icon, check "Options\CUDA\Debugger". Except TCC GPUs, the others are by default force "Software Preemption".

You can set "Desktop GPUS must use Software Preemption" and "Headless GPUs must use software preemption" to false. And make sure in you VisualStuido, the setting "Nsight\Options\CUDA\Preemption Preference" is "Prefer no Software Preemption". 

모니터의 트레이 아이콘을 마우스 오른쪽 버튼으로 클릭하고 "Options\CUDA\Debugger"를 체크하십시오.
TCC GPU를 제외하고 다른 것들은 기본적으로 "소프트웨어 선점 (Software Preemption)"을 강요합니다.

"데스크톱 GPUS는 소프트웨어 선점을 사용 (Desktop GPUS must use Software Preemption)"및
"헤드리스 GPU는 소프트웨어 선점을 사용 (Headless GPUs must use software preemption)"를 false로 설정할 수 있습니다.
VisualStuido에서 "Nsight\Options\CUDA\Preemption Preference"설정이 "Prefer no Software Preemption"이 아닌지 확인하십시오.

cudaErrorLaunchOutOfResources

This indicates that a launch did not occur because it did not have appropriate resources. Although this error is similar to cudaErrorInvalidConfiguration, this error usually indicates that the user has attempted to pass too many arguments to the device kernel, or the kernel launch specifies too many threads for the kernel's register count.

One of the CUDA error types. If this error occurs, check the following.

  • Check the number of Threads passed to the kernel.
  • Check the number of Registers per Thread (set a limit)
  • Check the amount of Heap memory allocated.

== Depends: cuda-11-5 (>= 11.5.0) but it is not going to be installed == After downloading and installing the latest CUDA Toolkit, the following issue may occur. 

$ sudo apt-get -y install cuda
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda : Depends: cuda-11-5 (>= 11.5.0) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Checking with dpkg: 

$ dpkg -l | grep cuda
ii  cuda-command-line-tools-11-0                      11.0.3-1                                        amd64        CUDA command-line tools
ii  cuda-compiler-11-0                                11.0.3-1                                        amd64        CUDA compiler
ii  cuda-cudart-11-0                                  11.0.221-1                                      amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-11-0                              11.0.221-1                                      amd64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-11-0                               11.0.221-1                                      amd64        CUDA cuobjdump
ii  cuda-cupti-11-0                                   11.0.221-1                                      amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-11-0                               11.0.221-1                                      amd64        CUDA profiling tools interface.
ii  cuda-documentation-11-0                           11.0.228-1                                      amd64        CUDA documentation
ii  cuda-driver-dev-11-0                              11.0.221-1                                      amd64        CUDA Driver native dev stub library
ii  cuda-gdb-11-0                                     11.0.221-1                                      amd64        CUDA-GDB
ii  cuda-libraries-11-0                               11.0.3-1                                        amd64        CUDA Libraries 11.0 meta-package
ii  cuda-libraries-dev-11-0                           11.0.3-1                                        amd64        CUDA Libraries 11.0 development meta-package
ii  cuda-memcheck-11-0                                11.0.221-1                                      amd64        CUDA-MEMCHECK
ii  cuda-nsight-11-0                                  11.0.221-1                                      amd64        CUDA nsight
ii  cuda-nsight-compute-11-0                          11.0.3-1                                        amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-11-0                          11.0.3-1                                        amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-11-0                                    11.0.221-1                                      amd64        CUDA nvcc
ii  cuda-nvdisasm-11-0                                11.0.221-1                                      amd64        CUDA disassembler
ii  cuda-nvml-dev-11-0                                11.0.167-1                                      amd64        NVML native dev links, headers
ii  cuda-nvprof-11-0                                  11.0.221-1                                      amd64        CUDA Profiler tools
ii  cuda-nvprune-11-0                                 11.0.221-1                                      amd64        CUDA nvprune
ii  cuda-nvrtc-11-0                                   11.0.221-1                                      amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-11-0                               11.0.221-1                                      amd64        NVRTC native dev links, headers
ii  cuda-nvtx-11-0                                    11.0.167-1                                      amd64        NVIDIA Tools Extension
ii  cuda-nvvp-11-0                                    11.0.221-1                                      amd64        CUDA Profiler tools
ii  cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01 1.0-1                                           amd64        cuda repository configuration files
ii  cuda-repo-ubuntu1804-11-4-local                   11.4.2-470.57.02-1                              amd64        cuda repository configuration files
ii  cuda-repo-ubuntu1804-11-5-local                   11.5.0-495.29.05-1                              amd64        cuda repository configuration files
ii  cuda-samples-11-0                                 11.0.221-1                                      amd64        CUDA example applications
ii  cuda-sanitizer-11-0                               11.0.221-1                                      amd64        CUDA Sanitizer
ii  cuda-toolkit-11-0                                 11.0.3-1                                        amd64        CUDA Toolkit 11.0 meta-package
ii  cuda-tools-11-0                                   11.0.3-1                                        amd64        CUDA Tools meta-package
ii  cuda-visual-tools-11-0                            11.0.3-1                                        amd64        CUDA visual tools

The problem is that apt update was run using a downloaded local-install cuda toolkit, but the version was updated in the meantime, which caused this issue.

Remove the previously downloaded CUDA-related packages, 

$ sudo apt purge cuda
$ sudo apt autoremove

Find what apt can't find directly with dpkg: 

$ dpkg -l | grep cuda
rc  cuda-cudart-11-0                                  11.0.221-1                                      amd64        CUDA Runtime native Libraries
ii  cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01 1.0-1                                           amd64        cuda repository configuration files
ii  cuda-repo-ubuntu1804-11-4-local                   11.4.2-470.57.02-1                              amd64        cuda repository configuration files
ii  cuda-repo-ubuntu1804-11-5-local                   11.5.0-495.29.05-1                              amd64        cuda repository configuration files
rc  cuda-toolkit-11-0                                 11.0.3-1                                        amd64        CUDA Toolkit 11.0 meta-package
rc  cuda-visual-tools-11-0                            11.0.3-1                                        amd64        CUDA visual tools

Remove all cuda-repo- packages except the version you need. 

$ sudo dpkg -P cuda-repo-ubuntu1804-11-4-local
$ sudo dpkg -P cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01
$ dpkg -l | grep cuda
rc  cuda-cudart-11-0                           11.0.221-1                                      amd64        CUDA Runtime native Libraries
ii  cuda-repo-ubuntu1804-11-5-local            11.5.0-495.29.05-1                              amd64        cuda repository configuration files
rc  cuda-toolkit-11-0                          11.0.3-1                                        amd64        CUDA Toolkit 11.0 meta-package
rc  cuda-visual-tools-11-0                     11.0.3-1                                        amd64        CUDA visual tools

Then install cuda: 

$ sudo apt install cuda
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  cuda-11-5 cuda-cccl-11-5 cuda-command-line-tools-11-5 cuda-compiler-11-5 cuda-cudart-11-5 cuda-cudart-dev-11-5 cuda-cuobjdump-11-5 cuda-cupti-11-5 cuda-cupti-dev-11-5 cuda-cuxxfilt-11-5
  cuda-demo-suite-11-5 cuda-documentation-11-5 cuda-driver-dev-11-5 cuda-drivers cuda-drivers-495 cuda-gdb-11-5 cuda-libraries-11-5 cuda-libraries-dev-11-5 cuda-memcheck-11-5 cuda-nsight-11-5
  cuda-nsight-compute-11-5 cuda-nsight-systems-11-5 cuda-nvcc-11-5 cuda-nvdisasm-11-5 cuda-nvml-dev-11-5 cuda-nvprof-11-5 cuda-nvprune-11-5 cuda-nvrtc-11-5 cuda-nvrtc-dev-11-5 cuda-nvtx-11-5
  cuda-nvvp-11-5 cuda-runtime-11-5 cuda-samples-11-5 cuda-sanitizer-11-5 cuda-toolkit-11-5 cuda-toolkit-11-5-config-common cuda-toolkit-11-config-common cuda-toolkit-config-common cuda-tools-11-5
  cuda-visual-tools-11-5 dkms gds-tools-11-5 libcublas-11-5 libcublas-dev-11-5 libcufft-11-5 libcufft-dev-11-5 libcufile-11-5 libcufile-dev-11-5 libcurand-11-5 libcurand-dev-11-5 libcusolver-11-5
  libcusolver-dev-11-5 libcusparse-11-5 libcusparse-dev-11-5 libnpp-11-5 libnpp-dev-11-5 libnvidia-cfg1-495 libnvidia-common-495 libnvidia-compute-495 libnvidia-decode-495 libnvidia-encode-495
  libnvidia-extra-495 libnvidia-fbc1-495 libnvidia-gl-495 libnvjpeg-11-5 libnvjpeg-dev-11-5 libopengl0 liburcu6 libxnvctrl0 nsight-compute-2021.3.0 nsight-systems-2021.3.3 nvidia-compute-utils-495
  nvidia-dkms-495 nvidia-driver-495 nvidia-kernel-common-495 nvidia-kernel-source-495 nvidia-modprobe nvidia-prime nvidia-settings nvidia-utils-495 screen-resolution-extra xserver-xorg-video-nvidia-495
Suggested packages:
  menu
Recommended packages:
  libnvidia-compute-495:i386 libnvidia-decode-495:i386 libnvidia-encode-495:i386 libnvidia-fbc1-495:i386 libnvidia-gl-495:i386
The following NEW packages will be installed:
  cuda cuda-11-5 cuda-cccl-11-5 cuda-command-line-tools-11-5 cuda-compiler-11-5 cuda-cudart-11-5 cuda-cudart-dev-11-5 cuda-cuobjdump-11-5 cuda-cupti-11-5 cuda-cupti-dev-11-5 cuda-cuxxfilt-11-5
  cuda-demo-suite-11-5 cuda-documentation-11-5 cuda-driver-dev-11-5 cuda-drivers cuda-drivers-495 cuda-gdb-11-5 cuda-libraries-11-5 cuda-libraries-dev-11-5 cuda-memcheck-11-5 cuda-nsight-11-5
  cuda-nsight-compute-11-5 cuda-nsight-systems-11-5 cuda-nvcc-11-5 cuda-nvdisasm-11-5 cuda-nvml-dev-11-5 cuda-nvprof-11-5 cuda-nvprune-11-5 cuda-nvrtc-11-5 cuda-nvrtc-dev-11-5 cuda-nvtx-11-5
  cuda-nvvp-11-5 cuda-runtime-11-5 cuda-samples-11-5 cuda-sanitizer-11-5 cuda-toolkit-11-5 cuda-toolkit-11-5-config-common cuda-toolkit-11-config-common cuda-toolkit-config-common cuda-tools-11-5
  cuda-visual-tools-11-5 dkms gds-tools-11-5 libcublas-11-5 libcublas-dev-11-5 libcufft-11-5 libcufft-dev-11-5 libcufile-11-5 libcufile-dev-11-5 libcurand-11-5 libcurand-dev-11-5 libcusolver-11-5
  libcusolver-dev-11-5 libcusparse-11-5 libcusparse-dev-11-5 libnpp-11-5 libnpp-dev-11-5 libnvidia-cfg1-495 libnvidia-common-495 libnvidia-compute-495 libnvidia-decode-495 libnvidia-encode-495
  libnvidia-extra-495 libnvidia-fbc1-495 libnvidia-gl-495 libnvjpeg-11-5 libnvjpeg-dev-11-5 libopengl0 liburcu6 libxnvctrl0 nsight-compute-2021.3.0 nsight-systems-2021.3.3 nvidia-compute-utils-495
  nvidia-dkms-495 nvidia-driver-495 nvidia-kernel-common-495 nvidia-kernel-source-495 nvidia-modprobe nvidia-prime nvidia-settings nvidia-utils-495 screen-resolution-extra xserver-xorg-video-nvidia-495
0 upgraded, 83 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,532 MB of archives.
After this operation, 5,709 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y

RuntimeError: CUDA out of memory

First error 

RuntimeError: CUDA out of memory. Tried to allocate 94.00 MiB (GPU 1; 23.65 GiB total capacity; 0 bytes already allocated; 30.31 MiB free; 0 bytes reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

If you plan to tweak the PYTORCH_CUDA_ALLOC_CONF environment variable, see PyTorch#Memory management.

Second error 

RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Simply a lack of GPU memory. Free up more memory headroom.

Failed to initialize NVML: Driver/library version mismatch

Running nvidia-smi may print the following error. 

Failed to initialize NVML: Driver/library version mismatch

If it was working fine and one day, without doing anything, you log in and suddenly see this message, suspect automatic updates.

Unable to determine the device handle for GPUXXXX:XX:XX.X: Unknown Error 

Unable to determine the device handle for GPU0000:18:00.0: Unknown Error

This message usually occurs when the NVIDIA driver or CUDA library cannot properly access the GPU device.

Result of running ./cuda_12.6.0_560.28.03_linux.run: 

 Installation failed. See log at /var/log/cuda-installer.log for details.

Contents of /var/log/cuda-installer.log 

[INFO]: Driver installation detected by command: apt list --installed | grep -e nvidia-driver-[0-9][0-9][0-9] -e nvidia-[0-9][0-9][0-9]
[INFO]: Cleaning up window
[INFO]: Complete
[INFO]: Checking compiler version...
[INFO]: gcc location: /usr/bin/gcc

[INFO]: gcc version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)

[INFO]: Initializing menu
[INFO]: nvidia-fs.setKOVersion(2.22.3)
[INFO]: Setup complete
[INFO]: Installing: Driver
[INFO]: Installing: 560.28.03
[INFO]: Executing NVIDIA-Linux-x86_64-560.28.03.run --ui=none --no-questions --accept-license --disable-nouveau --no-cc-version-check --install-libglvnd  2>&1
[INFO]: Finished with code: 256
[ERROR]: Install of driver component failed. Consult the driver log at /var/log/nvidia-installer.log for more details.
[ERROR]: Install of 560.28.03 failed, quitting

Contents of /var/log/nvidia-installer.log: 

nvidia-installer log file '/var/log/nvidia-installer.log'
creation time: Thu May 22 21:43:52 2025
installer version: 560.28.03

PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

nvidia-installer command line:
    ./nvidia-installer
    --ui=none
    --no-questions
    --accept-license
    --disable-nouveau
    --no-cc-version-check
    --install-libglvnd

Using built-in stream user interface
-> Detected 32 CPUs online; setting concurrency level to 32.
-> Scanning the initramfs with lsinitramfs...
-> Executing: /usr/bin/lsinitramfs   -l /boot/initrd.img-5.4.0-193-generic
-> Multiple kernel module types are available for this system. Which would you like to use? (Answer: MIT/GPL)
WARNING: An NVIDIA kernel module 'nvidia-uvm' appears to be already loaded in your kernel.  This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading.  Some of the sanity checks that nvidia-installer performs to detect potential installation problems are not possible while an NVIDIA kernel module is running.
-> Would you like to continue installation and skip the sanity checks? If not, please abort the installation, then close any programs which may be using the NVIDIA GPU(s), and attempt installation again. (Answer: Abort installation)
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.

cudaErrorInitializationError: initialization error

If this error occurs when using multiprocessing in Python, add the following code and run it again. 

import multiprocessing

multiprocessing.set_start_method("spawn", force=True)

== unrecognized command-line option '-ftrivial-auto-var-init=zero' == On Ubuntu 22.04, checking the /var/log/nvidia-installer.log file shows the following error: 

cc: error: unrecognized command-line option '-ftrivial-auto-var-init=zero'

If your GCC version is 11, upgrade GCC to 12 with the update-alternatives command. 

sudo apt install gcc-12 
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 110
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 120
sudo update-alternatives --config gcc

NVRM: GPU 0000:01:00.0 is already bound to nouveau

Switching to Ctrl + Alt + F4 keeps printing the following message: 

NVRM: GPU 0000:01:00.0 is already bound to nouveau
NVRM: GPU 0000:01:00.0 is already bound to nouveau
...

If nouveau already holds the GPU, the NVIDIA driver cannot be installed/loaded. Add it to the blacklist, regenerate the initramfs, and reboot.

Blacklist nouveau. 

# 블랙리스트 추가
echo -e "blacklist nouveau\noptions nouveau modeset=0" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf

# initramfs 재생성
sudo update-initramfs -u

# 재부팅
sudo reboot

See also

author Zer0 sitemap