Nvidia-persistenced
lastmod 2023-03-16created 2022-02-241 edits
Persistence Daemon
NVIDIA provides a user-space daemon on Linux to support persistence of driver state across Cuda job runs. The daemon approach provides a more elegant and robust solution to this problem than persistence mode.
NVIDIA will support both solutions in the near future (possibly through Cuda 8.0), but will focus all future development and bug fixes on the daemon.
The daemon is installed in /usr/bin, and sample installation and init scripts are included with the driver in the documentation directory. The scripts are provided as a guide for installing the daemon to run at system startup on some common init systems. Due to the variety of init system configurations, some changes may be needed for specific distributions.
NVIDIA recommends that customers switch to this daemon approach as soon as possible.
Supported Environments
- Drivers: R319 and higher
- OSes: All standard driver-supported Linux platforms
- GPUs: All shipping Tesla, Quadro and GRID products
Implementation Details
On Linux systems running the NVIDIA GPU driver, a client attaches to the GPU by opening the device file. Conversely, the GPU is detached by closing the device file. The GPU state stays loaded in the driver as long as one or more clients have the device file open. When all clients close the device file, the GPU state is unloaded unless persistence mode is enabled.
To simulate a graphical environment without incurring the overhead of a user-space graphics driver, we implemented the NVIDIA Persistence Daemon, which by default runs in the background and goes into sleep mode with the device files open. The daemon uses libnvidia-cfg to open and close the correct device files based on PCI bus address, and provides an RPC interface to control persistence mode for each GPU individually. So while the daemon keeps the device files open, the GPU has at least one client, the daemon, attached, and the driver does not unload the GPU state. Once the daemon starts running, it stays in the background until it exits, even if persistence mode is disabled for all devices.
Due to the nature of the solution, the daemon can be used as a drop-in replacement for what we currently call "legacy persistence mode," implemented in the NVIDIA kernel-mode driver. NVIDIA SMI was updated in driver version 319 to use the daemon's RPC interface to set persistence mode through the daemon if it is running, and falls back to the legacy persistence mode setting in the kernel-mode driver if the daemon is not running. Running. This is all handled transparently by NVIDIA SMI, so there should be no change in how persistence mode is configured. Eventually, once it is widely adopted in the relevant use cases, legacy persistence mode will be deprecated and removed in favor of the NVIDIA Persistence Daemon.
Permissions and Security
The NVIDIA Persistence Daemon provides a more robust persistence mode implementation on Linux, because it simply mimics an external client of the GPU but never actually uses the GPU for any work. In this way, it keeps the NVIDIA GPU driver operating within the assumptions of its original design.
Once the daemon is running, there is minimal overhead to keep persistence mode enabled. The daemon simply sleeps while waiting for commands.
Superuser privileges are not required to run the daemon, but they are required to set up some runtime data in /var/run. The daemon allows two mechanisms for running as a user without superuser privileges.
An administrator (or a script running with superuser privileges) can create the /var/run/nvidia-persistenced directory and chown it to the user the daemon will run as. The daemon can then be run as the intended user with su or something similar. In this case, the /var/run/nvidia-persistenced directory is not removed when the daemon exits.
The daemon can be started with superuser privileges and given the --user option. This makes the daemon drop superuser privileges as soon as possible after creating the /var/run/nvidia-persistenced directory and run as the specified user. With this mechanism, the user may not have write permission to /var/run, so the daemon may fail to remove the /var/run/nvidia-persistenced directory when it exits.
In both cases, the daemon may fail to remove the runtime data directory when it exits, so this should generally be handled by the init script or service for the daemon.
The daemon can also be run with permanent superuser privileges by simply omitting the --user option, but this is not recommended and is not needed for it to function.
The daemon also provides a --verbose option that increases logging output to syslog for debugging purposes.
The daemon's source code is provided under the MIT license, allowing second- and third-party security audits.
Usage
To run the NVIDIA Persistence Daemon, run the following (as root):
After performing the minimal setup tasks that require superuser privileges, the daemon drops superuser privileges and runs as user 'foo'.
You can change the persistence mode setting with NVIDIA SMI. For example, to disable persistence mode on all GPUs, simply run the following (as root).
Please see the nvidia-persistenced(1) manpage, which is installed by the NVIDIA GPU driver installer, or the output of:
Installation Caveats
The reason we cannot immediately discontinue legacy persistence mode and transparently switch to the NVIDIA Persistence Daemon is that, for now, we cannot guarantee that the NVIDIA Persistence Daemon will be running. That would be a feature regression, since persistence mode might not be available by default.
The NVIDIA Persistence daemon ships with the NVIDIA Linux GPU driver starting with driver version 319, and is installed by the installer as /usr/bin/nvidia-persistenced. Ideally, the daemon would start at system initialization and exit at system shutdown according to the Linux distribution's init system, transparently to the user. Unfortunately, there is no single standard on Linux for installing an application to start at system initialization, so this cannot be done reliably across the wide range of systems supported by the NVIDIA GPU driver.
Therefore, we would like to encourage individual distributions that repackage the NVIDIA GPU driver, typically for installation through a package manager, to install the NVIDIA Persistence Daemon to start at system initialization. This is an almost trivial task once the init system is known. To help with this installation, we provide sample "init scripts" in the driver package. These scripts try to cover the three most widespread init systems found in Linux distributions today: SystemV, systemd, and Upstart. The sample scripts also come with an installer script that tries to detect the init system and install the appropriate script for the user. The sample scripts and the installer script are installed by the NVIDIA GPU driver installer to /usr/share/doc/NVIDIA_GLX-1.0/sample/nvidia-persistenced-init.tar.bz2. Since they cannot be guaranteed to work correctly by default on every supported system, the driver installer does not extract or run them.
By default, the installer script tries to create a new system user to run the daemon, and the sample init scripts demonstrate the second option described in Permissions and Security for running the daemon without superuser privileges.
Checking the systemd service
Just add nvidia-persistenced.service as follows.