NVIDIA DeepStream

lastmod 2024-10-24created 2024-05-1727 edits

SDK

• DeepStream SDK | NVIDIA Developer | NVIDIA Developer
• DeepStream | NVIDIA NGC

DeepStream's multi-platform support lets you develop vision AI applications and services faster. You can deploy on-premises, edge to edge, and to the cloud with a single click.

Gst-nvtracker

• Gst-nvtracker — DeepStream documentation 6.4 documentation

The Gst-nvtracker plugin allows the DS pipeline to use a low-level tracker library to track the detected objects over time persistently with unique IDs. It supports any low-level library that implements NvDsTracker API, including the reference implementations provided by the NvMultiObjectTracker library: IOU, NvSORT, NvDeepSORT and NvDCF trackers. As part of this API, the plugin queries the low-level library for capabilities and requirements concerning the input format, memory type, and additional feature support. Based on these queries, the plugin then converts the input frame buffers into the format requested by the low-level tracker library. For example, the NvDeepSORT and NvDCF trackers use NV12 or RGBA, while IOU and NvSORT requires no video frame buffers at all.

Install

• Installation — DeepStream documentation

Install prerequisite packages

• Installation — DeepStream documentation # Install prerequisite packages

Enter the following commands to install the prerequisite packages:

$ sudo apt install \
libssl3 \
libssl-dev \
libgstreamer1.0-0 \
gstreamer1.0-tools \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-plugins-ugly \
gstreamer1.0-libav \
libgstreamer-plugins-base1.0-dev \
libgstrtspserver-1.0-0 \
libjansson4 \
libyaml-cpp-dev

DeepStream Single-View 3D Tracking

• NVIDIA DeepStream 7.0 Milestone Release for Next-Gen Vision AI Development | NVIDIA Technical Blog
• See 3D Multi-Object Tracking/Multi-Object Tracking/Multiple object tracking/MOT/Motion estimation/Object Tracking.

DeepStream container for Enterprise Grade GPUs

• DeepStream | NVIDIA NGC

Container list:

If unsure, use "deepstream:7.1-samples-multiarch":

docker pull nvcr.io/nvidia/deepstream:7.1-samples-multiarch

GStreamer Plugin Overview

• GStreamer Plugin Overview — DeepStream documentation
• MetaData in the DeepStream SDK
• Gst-nvdspreprocess (Alpha)
• Gst-nvinfer
• Gst-nvinferserver
• Gst-nvtracker
• Gst-nvstreammux
• Gst-nvstreammux New
• Gst-nvstreamdemux
• Gst-nvmultistreamtiler
• Gst-nvdsosd
• Gst-nvdsmetautils
• Gst-nvdsvideotemplate
• Gst-nvdsaudiotemplate
• Gst-nvvideoconvert
• Gst-nvdewarper
• Gst-nvof
• Gst-nvofvisual
• Gst-nvsegvisual
• Gst-nvvideo4linux2
• Gst-nvjpegdec
• Gst-nvimagedec
• Gst-nvjpegenc
• Gst-nvimageenc
• Gst-nvmsgconv
• Gst-nvmsgbroker
• Gst-nvdsanalytics
• Gst-nvdsudpsrc
• Gst-nvdsudpsink
• Gst-nvdspostprocess (Alpha)
• Gst-nvds3dfilter
• Gst-nvds3dbridge
• Gst-nvds3dmixer
• Gst-NvDsUcx
• Gst-nvdsxfer
• Gst-nvvideotestsrc
• Gst-nvmultiurisrcbin
• Gst-nvurisrcbin

How to install CUDA libraries

• Docker Containers — DeepStream documentation # Suggested Setup for Video Subsystem on x86 dGPU docker

1. sudo apt-get -y update
2. sudo apt-get -y install cuda
3. sudo apt-get -y install cuda-toolkit-12-2
4. Change the default CUDA version inside Docker to Cuda-12.2 using update-alternatives: update-alternatives --set cuda /usr/local/cuda-12.2
5. To check the CUDA version currently in use inside Docker, run: update-alternatives --display cuda

What is the .etlt file extension?

The DeepStream SDK is a streaming analytics toolkit provided by NVIDIA, mainly used for real-time video analytics, AI-based video processing, and so on. Among the file extensions used in DeepStream, the .etlt file is known as "Efficient TensorRT runtime files". An ETLT file is an intermediate file format used in DeepStream applications to optimize TensorFlow or ONNX models and build TensorRT engines.

These .etlt files are generated by the NVIDIA Transfer Learning Toolkit (TLT). TLT is a tool that lets users retrain pretrained AI models and customize them for their own applications. An ETLT file is the file format provided before a TLT model is converted to TensorRT. Using ETLT files in DeepStream lets you convert them to TensorRT for high-performance AI inference.

Summary:

• ETLT file: an intermediate file format generated by the NVIDIA Transfer Learning Toolkit (TLT).
• Purpose: converted to a TensorRT engine and used in DeepStream applications.

To use this file, you must train or retrain a model with NVIDIA's TLT and then convert it to a TensorRT engine.

How to load and call it from the DeepStream SDK at the C level

To load and call a model in the DeepStream SDK using an .etlt file, you basically need to build a TensorRT engine. NVIDIA's TensorRT is a deep learning optimization tool for high-performance inference, and the DeepStream SDK uses it to run AI models. The general process for implementing this through the C API is as follows.

Convert the ETLT file to a TensorRT engine

The DeepStream SDK itself does not use ETLT files directly; they must be converted to TensorRT engine files. This is done using the NVIDIA Transfer Learning Toolkit (TLT) or the TensorRT Python API. A typical conversion command is:

tlt-converter -k <your_key> -o <output_layer> -e <output_engine.trt> <input_model.etlt>

In the command above:

• -k - model conversion key (needed to decrypt the model encrypted by TLT).
• -o - output layer.
• -e - path of the converted TensorRT engine file.
• input_model.etlt - the ETLT file to convert.

Load the TensorRT engine in DeepStream

After converting the ETLT file to a TensorRT engine file, you can load this engine file in a DeepStream application and use it for inference. The basic C API code structure is as follows.

Loading the model

Loading a TensorRT engine in C uses the nvinfer plugin provided by DeepStream. This plugin loads a pretrained model and performs inference based on it. nvinfer is used through a GstElement.

// nvinfer 플러그인 요소 생성
GstElement *nvinfer = gst_element_factory_make("nvinfer", "primary-nvinference-engine");

// TensorRT 엔진 파일을 지정
g_object_set(G_OBJECT(nvinfer), "model-engine-file", "path/to/your/model.trt", NULL);

Calling inference

After loading the model, you can build a GstPipeline and perform inference while streaming video. The basic pipeline structure is as follows:

GstElement *pipeline = gst_pipeline_new("deepstream-pipeline");

GstElement *source = gst_element_factory_make("filesrc", "file-source");
GstElement *decode = gst_element_factory_make("decodebin", "decode");
GstElement *nvinfer = gst_element_factory_make("nvinfer", "primary-nvinference-engine");
GstElement *sink = gst_element_factory_make("nveglglessink", "video-output");

// 파이프라인에 요소 추가
gst_bin_add_many(GST_BIN(pipeline), source, decode, nvinfer, sink, NULL);

// 요소 간 링크
gst_element_link_many(source, decode, nvinfer, sink, NULL);

Inference settings

Various inference settings can be configured on the nvinfer element. For example, you can set the batch size, input size, label file, and so on.

g_object_set(G_OBJECT(nvinfer), "batch-size", 1, NULL);
g_object_set(G_OBJECT(nvinfer), "gpu-id", 0, NULL);
g_object_set(G_OBJECT(nvinfer), "config-file-path", "path/to/config_infer_primary.txt", NULL);

The file specified in config-file-path contains additional settings for the inference model (labels, pre-/post-processing methods, etc.).

Running the DeepStream pipeline

After building the pipeline, running it with gst_element_set_state() makes DeepStream perform inference while processing the video stream.

gst_element_set_state(pipeline, GST_STATE_PLAYING);

Example config file (config_infer_primary.txt)

[property]
gpu-id=0
net-scale-factor=0.017
model-engine-file=path/to/your/model.trt
labelfile-path=path/to/labels.txt
batch-size=1
network-mode=2
num-detected-classes=80
interval=0
gie-unique-id=1

In this way, you can use the C API in DeepStream to load a TensorRT engine converted from an ETLT file and perform inference.

Notes

• Converting the .etlt file to a .trt file first is mandatory.
• In the DeepStream pipeline, the model is loaded and inference is run through the nvinfer element.
• Inference settings can be managed through a separate config file.

Applying this method, you can perform inference with an ETLT model in C on the DeepStream SDK.

answer-jet setup

See answer-jet.

Troubleshooting

=== unsatisfied condition: cuda>=12.6 ===

docker run --gpus all --rm --name deepstream nvcr.io/nvidia/deepstream:7.1-samples-multiarch

Output:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.6, please update your driver to a newer version, or use an earlier cuda container: unknown.

Update the driver. Or use version 6.4 instead of 7.1.

docker run --gpus all --rm --name deepstream nvcr.io/nvidia/deepstream:6.4-samples-multiarch

GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument. Aborting.

• Installation — DeepStream documentation # Prerequisites

In short, the problem is caused by a bug in glib 2.0-2.72, which ships with Ubuntu 22.04. It was fixed in glib-2.76, and installing it is required to fix the problem.

Before proceeding, check the glib version.

pkg-config --modversion glib-2.0

If it is lower than 2.76, you need to install it yourself.

Migrate glib to newer version

In order to migrate to newer glib version (e.g. 2.76.6) follow below steps:

Prerequisites: Install below packages:

pip3 install meson
pip3 install ninja

(If the pip3 command is missing):

apt install python3 python3-pip

Compilation and installation steps:

git clone https://github.com/GNOME/glib.git
cd glib
git checkout <glib-version-branch>
# e.g. 2.76.6
meson build --prefix=/usr
ninja -C build/
cd build/
ninja install

Check and confirm the newly installed glib version:

pkg-config --modversion glib-2.0

cannot open shared object file

deepstream-app test

Warnings that follow:

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.674: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.709: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstopenmpt.so': libmpg123.so.0: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.742: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstmpg123.so': libmpg123.so.0: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.748: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstchromaprint.so': libavcodec.so.58: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.775: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstmpeg2enc.so': libmpeg2encpp-2.1.so.0: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.814: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstmpeg2dec.so': libmpeg2.so.0: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.857: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.1: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.921: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(deepstream-app:16): GStreamer-WARNING **: 02:35:33.935: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_ucx.so': libucs.so.0: cannot open shared object file: No such file or directory
...

The related plugins failed to load. As a result, the corresponding libraries need to be installed.

If the issue repeats even after installing, see Advanced Package Tool#When a library cannot be found but the package is reported as installed.

If needed, you may have to reinstall with apt install --reinstall ....

See also

• NVIDIA
• JetPack
• Xavier