Skip to content
Search

NVidia Triton Inference Server

lastmod 2025-07-14created 2024-05-1430 edits

(For self-hostable S3-compatible object storage, see Triton Object Storage)

NVIDIA Triton™ Inference Server is open-source inference serving software that helps standardize model deployment and execution and delivers fast and scalable AI in production.

Part of the NVIDIA AI platform, Triton Inference Server streamlines and standardizes AI inference by enabling teams to deploy, run, and scale AI models trained in any framework on GPU- or CPU-based infrastructure. With Triton, AI researchers and data scientists can freely choose the right framework for their projects without impacting production deployment, and developers can deliver high-performance inference across cloud, on-premises, edge, and embedded devices.

Run on System with GPUs 

docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/full/path/to/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models

For reference, here is how I did it: 

docker pull nvcr.io/nvidia/tritonserver:24.05-py3
docker run --rm -it \
  --gpus=all \
  -p 8000:8000 \
  -p 8001:8001 \
  -p 8002:8002 \
  -v ./model:/models \
  nvcr.io/nvidia/tritonserver:24.05-py3 \
  tritonserver \
  --model-repository=/models
  • 8000 - for HTTP communication
  • 8001 - for gRPC communication
  • 8002 - for Metrics
  • -v {로컬 모델 경로}/models - the local model path specified here is where the trained models to be served are located.

Verify Triton Is Running Correctly

Use Triton’s ready endpoint to verify that the server and the models are ready for inference. From the host system use curl to access the HTTP endpoint that indicates server status. 

curl -v localhost:8000/v2/health/ready

Docker images

Docker images are available:

  • The xx.yy-py3 image contains the Triton inference server with support for Tensorflow, PyTorch, TensorRT, ONNX and OpenVINO models. <- If unsure, get this one
  • The xx.yy-py3-sdk image contains Python and C++ client libraries, client examples, and the Model Analyzer.
  • The xx.yy-py3-min image is used as the base for creating custom Triton server containers as described in Customize Triton Container.
  • The xx.yy-pyt-python-py3 image contains the Triton Inference Server with support for PyTorch and Python backends only.
  • The xx.yy-tf2-python-py3 image contains the Triton Inference Server with support for TensorFlow 2.x and Python backends only.

Check the NVIDIA driver version

The triton container version to use can differ depending on the NVIDIA driver version. In my case, the driver version was 515.86.01, so I built the docker image with triton inference server container version 22.08. If your driver is older than the minimum driver version supported by the container, errors may occur, so be sure to check!

Check the container version that matches your driver version in the official documentation below.

Triton Inference Server release notes: https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html

Or check the https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html site.

Model serving

tritonclient 

pip install tritonclient[all]

Using all installs both the HTTP/REST and GRPC client libraries. There are two optional packages available, grpc and http that can be used to install support specifically for the protocol. For example, to install only the HTTP/REST client library use, 

pip install tritonclient[http]

There is another optional package namely cuda, that must be installed in order to use cuda_shared_memory utilities. all specification will install the cuda package by default but in other cases cuda needs to be explicitly specified for installing client with cuda_shared_memory support. 

pip install tritonclient[http, cuda]

How to connect over HTTP 

import tritonclient.http as httpclient
import numpy as np

# Triton Inference Server URL
url = "localhost:8000"

# 클라이언트 초기화
client = httpclient.InferenceServerClient(url=url)

# 입력 데이터 준비
inputs = []
outputs = []
input_data = np.random.rand(1, 3, 224, 224).astype(np.float32)  # 예시 입력 데이터

inputs.append(httpclient.InferInput("input_name", input_data.shape, "FP32"))
inputs[-1].set_data_from_numpy(input_data)

outputs.append(httpclient.InferRequestedOutput("output_name"))

# 추론 요청
results = client.infer("my_model", inputs, outputs=outputs)

# 결과 확인
output_data = results.as_numpy("output_name")
print(output_data)

How to use Shared Memory

When running the server, add ulimit options and options for accessing shared memory (/dev/shm). 

docker run --gpus=1 --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
  -p8000:8000 -p8001:8001 -p8002:8002 \
  -v /path/to/model/repository:/models nvcr.io/nvidia/tritonserver:23.04-py3 \
  tritonserver --model-repository=/models

Client: 

import tritonclient.http as httpclient
import tritonclient.utils.shared_memory as shm
import numpy as np
import sys

# Triton Inference Server URL
url = "localhost:8000"

# 클라이언트 초기화
client = httpclient.InferenceServerClient(url=url)

# 모델 정보
model_name = "my_model"
input_name = "input_name"
output_name = "output_name"

# 입력 데이터 준비
input_data = np.random.rand(1, 3, 224, 224).astype(np.float32)

# Shared memory 설정
input_byte_size = input_data.nbytes
output_byte_size = input_data.nbytes  # 예제에서는 입력과 출력 크기가 같다고 가정

# Shared memory 등록
input_shm_region = shm.create_shared_memory_region("input_data", "/input", input_byte_size)
output_shm_region = shm.create_shared_memory_region("output_data", "/output", output_byte_size)

# Shared memory에 데이터 쓰기
shm.set_shared_memory_region(input_shm_region, [input_data])

# Shared memory 등록
client.register_system_shared_memory("input_data", "/input", input_byte_size)
client.register_system_shared_memory("output_data", "/output", output_byte_size)

# 입력 및 출력 설정
inputs = []
outputs = []

inputs.append(httpclient.InferInput(input_name, input_data.shape, "FP32"))
inputs[-1].set_shared_memory("input_data", input_byte_size)

outputs.append(httpclient.InferRequestedOutput(output_name, binary_data=True))
outputs[-1].set_shared_memory("output_data", output_byte_size)

# 추론 요청
results = client.infer(model_name, inputs, outputs=outputs)

# 결과 읽기
output_data = shm.get_shared_memory_region(output_shm_region, output_byte_size, np.float32)
print(output_data)

# Shared memory 해제
client.unregister_system_shared_memory("input_data")
client.unregister_system_shared_memory("output_data")
shm.destroy_shared_memory_region(input_shm_region)
shm.destroy_shared_memory_region(output_shm_region)

Shared-Memory Extension

System Shared Memory

Using system shared memory to communicate tensors between the client library and triton can improve performance.

Because python has no standard way to allocate and access shared memory1, a simple example module is provided.

This example lets you create, configure, and delete system shared memory using the python client library.

CUDA Shared Memory

Quick Start

Building a k8s server

Python Client Examples

List of client example files

  • ensemble_image_client.py
  • grpc_client.py
  • grpc_explicit_byte_content_client.py
  • grpc_explicit_int8_content_client.py
  • grpc_explicit_int_content_client.py
  • grpc_image_client.py
  • image_client.py
  • memory_growth_test.py
  • reuse_infer_objects_client.py
  • simple_grpc_aio_infer_client.py
  • simple_grpc_aio_sequence_stream_infer_client.py
  • simple_grpc_async_infer_client.py
  • simple_grpc_cudashm_client.py
  • simple_grpc_custom_args_client.py
  • simple_grpc_custom_repeat.py
  • simple_grpc_health_metadata.py
  • simple_grpc_infer_client.py
  • simple_grpc_keepalive_client.py
  • simple_grpc_model_control.py
  • simple_grpc_sequence_stream_infer_client.py
  • simple_grpc_sequence_sync_infer_client.py
  • simple_grpc_shm_client.py
  • simple_grpc_shm_string_client.py
  • simple_grpc_string_infer_client.py
  • simple_http_aio_infer_client.py
  • simple_http_async_infer_client.py
  • simple_http_cudashm_client.py
  • simple_http_health_metadata.py
  • simple_http_infer_client.py
  • simple_http_model_control.py
  • simple_http_sequence_sync_infer_client.py
  • simple_http_shm_client.py
  • simple_http_shm_string_client.py
  • simple_http_string_infer_client.py

ONNX Runtime Backend

See also

Favorite site

Getting started with Triton Inference Server

References

  1. Needs verification ↩

author Zer0 sitemap