Open Neural Network Exchange

lastmod 2024-07-04created 2024-05-1615 edits

The Open Neural Network Exchange (ONNX) is an open-source artificial intelligence ecosystem of technology companies and research organizations that establish open standards for representing machine learning algorithms and software tools to promote innovation and collaboration in the AI sector. ONNX is available on GitHub.

History

ONNX was originally developed by the PyTorch team at Facebook under the name Toffee. In September 2017 it was renamed ONNX and announced by Facebook and Microsoft. Later, IBM, Huawei, Intel, AMD, Arm and Qualcomm announced support for the initiative.

In October 2017, Microsoft announced that it would add its Cognitive Toolkit and Project Brainwave platform to the initiative.

In November 2019, ONNX was accepted as a graduate project in Linux Foundation AI.

In October 2020, Zetane Systems became a member of the ONNX ecosystem.

Install

Install ONNX Runtime | onnxruntime

GPU support

deep learning - How do you run a ONNX model on a GPU? - Stack Overflow

If you need the GPU, remove the CPU-only version and install the GPU runtime.

pip uninstall onnxruntime
pip install onnxruntime-gpu

Verify the device support for onnxruntime environment

import onnxruntime as rt
print(rt.get_device()) # "GPU"

If you still encounter any issue please check with your cuda and CuDNN versions, that must be compatible to each other. Please refer this link here to understand about the version compatibility between cuda and CuDNN.

ONNX Runtime Execution Providers

Execution Providers | onnxruntime

Summary of supported Execution Providers

CPU	GPU	IoT/Edge/Mobile	Other
Default CPU	NVIDIA CUDA	Intel OpenVINO	Rockchip NPU (preview)
Intel DNNL	NVIDIA TensorRT	ARM Compute Library (preview)	Xilinx Vitis-AI (preview)
TVM (preview)	DirectML	Android Neural Networks API	Huawei CANN (preview)
Intel OpenVINO	AMD MIGraphX	ARM-NN (preview)	AZURE (preview)
XNNPACK	Intel OpenVINO	CoreML (preview)
	AMD ROCm	TVM (preview)
	TVM (preview)	Qualcomm QNN
		XNNPACK

APIs for Execution Provider

The same ONNX Runtime API is used across all EPs. This provides the consistent interface for applications to run with different HW acceleration platforms. The APIs to set EP options are available across Python, C/C++/C#, Java and node.js.

Note we are updating our API support to get parity across all language binding and will update specifics here.

get_providers - Return list of registered execution providers.
get_provider_options - Return the registered execution providers' configurations.
set_providers - Register the given list of execution providers. The underlying session is re-created. The list of providers is ordered by Priority. For example ['CUDAExecutionProvider', 'CPUExecutionProvider'] means execute a node using CUDAExecutionProvider if capable, otherwise execute using CPUExecutionProvider.

Use Execution Providers

import onnxruntime as rt

#define the priority order for the execution providers
# prefer CUDA Execution Provider over CPU Execution Provider
EP_list = ['CUDAExecutionProvider', 'CPUExecutionProvider']

# initialize the model.onnx
sess = rt.InferenceSession("model.onnx", providers=EP_list)

# get the outputs metadata as a list of :class:`onnxruntime.NodeArg`
output_name = sess.get_outputs()[0].name

# get the inputs metadata as a list of :class:`onnxruntime.NodeArg`
input_name = sess.get_inputs()[0].name

# inference run using image_data as the input to the model 
detections = sess.run([output_name], {input_name: image_data})[0]

print("Output shape:", detections.shape)

# Process the image to mark the inference points 
image = post.image_postprocess(original_image, input_size, detections)
image = Image.fromarray(image)
image.save("kite-with-objects.jpg")

# Update EP priority to only CPUExecutionProvider
sess.set_providers(['CPUExecutionProvider'])

cpu_detection = sess.run(...)

Load and run a model

API - Python API documentation

InferenceSession is the main class of ONNX Runtime. It is used to load and run an ONNX model, and to specify environment and application configuration options.

session = onnxruntime.InferenceSession('model.onnx')

outputs = session.run([output names], inputs)

ONNX and ORT format models consist of a computation graph, modeled as operators, and implemented as optimized operator kernels for different hardware targets. ONNX Runtime orchestrates the execution of operator kernels via execution providers. An execution provider contains the set of kernels for a specific execution target (CPU, GPU, IoT etc). Execution providers are configured using the providers parameter. Kernels from different execution providers are chosen in the priority order given in the list of providers. In the example below, if there is a kernel in the CUDA execution provider, ONNX Runtime executes that kernel on the GPU. If not, the kernel is executed on the CPU.

session = onnxruntime.InferenceSession(
        model, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)

The list of available execution providers can be found here: #ONNX Runtime Execution Providers.

Since ONNX Runtime 1.10, you must explicitly specify the execution provider for your target. Running on CPU is the only time the API allows no explicit setting of the providers parameter. In the following examples, CUDAExecutionProvider and CPUExecutionProvider are used, assuming the application is running on NVIDIA GPUs. Replace these with the execution provider specific to your environment.

You can supply other session configurations via the session options parameter. For example, to enable profiling on the session:

options = onnxruntime.SessionOptions()
options.enable_profiling=True
session = onnxruntime.InferenceSession(
        'model.onnx',
        sess_options=options,
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)

Data inputs and outputs

API - Python API documentation

The ONNX Runtime inference session consumes and produces data using its OrtValue class.

Data on CPU

On CPU (the default), OrtValues can be mapped to and from native Python data structures: numpy arrays, dictionaries and lists of numpy arrays.

# X is numpy array on cpu
ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X)
ortvalue.device_name()  # 'cpu'
ortvalue.shape()        # shape of the numpy array X
ortvalue.data_type()    # 'tensor(float)'
ortvalue.is_tensor()    # 'True'
np.array_equal(ortvalue.numpy(), X)  # 'True'

# ortvalue can be provided as part of the input feed to a model
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
results = session.run(["Y"], {"X": ortvalue})

By default, ONNX Runtime always places input(s) and output(s) on CPU. Having the data on CPU may not be optimal if the input or output is consumed and produced on a device other than CPU, because it introduces data copies between CPU and the device.

Data on device

ONNX Runtime supports a custom data structure that supports all ONNX data formats, allowing users to place the data backing these on a device (for example, a CUDA-supported device). In ONNX Runtime, this is called IOBinding.

To use the IOBinding feature, replace InferenceSession.run() with InferenceSession.run_with_iobinding().

The graph is executed on a device other than CPU, for example CUDA. Users can use IOBinding to copy the data onto the GPU.

# X is numpy array on cpu
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
# OnnxRuntime은 CUDA 장치의 노드에서 'input'이 소비될 때 데이터를 CUDA 장치로 복사합니다.
io_binding.bind_cpu_input('input', X)
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

The input data is on a device, and users use the input directly. The output data is on CPU.

# X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr())
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

Both the input data and the output data are on a device; users use the input directly and also place the output on the device.

#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
Y_ortvalue = onnxruntime.OrtValue.ortvalue_from_shape_and_type([3, 2], np.float32, 'cuda', 0)  # Change the shape to the actual shape of the output being bound
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
io_binding.bind_input(
        name='input',
        device_type=X_ortvalue.device_name(),
        device_id=0,
        element_type=np.float32,
        shape=X_ortvalue.shape(),
        buffer_ptr=X_ortvalue.data_ptr()
)
io_binding.bind_output(
        name='output',
        device_type=Y_ortvalue.device_name(),
        device_id=0,
        element_type=np.float32,
        shape=Y_ortvalue.shape(),
        buffer_ptr=Y_ortvalue.data_ptr()
)
session.run_with_iobinding(io_binding)

Users can request ONNX Runtime to allocate an output on a device. This is particularly useful for dynamically shaped outputs. Users can use the get_outputs() API to access the OrtValue(s) corresponding to the allocated output(s). Users can thus consume the memory allocated by ONNX Runtime for the output as an OrtValue.

#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
io_binding.bind_input(
        name='input',
        device_type=X_ortvalue.device_name(),
        device_id=0,
        element_type=np.float32,
        shape=X_ortvalue.shape(),
        buffer_ptr=X_ortvalue.data_ptr()
)
#Request ONNX Runtime to bind and allocate memory on CUDA for 'output'
io_binding.bind_output('output', 'cuda')
session.run_with_iobinding(io_binding)
# The following call returns an OrtValue which has data allocated by ONNX Runtime on CUDA
ort_output = io_binding.get_outputs()[0]

In addition, ONNX Runtime supports directly working with OrtValue (s) while inferencing a model if provided as part of the input feed.

Users can bind OrtValue(s) directly.

#X is numpy array on cpu
#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
Y_ortvalue = onnxruntime.OrtValue.ortvalue_from_shape_and_type([3, 2], np.float32, 'cuda', 0)  # Change the shape to the actual shape of the output being bound
session = onnxruntime.InferenceSession(
        'model.onnx',
        providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
)
io_binding = session.io_binding()
io_binding.bind_ortvalue_input('input', X_ortvalue)
io_binding.bind_ortvalue_output('output', Y_ortvalue)
session.run_with_iobinding(io_binding)

You can also bind inputs and outputs directly to a PyTorch tensor.

# X is a PyTorch tensor on device
session = onnxruntime.InferenceSession('model.onnx', providers=['CUDAExecutionProvider', 'CPUExecutionProvider']))
binding = session.io_binding()

X_tensor = X.contiguous()

binding.bind_input(
    name='X',
    device_type='cuda',
    device_id=0,
    element_type=np.float32,
    shape=tuple(x_tensor.shape),
    buffer_ptr=x_tensor.data_ptr(),
    )

## Allocate the PyTorch tensor for the model output
Y_shape = ... # You need to specify the output PyTorch tensor shape
Y_tensor = torch.empty(Y_shape, dtype=torch.float32, device='cuda:0').contiguous()
binding.bind_output(
    name='Y',
    device_type='cuda',
    device_id=0,
    element_type=np.float32,
    shape=tuple(Y_tensor.shape),
    buffer_ptr=Y_tensor.data_ptr(),
)

session.run_with_iobinding(binding)

You can also see code examples of this API in in the ONNX Runtime inferences examples.

CUDA Input/Output Binding Example

(Python) How to run inference directly on data uploaded to the onnxruntime GPU (device) without downloading it to the CPU (host)

import torch
import torchvision
import numpy as np
import onnxruntime

model = torchvision.models.resnet18(weights=None)
model.eval()

input_on_cpu = torch.randn((1, 3, 224, 224), dtype=torch.float32)
torch.onnx.export(model, input_on_cpu, "resnet18.onnx", input_names=['input'], output_names=['output'])

session = onnxruntime.InferenceSession("resnet18.onnx", providers=['CUDAExecutionProvider'])

output = session.run(None, {'input': input_on_cpu.numpy()})[0]
print(output.sum())

input_on_gpu = input_on_cpu.to('cuda')
iobinding = session.io_binding()
iobinding.bind_input(name='input', 
                     device_type='cuda', 
                     device_id=0, 
                     element_type=np.float32,
                     shape=input_on_gpu.shape,
                     buffer_ptr=input_on_gpu.data_ptr())
iobinding.bind_output(name='output', device_type='cuda', device_id=0)

session.run_with_iobinding(iobinding)
output = iobinding.copy_outputs_to_cpu()[0]
print(output.sum())

ONNX Runtime Performance

Performance | onnxruntime

Ways to optimize performance:

Tune performance
Model optimizations
Transformers optimizer
End to end optimization with Olive
Device tensors
Tune Mobile Performance (ORT <1.10 only)

Favorite site

author Zer0 sitemap