Skip to content

ONNX

ONNX (Open Neural Network Exchange) is an open-source format for representing machine learning models, designed to enable interoperability between different deep learning frameworks

C++

---

---

#TODO: move to it's on section

Install on nvidia jetson with jetpack 6.2

!!! warning "onnx version" - The last package from Jetson Zoo is for jetpack 6.0 - I found the last version 1.23 pypi.jetson-ai-lab.io it's installed successfully

pypi.jetson-ai-lab.io

many sites refer to nvidia url: https://pypi.jetson-ai-lab.dev/jp6/cu126, note to the dev suffix

1
2
the `dev` part reploace by `io`
https://pypi.jetson-ai-lab.io/jp6/cu126

Installed on docker

Install onnxruntime-gpu on docker that run on jetson orin using vscode and devcontainer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
.
|-- .devcontainer
|   |-- Dockerfile
|   `-- devcontainer.json
|-- .gitignore
|-- .vscode
|-- README.md
|-- docker-compose.yaml
|-- requirements.txt
`-- scripts
    `-- check.py

At last i update my host with the command sudo apt install nvidia-jetpack And use the docker image nvcr.io/nvidia/l4t-jetpack:r36.4.0 find at NGC Catalog

Package version note
jetpack R36.4.3 cat /etc/nv_tegra_release
cuda 2.6 nvcc --version
cudnn 9.3.0 dpkg -l \| grep cudnn
tensorrt 10.3.0.30 dpkg -l \| grep tensorrt
onnxruntime 1.23.0 python3 -c "import onnxruntime as ort; print('ONNX Runtime version:', ort.__version__)"

Code

.devcontainer/devcontainer.json 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
    "name": "onnx",
    "dockerComposeFile": "../docker-compose.yaml",
    "service": "dev",
    "shutdownAction": "stopCompose",
    "workspaceFolder": "/workspace",
    "customizations": {
      "vscode": {
        "extensions": [
            "ms-python.python",
            "ms-vscode.cpptools",
            "twxs.cmake",
            "redhat.vscode-xml",
            "redhat.vscode-yaml",
            "albert.tabout",
            "actboy168.tasks",
            "streetsidesoftware.code-spell-checker",
            "mhutchie.git-graph",
            "dlech.chmod",
            "smilerobotics.urdf"
        ],
        "settings": {}
      }
    }
  }
.devcontainer/Dockerfile 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# FROM nvidia/cuda:12.6.0-cudnn-runtime-ubuntu22.04
FROM nvcr.io/nvidia/l4t-jetpack:r36.4.0

ARG USERNAME=user
ARG USER_UID=1000
ARG USER_GID=$USER_UID

# Check if "ubuntu" user exists, delete it if it does, then create the desired user
RUN if getent passwd ubuntu > /dev/null 2>&1; then \
        userdel -r ubuntu && \
        echo "Deleted existing ubuntu user"; \
    fi && \
    groupadd --gid $USER_GID $USERNAME && \
    useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME && \
    echo "Created new user $USERNAME"

# Add sudo support for the non-root user
RUN apt-get update && apt-get install -y sudo \
  && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME\
  && chmod 0440 /etc/sudoers.d/$USERNAME \
  && rm -rf /var/lib/apt/lists/*

# Install Python and pip
RUN apt-get update \
    && apt-get -y install --no-install-recommends \
        python3-pip \
        python3-dev \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Copy and install requirements
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt

# RUN pip3 install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl
RUN pip3 install https://pypi.jetson-ai-lab.io/jp6/cu126/+f/e1e/9e3dc2f4d5551/onnxruntime_gpu-1.23.0-cp310-cp310-linux_aarch64.whl#sha256=e1e9e3dc2f4d5551c5f0b5554cf490d141cd0d339a5a7f4826ac0e04f20a35fc

# RUN apt-get update \
#     && apt-get -y install --no-install-recommends \
#         wget \
#         libgl1 \
#         libglib2.0-0 \
#         libsm6 \
#         libxext6 \
#         libxrender-dev \
#     && apt-get clean -y \
#     && rm -rf /var/lib/apt/lists/*
docker-compose.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
services:
  dev:
    build:
      context: .
      dockerfile: .devcontainer/Dockerfile
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    volumes:
      - .:/workspace:cached
    hostname: dev
    network_mode: host
    stdin_open: true
    tty: true
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
requirements.txt 
1
2
opencv-python-headless
numpy<2

Run

1
2
3
4
5
6
7
8
9
import onnxruntime as ort

ort.set_default_logger_severity(0)  # 0 = VERBOSE, 1 = INFO, 2 = WARNING, 3 = ERROR, 4 = FATAL


# Show all available providers on your machine
print("Available providers:", ort.get_available_providers())
print("ONNX Runtime version:", ort.__version__)
print("Build info:", ort.get_device())

1
2
3
Available providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
ONNX Runtime version: 1.20.0
Build info: GPU

Demo: mnist

MNIST model home page

mnist-8.onnx

[W:onnxruntime:Default, device_discovery.cc:164 DiscoverDevicesForPlatform] GPU device discovery failed: device_discovery.cc:89 ReadFileContents Failed to open file: "/sys/class/drm/card1/device/vendor"

1
for known i don't found and reference that help resolve the issue
  • work with onnxruntime-gpu version 1.23
  • need numpy <2
  • images zero, six
  • model: mnist-8.onnx
  • try with the 3 providers
  • on the docker side success only with the cpu (TODO)
  • on the host all off them work
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import onnxruntime as ort
import numpy as np
import cv2

ort.set_default_logger_severity(0)  # 0 = VERBOSE, 1 = INFO, 2 = WARNING, 3 = ERROR, 4 = FATAL


# --- Load ONNX model ---
# "CPUExecutionProvider"
# CUDAExecutionProvider
# TensorrtExecutionProvider
session = ort.InferenceSession("model/mnist-8.onnx", providers=["CUDAExecutionProvider"])

# --- Load and preprocess an image (28×28 grayscale) ---
# You can replace 'digit.png' with your own file.
img = cv2.imread("images/six_1.png", cv2.IMREAD_GRAYSCALE)

if img is None:
    # fallback: create a synthetic "3"-like shape for demo
    img = np.zeros((28, 28), np.uint8)
    cv2.putText(img, "3", (4, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.9, 255, 2)

# Normalize to [0,1]
img = img.astype(np.float32) / 255.0

# Reshape to NCHW = (1,1,28,28)
img = img.reshape(1, 1, 28, 28)

# --- Run inference ---
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name

pred = session.run([output_name], {input_name: img})[0]
digit = int(np.argmax(pred))

print(f"Predicted digit: {digit}")

tutorials

References