ONNX

ONNX (Open Neural Network Exchange) is an open-source format for representing machine learning models, designed to enable interoperability between different deep learning frameworks

C++

---

#TODO: move to it's on section

Install on nvidia jetson with jetpack 6.2

onnx version

The last package from Jetson Zoo is for jetpack 6.0
I found the last version 1.23 pypi.jetson-ai-lab.io it's installed successfully
~~I try version 1.20 from ultralytics that installed and run without any issue~~

many sites refer to nvidia url: https://pypi.jetson-ai-lab.dev/jp6/cu126

the dev part reploace by io https://pypi.jetson-ai-lab.io/jp6/cu126

Installed on docker

Install onnxruntime-gpu on docker that run on jetson orin using vscode and devcontainer

.
|-- .devcontainer
|   |-- Dockerfile
|   `-- devcontainer.json
|-- .gitignore
|-- .vscode
|-- README.md
|-- docker-compose.yaml
|-- requirements.txt
`-- scripts
    `-- check.py

.devcontainer/devcontainer.json

{
    "name": "onnx",
    "dockerComposeFile": "../docker-compose.yaml",
    "service": "dev",
    "shutdownAction": "stopCompose",
    "workspaceFolder": "/workspace",
    "customizations": {
      "vscode": {
        "extensions": [
            "ms-python.python",
            "ms-vscode.cpptools",
            "twxs.cmake",
            "redhat.vscode-xml",
            "redhat.vscode-yaml",
            "albert.tabout",
            "actboy168.tasks",
            "streetsidesoftware.code-spell-checker",
            "mhutchie.git-graph",
            "dlech.chmod",
            "smilerobotics.urdf"
        ],
        "settings": {}
      }
    }
  }

.devcontainer/Dockerfile

FROM nvidia/cuda:12.6.0-cudnn-runtime-ubuntu22.04

ARG USERNAME=user
ARG USER_UID=1000
ARG USER_GID=$USER_UID

# Check if "ubuntu" user exists, delete it if it does, then create the desired user
RUN if getent passwd ubuntu > /dev/null 2>&1; then \
        userdel -r ubuntu && \
        echo "Deleted existing ubuntu user"; \
    fi && \
    groupadd --gid $USER_GID $USERNAME && \
    useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME && \
    echo "Created new user $USERNAME"

# Add sudo support for the non-root user
RUN apt-get update && apt-get install -y sudo \
  && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME\
  && chmod 0440 /etc/sudoers.d/$USERNAME \
  && rm -rf /var/lib/apt/lists/*

# Install Python and pip
RUN apt-get update \
    && apt-get -y install --no-install-recommends \
        python3-pip \
        python3-dev \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Copy and install requirements
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt

# RUN pip3 install https://github.com/ultralytics/assets/releases/download/v0.0.0/onnxruntime_gpu-1.20.0-cp310-cp310-linux_aarch64.whl
RUN pip3 install https://pypi.jetson-ai-lab.io/jp6/cu126/+f/e1e/9e3dc2f4d5551/onnxruntime_gpu-1.23.0-cp310-cp310-linux_aarch64.whl#sha256=e1e9e3dc2f4d5551c5f0b5554cf490d141cd0d339a5a7f4826ac0e04f20a35fc

# RUN apt-get update \
#     && apt-get -y install --no-install-recommends \
#         wget \
#         libgl1 \
#         libglib2.0-0 \
#         libsm6 \
#         libxext6 \
#         libxrender-dev \
#     && apt-get clean -y \
#     && rm -rf /var/lib/apt/lists/*

docker-compose.yaml

services:
  dev:
    build:
      context: .
      dockerfile: .devcontainer/Dockerfile
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    volumes:
      - .:/workspace:cached
    hostname: dev
    network_mode: host
    stdin_open: true
    tty: true
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility

requirements.txt

1 2	`opencv-python-headless numpy<2`

Run

import onnxruntime as ort

# Show all available providers on your machine
print("Available providers:", ort.get_available_providers())
print("ONNX Runtime version:", ort.__version__)
print("Build info:", ort.get_device())

1
2
3

Available providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
ONNX Runtime version: 1.20.0
Build info: GPU

Demo: mnist

MNIST model home page

mnist-8.onnx

[W:onnxruntime:Default, device_discovery.cc:164 DiscoverDevicesForPlatform] GPU device discovery failed: device_discovery.cc:89 ReadFileContents Failed to open file: "/sys/class/drm/card1/device/vendor"

for known i don't found and reference that help resolve the issue

work with onnxruntime-gpu version 1.23
need numpy <2
images zero, six
model: mnist-8.onnx
try with the 3 providers
on the docker side success only with the cpu (TODO)

on the host all off them work

import onnxruntime as ort
import numpy as np
import cv2
# import matplotlib.pyplot as plt

# --- Load ONNX model ---
# "CPUExecutionProvider"
# CUDAExecutionProvider
# TensorrtExecutionProvider
session = ort.InferenceSession("model/mnist-8.onnx", providers=["CUDAExecutionProvider"])

# --- Load and preprocess an image (28×28 grayscale) ---
# You can replace 'digit.png' with your own file.
img = cv2.imread("images/six_1.png", cv2.IMREAD_GRAYSCALE)

if img is None:
    # fallback: create a synthetic "3"-like shape for demo
    img = np.zeros((28, 28), np.uint8)
    cv2.putText(img, "3", (4, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.9, 255, 2)

# Normalize to [0,1]
img = img.astype(np.float32) / 255.0

# Reshape to NCHW = (1,1,28,28)
img = img.reshape(1, 1, 28, 28)

# --- Run inference ---
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name

pred = session.run([output_name], {input_name: img})[0]
digit = int(np.argmax(pred))

print(f"Predicted digit: {digit}")

ONNX

#TODO: move to it's on section

Install on nvidia jetson with jetpack 6.2

Installed on docker

Run

Demo: mnist

tutorials

References