RKNN YOLO ZOO

Demo: pre-trained yolo to RKNN

Target: radxa zero 3w: RK3566

• Clone zoo
• Clone toolkit v 2.3.2
• Create venv in zoo folder using uv
• Install other dependencies
• Run Convert
• Check on device

https://github.com/airockchip/rknn_model_zoo.git

git clone -b v2.3.2 https://github.com/airockchip/rknn-toolkit2.git

Create virtual environment

• create venv in zoo root folder using uv

uv venv

Tip

The repository has .python-version file that set the version 3.11 when create venv using uv it use it to set the python venv version

install toolkit version

All the version locate in repository packages folder

uv pip install ../rknn-toolkit2/rknn-toolkit2/packages/x86_64/rknn_toolkit2-2.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

uv add "setuptools<82"
uv add "onnx==1.16.1"
uv add "protobuf==4.25.4"

Run convert

# switch to 
cd examples/yolov8/python/
# convert model target_platform type dest_name (path and file name)
uv run convert.py ~/git/rknn-toolkit2/yolov8n.onnx rk3566 i8 `pwd`/yolov8n_i8.rknn

Check the model

Copy the code into device, don't forget the mode and source image

Inference code

#!/usr/bin/env python3
"""Minimal YOLOv8 RKNN inference example for a Rockchip board."""

import cv2
import numpy as np
from rknnlite.api import RKNNLite

OBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = (640, 640)  # width, height
MODEL_PATH = "yolov8n_i8.rknn"
IMAGE_PATH = "bus.jpg"
OUTPUT_PATH = "bus_detected.jpg"


def dfl(position):
    """Convert YOLOv8 distribution logits into four box distances."""
    n, channels, height, width = position.shape
    bins = channels // 4
    values = position.reshape(n, 4, bins, height, width)
    values = np.exp(values - values.max(axis=2, keepdims=True))
    values /= values.sum(axis=2, keepdims=True)
    return (values * np.arange(bins).reshape(1, 1, bins, 1, 1)).sum(axis=2)


def box_process(position):
    """Decode one output branch into x1, y1, x2, y2 boxes."""
    grid_h, grid_w = position.shape[2:]
    col, row = np.meshgrid(np.arange(grid_w), np.arange(grid_h))
    grid = np.stack((col, row)).reshape(1, 2, grid_h, grid_w)
    stride = np.array([IMG_SIZE[0] // grid_w, IMG_SIZE[1] // grid_h]).reshape(1, 2, 1, 1)
    position = dfl(position)
    return np.concatenate(((grid + 0.5 - position[:, :2]) * stride,
                           (grid + 0.5 + position[:, 2:]) * stride), axis=1)


def nms_boxes(boxes, scores):
    """Keep the highest-scoring box among overlapping boxes."""
    x1, y1, x2, y2 = boxes.T
    areas = (x2 - x1) * (y2 - y1)
    order, keep = scores.argsort()[::-1], []
    while order.size:
        i = order[0]
        keep.append(i)
        intersection = (
            np.maximum(0, np.minimum(x2[i], x2[order[1:]]) - np.maximum(x1[i], x1[order[1:]]) + 1e-5)
            * np.maximum(0, np.minimum(y2[i], y2[order[1:]]) - np.maximum(y1[i], y1[order[1:]]) + 1e-5)
        )
        overlap = intersection / (areas[i] + areas[order[1:]] - intersection)
        order = order[np.where(overlap <= NMS_THRESH)[0] + 1]
    return np.asarray(keep)


def post_process(outputs):
    """Decode three YOLO scales, filter weak boxes, then run NMS per class."""
    boxes, probabilities = [], []
    branches = 3
    outputs_per_branch = len(outputs) // branches
    if outputs_per_branch < 2:
        raise ValueError(f"Expected at least 6 model outputs, got {len(outputs)}")
    for i in range(branches):
        boxes.append(box_process(outputs[outputs_per_branch * i]))
        probabilities.append(outputs[outputs_per_branch * i + 1])

    flatten = lambda value: value.transpose(0, 2, 3, 1).reshape(-1, value.shape[1])
    boxes = np.concatenate([flatten(value) for value in boxes])
    probabilities = np.concatenate([flatten(value) for value in probabilities])
    classes = probabilities.argmax(axis=1)
    scores = probabilities.max(axis=1)
    selected = scores >= OBJ_THRESH
    boxes, classes, scores = boxes[selected], classes[selected], scores[selected]

    kept = []
    for class_id in np.unique(classes):
        indices = np.where(classes == class_id)[0]
        kept.extend(indices[nms_boxes(boxes[indices], scores[indices])])
    if not kept:
        return None, None, None
    kept = np.asarray(kept)
    return boxes[kept], classes[kept], scores[kept]


def main():
    model = RKNNLite()
    try:
        # 1. Load the converted model and connect RKNNLite to the board's NPU.
        if model.load_rknn(MODEL_PATH) != 0:
            raise RuntimeError(f"Failed to load {MODEL_PATH}")
        if model.init_runtime() != 0:
            raise RuntimeError("Failed to initialize RKNN runtime")

        # 2. The example image already matches the model's 640 x 640 input.
        # ponytail: fixed-size input keeps the demo focused; add letterboxing for arbitrary images.
        source = cv2.imread(IMAGE_PATH)
        if source is None:
            raise RuntimeError(f"Failed to read {IMAGE_PATH}")
        if source.shape[:2] != IMG_SIZE[::-1]:
            raise ValueError(
                f"Expected a {IMG_SIZE[0]} x {IMG_SIZE[1]} image, got {source.shape[1]} x {source.shape[0]}"
            )

        # 3. RKNN expects RGB with a leading batch dimension: (1, 640, 640, 3).
        image = cv2.cvtColor(source, cv2.COLOR_BGR2RGB)
        # image
        #     └── 640 rows × 640 columns × 3 channels
# 
        # input_tensor
        #     └── batch containing 1 image
        #         └── 640 rows × 640 columns × 3 channels
        # np.expand_dims(image, axis=0).shape
        # (1, 640, 640, 3)
        #
        #  These two expressions are equivalent:
        #   input_tensor = np.expand_dims(image, axis=0)
        #   input_tensor = image[None]
        outputs = model.inference(inputs=[image[None]])
        if outputs is None:
            raise RuntimeError("RKNN inference failed")

        # 4. Convert raw model outputs into final boxes, class IDs, and scores.
        boxes, classes, scores = post_process(outputs)
        if boxes is not None:
            for box, class_id, score in zip(boxes, classes, scores):
                x1, y1, x2, y2 = map(int, box)
                label = f"class {class_id}: {score:.2f}"
                print(f"{label} @ ({x1} {y1} {x2} {y2})")
                cv2.rectangle(source, (x1, y1), (x2, y2), (255, 0, 0), 2)
                cv2.putText(source, label, (x1, max(y1 - 6, 0)), cv2.FONT_HERSHEY_SIMPLEX,
                            0.6, (0, 0, 255), 2)

        # 5. Save the annotated result even when no objects pass the threshold.
        if not cv2.imwrite(OUTPUT_PATH, source):
            raise RuntimeError(f"Failed to write {OUTPUT_PATH}")
        print(f"Saved {OUTPUT_PATH}")
    finally:
        model.release()


if __name__ == "__main__":
    main()

flowchart LR
    A[Load .rknn model] --> B[Initialize RKNN NPU runtime]
    B --> C[Read bus.jpg with OpenCV]
    C --> D{Image is 640 x 640?}
    D -- No --> X[Stop with size error]
    D -- Yes --> E[Convert OpenCV BGR to model RGB]
    E --> F[Add batch dimension<br/>shape 1 x 640 x 640 x 3]
    F --> G[Run model.inference on the NPU]
    G --> H[Raw box and class tensors<br/>at 3 detection scales]
    H --> I[Decode DFL box distances]
    I --> J[Convert to x1 y1 x2 y2]
    J --> K[Flatten and join all scales]
    K --> L[Choose best class and score]
    L --> M{Score at least OBJ_THRESH?}
    M -- No --> N[Discard candidate]
    M -- Yes --> O[Apply NMS separately per class]
    O --> P[Draw retained boxes and labels]
    P --> Q[Save bus_detected.jpg]
    Q --> R[Release RKNN runtime]
    X --> R

The important data changes are:

1. cv2.imread() loads the file as BGR, OpenCV's default channel order.
2. cv2.cvtColor(..., cv2.COLOR_BGR2RGB) changes it to RGB, the channel order expected by this model.
3. image[None] adds the batch dimension, changing (640, 640, 3) into (1, 640, 640, 3).
4. model.inference() sends that tensor to the RK3566 NPU. Its result is a collection of raw NumPy tensors, not ready-to-draw boxes.
5. post_process() decodes those tensors, removes low-confidence and overlapping candidates, and returns the final boxes, class IDs, and scores.

uv run zoo_inference.py 
I RKNN: [19:33:43.881] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [19:33:43.882] RKNN Driver Information, version: 0.9.8
I RKNN: [19:33:43.883] RKNN Model Information, version: 6, toolkit version: 2.3.2(compiler version: 2.3.2 (e045de294f@2025-04-07T19:48:25)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
W RKNN: [19:33:43.934] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
person @ (211 241 282 506) 0.864
person @ (109 235 225 535) 0.860
person @ (477 226 560 522) 0.848
person @ (79 327 116 513) 0.305
bus @ (96 136 549 449) 0.864
Saved bus_detected.jpg

Explain output

The run completed successfully. The log contains: - one harmless warning, - detection results, - saved-image path.

Runtime and model information

librknnrt version: 2.3.0
RKNN Driver Information, version: 0.9.8
toolkit version: 2.3.2
target platform: rk3566
framework name: ONNX
model inference type: static_shape

• librknnrt is the RKNN runtime installed on the board.
• The driver connects that runtime to the NPU.
• Toolkit 2.3.2 converted the original ONNX model for the RK3566.
• static_shape means the model accepts its fixed 640 × 640 input shape.

Static-shape warning

query RKNN_QUERY_INPUT_DYNAMIC_RANGE error
If it is a static shape RKNN model, please ignore the above warning message.

This warning is expected for this model. RKNNLite tried to query dynamic input dimensions, but the model was intentionally exported with a fixed shape. The example supplies a 640 × 640 image, so no change is required.

Detections

Each result uses this format:

class @ (x1 y1 x2 y2) confidence

For example:

person @ (211 241 282 506) 0.864

Field	Meaning
person	Predicted COCO class.
(211, 241)	Top-left corner of the bounding box.
(282, 506)	Bottom-right corner of the bounding box.
0.864	Confidence score, approximately 86.4%.

The model found four people and one bus. The last person has a score of 0.305 and remains because OBJ_THRESH is 0.25. Increase the threshold to reject weaker detections.

The bus box can overlap person boxes because non-maximum suppression runs separately for each class.

Saved result

Saved bus_detected.jpg

Open bus_detected.jpg to inspect the bounding boxes drawn over the source image.

---

Reference

• RKNN Toolkit Lite2 YOLOv8
• Convert Custom Trained YOLO Models