Skip to content

What is RKNN?

RKNN is Rockchip’s model format for running neural networks on Rockchip NPUs. To convert onnx or other model format to rknn we use github rknn-toolkit, the current version that we test for this post v2.3.2

Typical workflow:

1
2
3
4
5
6
7
8
9
  PyTorch / ONNX model
          ↓
  RKNN-Toolkit2 on PC
          ↓
  model.rknn
          ↓
  RKNN Runtime on Rockchip board
          ↓
  NPU inference

What is RKNN-Toolkit?

For modern chips such as RK3566 and RK3588, the RKNN-Toolkit2 runs on the development PC and provides:

  • Model conversion to .rknn
  • INT8 quantization
  • Model validation
  • Accuracy and performance analysis
  • PC simulation or connected-board testing

What to install

Machine Install Purpose
Development PC RKNN-Toolkit2 Convert ONNX/PyTorch models into .rknn
Board using Python RKNN-Toolkit-Lite2 Load and run .rknn models
Board using C/C++ RKNN Runtime, usually librknnrt.so Native deployment and inference
Board firmware/kernel RKNPU driver Communicate with the NPU hardware

Do not install the complete RKNN-Toolkit2 package on the board. It belongs on the PC. Rockchip describes Lite2 as the board-side Python API and RKNN Runtime as the board-side C/C++ API. Official architecture

For C/C++ deployment, the board image often already contains the NPU driver and runtime. For Python deployment, install the matching Lite2 wheel from the Toolkit2 repository. Keep Toolkit2, Lite2/runtime, driver, and model versions compatible.

PC

install

Install python whell from github airockchip/rknn-toolkit2

Convert model

ultralytics yolo26

1
yolo export model=yolo26n.pt format=rknn name=rk3588 opset=13

opset

ONNX has versions of its operator specification called opsets.

An ONNX opset defines the behavior and available versions of those operators.

1
2
3
4
5
6
7
8
9
yolo26n.pt
    │
    │ Ultralytics + PyTorch
    ▼
yolo26n.onnx       ← opset=13 applies HERE
    │
    │ rknn-toolkit2
    ▼
yolo26n-rk3588.rknn

RKNN Model Zoo YOLOv8
More...

Convert a pretrained YOLOv8 ONNX model for RK3566, copy it to the board, run RKNNLite inference, and understand the output.

Convert to INT8

INT8 quantization requires a representative calibration dataset. During RKNN compilation, representative images are passed through the network so RKNN Toolkit2 can determine quantization ranges/scales.

Info

For the pretrained YOLO26n COCO model, you don't need a special set of “YOLO26 calibration images.” You need a collection of representative images similar to what the model will see during inference.

!!! tip coco8.yaml For a first test, the easiest choice is Ultralytics' small COCO dataset, coco8.yaml. Ultralytics' RKNN exporter accepts a dataset YAML via data=... and internally creates the image list that RKNN Toolkit2 uses for calibration.

1
2
3
4
5
6
uv run yolo export \
    model=yolo26n.pt \
    format=rknn \
    name=rk3566 \
    quantize=8 \
    data=coco8.yaml

Board

1
2
3
4
5
6
uv venv --python 3.12

uv pip install \
https://raw.githubusercontent.com/airockchip/rknn-toolkit2/master/rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

uv pip install opencv-python-headless
check imports
1
2
3
4
import cv2
import numpy as np

from rknnlite.api import RKNNLite

Demo:

Python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import cv2
import numpy as np
from rknnlite.api import RKNNLite


MODEL = "yolo26n-rk3566.rknn"
IMAGE = "bus.jpg"

rknn = RKNNLite()

ret = rknn.load_rknn(MODEL)
if ret != 0:
    raise RuntimeError("Could not load RKNN")

ret = rknn.init_runtime()
if ret != 0:
    raise RuntimeError("Could not initialize RKNN runtime")


image = cv2.imread(IMAGE)

image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

image = cv2.resize(image, (640, 640))

input_tensor = np.expand_dims(image, axis=0)

outputs = rknn.inference(
    inputs=[input_tensor]
)

for i, output in enumerate(outputs):
    print(i, output.shape)

rknn.release()
usage
1
uv run python demo3.py

cpp

1
2
3
4
5
sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    libopencv-dev
main.cpp 
1
CMakeLists.txt 
1

Demo: run quantization models

Tip

Use convert models form Qengineering / YoloV8-NPU

Info

Rockchip SoC NPU Typical Radxa products RKNN target
RK3566 ~1 TOPS INT8 Zero 3W/3E, ROCK 3C, CM3/CM3S rk3566
RK3568 ~0.8–1 TOPS INT8 ROCK 3A, ROCK 3B, CM3I rk3568 / RK356X
RK3576 6 TOPS INT8 ROCK 4D, CM4, NX4 rk3576
RK3582 5 TOPS ROCK 5C Lite, E52C/E54C rk3588
RK3588S 6 TOPS ROCK 5A, CM5, NX5 rk3588
RK3588 6 TOPS ROCK 5B/5B+, ROCK 5T, ROCK 5 ITX rk3588
Yolo8n example 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import cv2
import numpy as np
from rknnlite.api import RKNNLite


MODEL = "yolov8n.rknn"
IMAGE = "bus.jpg"
rknn = RKNNLite()

ret = rknn.load_rknn(MODEL)
if ret != 0:
    raise RuntimeError("Could not load RKNN")

ret = rknn.init_runtime()
if ret != 0:
    raise RuntimeError("Could not initialize RKNN runtime")


image = cv2.imread(IMAGE)

image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

image = cv2.resize(image, (640, 640))

input_tensor = np.expand_dims(image, axis=0)

outputs = rknn.inference(
    inputs=[input_tensor]
)

for i, output in enumerate(outputs):
    print(i, output.shape)

rknn.release()

Demo: hybrid-quantization

Quantize most of the YOLO26 network to INT8, but keep a few sensitive output layers in floating point, usually FP16.

TODO: auto hybrid

prerequisite

  • yolo26n.onnx
  • calibration.txt

calibration.txt contains representative images (i use the coco8 images for simplicity)

convert code

Step1 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from rknn.api import RKNN

ONNX_MODEL = "yolo26n.onnx"
DATASET = "calibration.txt"

rknn = RKNN(verbose=True)

ret = rknn.config(
    target_platform="rk3566",
    mean_values=[[0, 0, 0]],
    std_values=[[255, 255, 255]],
)
assert ret == 0

ret = rknn.load_onnx(
    model=ONNX_MODEL
)
assert ret == 0

ret = rknn.hybrid_quantization_step1(
    dataset=DATASET,
    proposal=False,
)
assert ret == 0

rknn.release()

Edit yolo26n.quantization.cfg

  • Replace 

    1
    custom_quantize_layers: {}

  • With

1
2
3
custom_quantize_layers:
  output0-rs: float16
  output0: float16
Step2 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
from rknn.api import RKNN

MODEL = "yolo26n.model"
DATA = "yolo26n.data"
CFG = "yolo26n.quantization.cfg"
OUTPUT = "yolo26n_hybrid.rknn"

rknn = RKNN(verbose=True)

ret = rknn.hybrid_quantization_step2(
    model_input=MODEL,
    data_input=DATA,
    model_quantization_cfg=CFG,
)
assert ret == 0

ret = rknn.export_rknn(OUTPUT)
assert ret == 0

print(f"created: {OUTPUT}")

rknn.release()

Run and compare

Inference 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
import cv2
import numpy as np
from rknnlite.api import RKNNLite


MODEL = "yolo26n-rk3566.rknn"
#MODEL = "yolo26n_hybrid.rknn"
#MODEL = "yolo26n0fp16.rknn"
IMAGE = "bus.jpg"
OUTPUT_IMAGE = "bus_detected.jpg"

CONF_THRESHOLD = 0.25
IOU_THRESHOLD = 0.45


# --------------------------------------------------
# Load RKNN model
# --------------------------------------------------

rknn = RKNNLite()

ret = rknn.load_rknn(MODEL)
if ret != 0:
    raise RuntimeError("Could not load RKNN")

ret = rknn.init_runtime()
if ret != 0:
    raise RuntimeError("Could not initialize RKNN runtime")


# --------------------------------------------------
# Load image
# --------------------------------------------------

image_bgr = cv2.imread(IMAGE)

if image_bgr is None:
    raise RuntimeError(f"Could not read {IMAGE}")

# Resize for YOLO
image_bgr = cv2.resize(image_bgr, (640, 640))

# RKNN input uses RGB
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)

input_tensor = np.expand_dims(image_rgb, axis=0)


# --------------------------------------------------
# Inference
# --------------------------------------------------

outputs = rknn.inference(inputs=[input_tensor])

for i, output in enumerate(outputs):
    print(
        f"output[{i}]: "
        f"shape={output.shape}, "
        f"dtype={output.dtype}, "
        f"min={output.min()}, "
        f"max={output.max()}"
    )


# --------------------------------------------------
# Decode YOLO output
# --------------------------------------------------

output = outputs[0]

pred = output[0]

# (84, 8400) -> (8400, 84)
if pred.shape[0] < pred.shape[1]:
    pred = pred.T

boxes = []
scores = []
class_ids = []

for detection in pred:

    x, y, w, h = detection[:4]

    class_scores = detection[4:]

    class_id = np.argmax(class_scores)
    confidence = class_scores[class_id]

    if confidence < CONF_THRESHOLD:
        continue

    left = int(x - w / 2)
    top = int(y - h / 2)

    boxes.append([
        left,
        top,
        int(w),
        int(h)
    ])

    scores.append(float(confidence))
    class_ids.append(int(class_id))


# --------------------------------------------------
# NMS
# --------------------------------------------------

indices = cv2.dnn.NMSBoxes(
    boxes,
    scores,
    CONF_THRESHOLD,
    IOU_THRESHOLD
)


# --------------------------------------------------
# Draw detections
# --------------------------------------------------

for i in indices:

    x, y, w, h = boxes[i]

    class_id = class_ids[i]
    confidence = scores[i]

    # Rectangle
    cv2.rectangle(
        image_bgr,
        (x, y),
        (x + w, y + h),
        (0, 255, 0),
        2
    )

    # Text
    label = f"{class_id} {confidence:.2f}"

    cv2.putText(
        image_bgr,
        label,
        (x, max(y - 10, 20)),
        cv2.FONT_HERSHEY_SIMPLEX,
        0.6,
        (0, 255, 0),
        2
    )

    print(
        f"class={class_id} "
        f"confidence={confidence:.3f} "
        f"bbox=({x}, {y}, {w}, {h})"
    )


# --------------------------------------------------
# Save image
# --------------------------------------------------

cv2.imwrite(OUTPUT_IMAGE, image_bgr)

print(f"Saved: {OUTPUT_IMAGE}")

rknn.release()
hybrid
hybrid.model
1
2
3
4
5
6
output[0]: shape=(1, 84, 8400), dtype=float32, min=0.0, max=671.5
class=5 confidence=0.884 bbox=(84, 130, 470, 319)
class=0 confidence=0.863 bbox=(220, 251, 80, 260)
class=0 confidence=0.853 bbox=(106, 239, 110, 300)
class=0 confidence=0.846 bbox=(464, 230, 91, 300)
class=0 confidence=0.482 bbox=(84, 328, 34, 190)
fp16.mode
1
2
3
4
5
6
7
output[0]: shape=(1, 84, 8400), dtype=float32, min=0.0, max=671.0
class=5 confidence=0.896 bbox=(85, 136, 470, 308)
class=0 confidence=0.855 bbox=(108, 235, 115, 299)
class=0 confidence=0.843 bbox=(212, 241, 73, 269)
class=0 confidence=0.827 bbox=(477, 229, 84, 292)
class=0 confidence=0.554 bbox=(79, 329, 37, 186)
Saved: bus_detected.jpg

Reference