Skip to content

Programming / vision / nvidia

NVIDIA® Vision Programming Interface (VPI) is a software library that implements computer vision (CV) and image processing (IP) algorithms on several computing hardware platforms available in NVIDIA embedded and discrete devices. VPI provides seamless access to computer hardware. Within VPI, the same algorithm is implemented in different backends, such as CPU, GPU, PVA1, VIC2 and OFA READ More

nvidia

  • PVA: Programmable Vision Accelerator more
  • VIC: Video Image Compositor
  • OFA: Optical Flow Accelerator

Install

nvidia documents

vpi installation location

1
/opt/nvidia/vpi3

Basic Concepts

  • Stream
  • backends
  • Algorithms
  • Data Buffers

Backend

A backend comprises the compute hardware that ultimately runs an algorithm

Backend Device/platform
CPU All devices on x86 (linux) and Jetson aarch64 platforms
CUDA All devices on x86 (linux) with a Maxwell or superior NVIDIA GPU, and Jetson aarch64 platforms
PVA All Jetson AGX Orin and Jetson Orin NX devices
VIC All Jetson devices.
OFA All Jetson Orin devices.

Hello VPI

Very simple application convert image to gray using PIL and VPI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import sys
import vpi

import numpy as np
from PIL import Image


try:
    input = vpi.asimage(np.asarray(Image.open("/home/user/projects/nvidia_tutorial/assets/kodim08.png")))
except IOError:
    sys.exit("Input file not found")
except:
    sys.exit("Error with input file")

# 4. Convert it to grayscale .
# -----------------------------------------------------------------------------

# Enabling the CUDA backend in a python context like done below makes
# VPI algorithms use CUDA for execution by default. This can be overriden
# if needed by specifying the parameter `backend=` when calling the algorithm.
with vpi.Backend.CUDA:
    # `image.convert` will return a new VPI image with the desired format, in
    # this case U8 (grayscale, 8-bit unsigned pixels).
    # Algorithms returning a new VPI image allows for chaining operations like
    # done below, as the result of the conversion is then filtered.
    # The end result is finally stored in a new `output` VPI image.
    output = input.convert(vpi.Format.U8)

# -----------------------------------------------------------------------------

# The read-lock context enabled below makes sure all processing is finished and
# results are stored in `output`.
with output.rlock_cpu() as outData:
    # `outData` is a numpy array view (not a copy) of `output`
    # contents, accessible by host (cpu). 
    # The numpy array is then converted into a Pillo image and saved / show
    # to the disk
    Image.fromarray(outData).show()

Check if backend available

1
2
3
4
5
6
import vpi

print("CUDA available? ", bool(vpi.Backend.CUDA & vpi.Backend.AVAILABLE))
print("PVA available? ", bool(vpi.Backend.PVA & vpi.Backend.AVAILABLE))
print("VIC available? ", bool(vpi.Backend.VIC & vpi.Backend.AVAILABLE))
print("OFA available? ", bool(vpi.Backend.OFA & vpi.Backend.AVAILABLE))

Convert vpi image to numpy view

vpi to numpy view
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import matplotlib.pyplot as plt
import vpi

# img_vpi is a vpi.Image (any backend). Ensure it's 8-bit gray.
img_u8 = img_vpi.convert(vpi.Format.U8)        # or skip if already U8

with img_u8.rlock_cpu() as np_img:             # NumPy view (no copy)
    plt.imshow(np_img, cmap='gray', vmin=0, vmax=255)
    plt.axis('off')
    plt.show()

LK

1
2
3
4
5
6
 class vpi.OpticalFlowPyrLK(frame: vpi.Image, 
    keypoints: vpi.Array, 
    levels: int, *, 
    kptstatus: vpi.Array = None, 
    scale: float = 0.5, 
    backend: vpi.Backend = vpi.Backend.DEFAULT) → vpi.OpticalFlowPyrLK

Reference