Multi-stream inference status on Jetson Orin NX and Qualcomm IQ-8275

This topic describes a multi-stream inference benchmark for the iQS-Streampipe application on the NVIDIA Jetson Orin NX 16GB and the Qualcomm Dragonwing IQ-8275 platform. The purpose is to observe execution behavior as the number of channels and the video resolution increase. The results are not intended to be a general hardware performance ranking.

• Application: iQS-Streampipe
• Task: object detection with an INT8 model
• Input: 720p, 1080p, 2K, and 4K, 30 FPS H.264 video streams
• Platforms: NVIDIA Jetson Orin NX 16GB, Qualcomm Dragonwing IQ-8275
• Metrics:
  • Output average end-to-end FPS per channel
  • Average CPU loading (%)
  • Average CPU memory usage (%)
  • Power (W) and FPS/Watt
  • Output end-to-end FPS percentiles (P95, P75, P50, P25)
• Channel counts: 1, 4, 9, and 16 at 720p and 1080p; 1, 4, and 9 at 2K; 1 and 4 at 4K

Hardware

The benchmark uses two edge AI platforms with the configurations shown below.

AI model

Videos

The results in this report were measured with four videos, one for each resolution.

The four videos were made from one licensed stock video from Envato, and they are not distributed with iQ-Studio. To reproduce the results, license a video and prepare your own copies at each resolution. The video must be H.264 in an MP4 file.

Results

Measurement method

• Each run starts after the platform has cooled down, then warms up for 300 seconds. The measurement window is the following 120 seconds.
• FPS and CPU values come from the measurement window. FPS is the end-to-end output rate of each channel.
• The videos are described in the Videos section. Every channel plays the same video.
• Power (W) is the reading of a power meter on the wall socket, read by hand during the measurement window. The benchmark script does not measure it. It covers the platform only and includes the power adapter loss. FPS/Watt = channels × output average E2E FPS per channel ÷ Power (W).

720P@30FPS

1080P@30FPS

2K@30FPS

4K@30FPS

Observations

• In the multi-channel cases at 720p and 1080p, the IQ-8275 delivers a higher per-channel FPS than the Jetson Orin NX. At 720p, the 9- and 16-channel cases reach about 25.2 and 13.1 FPS, and at 1080p, about 10.3 and 7.4 FPS.
• The IQ-8275 draws up to 13.9 W over the whole test range, while the Jetson Orin NX draws up to 31.6 W.
• The Jetson Orin NX has 8× Cortex-A78AE cores, and the IQ-8275 has 4× Cortex-A78C + 4× Cortex-A55 cores. The four efficiency cores of the IQ-8275 reach high utilization first, so its overall CPU loading is higher than that of the Jetson Orin NX.

Conclusion

Both platforms follow the same pattern: per-channel FPS stays near the source rate at low channel counts, then drops as the number of channels increases. Because the CPU of the IQ-8275 is less capable (four of its eight cores are efficiency cores), its per-channel FPS is lower at high resolutions (4K, and 2K with 4 channels). But in the multi-channel cases, it still performs at a good level: at 720p, 9 channels keep 25.2 FPS and 16 channels reach 13.1 FPS, and at 1080p, 9 and 16 channels reach 10.3 and 7.4 FPS, with a lower power consumption.

How to Use

Below is an overview of how we conduct platform benchmark testing.

IQ-8275 procedure

1. 1
   Install iQ-Studio:

   {{ mc1.fname }}bash

   $ git clone https://github.com/InnoIPA/iQ-Studio.git
   $ cd iQ-Studio
   $ ./install.sh

2. 2
   From the repository root, change to the benchmark directory:

   {{ mc2.fname }}bash

   $ cd benchmarks/iq-8275/multi-stream/iqs-streampipe

3. 3
   Edit config.json so that the streams array contains 1, 4, 9, or 16 entries, depending on the channel count being tested. The file contains 9 streams of the YOLO26n model and the 1080p videos that come with the Docker image.
4. 4
   Run the benchmark script with the Qualcomm platform selected:

   {{ mc3.fname }}bash

   $ ./scripts/auto_benchmark.sh \
       --platform qcom \
       --test_time 420 \
       --warmup_time 300 \
       --output ./logs/qcom_9ch.txt

   On the first run, iqs-launcher pulls the iqs-streampipe Docker image, so the board needs network access.

   The results are saved in the file given by --output. The file starts with the test settings (channels, video, warm-up, and test time), followed by the average CPU loading and memory usage, and the inference time and FPS tables. Use a different file name for each run, because the file is overwritten.

NVIDIA Jetson Orin NX 16GB procedure

1. 1
   Go to the benchmark directory of the iQ-Studio repository and download the archive:

   {{ mc4.fname }}bash

   $ cd benchmarks/iq-8275/multi-stream/iqs-streampipe
   $ wget https://github.com/InnoIPA/iQ-Studio/releases/download/v0.0.13/benchmark_orinnx_sources.tar.gz

2. 2
   In the same directory, extract the NVIDIA benchmark archive and check its contents:

   {{ mc5.fname }}bash

   $ tar xzf benchmark_orinnx_sources.tar.gz --strip-components=1
   $ ls -1 config_nv.json nvidia/* videos/*

   The output should list these files:

   {{ mc6.fname }}text

   config_nv.json            # stream configuration template
   nvidia/model_int8.engine  # TensorRT engine
   nvidia/streampipe_nv      # streampipe executable for the NVIDIA platform
   videos/japan_walk.mp4     # included 1080p video

3. 3
   Set the power mode to MAXN and check it:

   {{ mc7.fname }}bash

   $ sudo nvpmodel -m 0
   $ nvpmodel -q

   The output should be:

   {{ mc8.fname }}text

   NV Power Mode: MAXN
   0

4. 4
   Install the Python virtual environment package if it is missing. The script uses it to keep its Python packages separate from the system (the first run also downloads them, so it needs network access):

   {{ mc9.fname }}bash

   $ sudo apt install python3-venv

5. 5
   From a desktop session on the Jetson, run the benchmark script with the NVIDIA platform selected:

   {{ mc10.fname }}bash

   $ ./scripts/auto_benchmark.sh \
       --platform nv \
       --channels 9 \
       --test_time 420 \
       --warmup_time 300 \
       --output ./logs/nv_1080p_9ch.txt \
       --video <your video path>   # replace with the path of your own video

   Without --channels and --video, it runs config_nv.json as is, which is 1 channel of videos/japan_walk.mp4.

   The results are saved in the file given by --output. The file starts with the test settings (channels, video, warm-up, and test time), followed by the average CPU loading and memory usage, and the inference time and FPS tables. Use a different file name for each run, because the file is overwritten.