Welcome to Jetson Orin NANO Developer Kit repository
03.Parallel multi-model inference
DeepStream Parallel Multi-Model Inference
DeepStream Parallel Multi-Model Inference1. Factory Environment and First Deployment Reference1.1 Obtaining the Official Source Code1.2 Preparing Triton, Models, and TensorRT Engines1.3 Compiling the Official Parallel Application2. Example Goal3. Launching via Scripts: Choosing the Input Source4. Local Video Verification5. Real-Time Verification with a USB Camera5.1 Direct Run5.2 Common Parameters6. Real-Time Verification with a CSI Camera6.1 Direct Run6.2 Common Parameters7. Traffic Tracking and Line-Crossing Counting8. Output, Stop, and Checks9. Common Symptoms
The factory system already has the DeepStream, Triton runtime, models, TensorRT engines, course launch entry points, and compatibility configurations required by this example ready. For normal study, simply run from Section 3 by choosing an input source; the deployment commands in this section are only for reference when the environment is missing, needs restoration, or is being redeployed.
After the power mode is set to MAXN, running the example may pop up an over-current warning; just ignore it
1. Factory Environment and First Deployment Reference
The factory system already contains:
The video, USB, and CSI Python entry points under
~/DeepStream/course-examples/.The Triton runtime, launch scripts, and the single-camera four-pane program under
~/DeepStream/parallel-inference/.The models and TensorRT engines required by YOLOv4, TrafficCamNet, and BodyPose2D.
The compatibility launch configuration for embedded Triton.
Therefore, do not repeat the commands in this section; only when the environment is missing, needs restoration, or is being deployed on your own, follow the order below.
1.1 Obtaining the Official Source Code
cd ~git clone https://github.com/NVIDIA/DeepStream.gitcd ~/DeepStreamgit checkout v9.1.0git submodule update --init --recursiveConfirm the version:
xxxxxxxxxxgit describe --tags --exact-match
1.2 Preparing Triton, Models, and TensorRT Engines
x# This script is located in the DeepStream repository root, not in the parallel multi-model example directory.cd ~/DeepStream/scriptssudo ./triton_backend_setup.sh
cd ~/DeepStream/src/apps/reference_apps/deepstream_parallel_inference_app/tritonserversudo ./build_engine.shThe first build downloads the models and generates engines for the Jetson, which takes a long time. If it reports that trtexec cannot be found, first check:
xxxxxxxxxxcommand -v trtexecThis course environment uses a toolchain matched to CUDA 13.2 and TensorRT 10.16; do not directly upgrade the CUDA or TensorRT metapackages just to install trtexec.
1.3 Compiling the Official Parallel Application
xxxxxxxxxxcd ~/DeepStream/src/apps/reference_apps/deepstream_parallel_inference_app/tritonclient/samplemake -C apps/deepstream-parallel-infer CUDA_VER=13.22. Example Goal
This example derives multiple processing branches from the same video frame or the same camera frame, and is used to learn how nvinfer, nvinferserver (Triton), the Tracker, nvdsanalytics, and human pose post-processing work together.
The final window is a four-pane layout:
| Position | Displayed Content |
|---|---|
| Top left | YOLOv4 general object detection: persons, vehicles, bicycles, backpacks, etc. |
| Top right | YOLO boxes + NVTracker + TrafficCamNet + line-crossing counting |
| Bottom left | YOLO boxes + BodyPose2D: human keypoints and skeleton |
| Bottom right | Original frame for comparison |
For a USB camera, the frame is captured only once; DeepStream copies the same frame internally to the four display branches and does not open the camera four times.
Official reference source code: https://github.com/NVIDIA/DeepStream/tree/main/src/apps/reference_apps/deepstream_parallel_inference_app
3. Launching via Scripts: Choosing the Input Source
The current unified launch script is located at ~/DeepStream/deepstream-shell-demo/:
xxxxxxxxxxcd ~/DeepStream/deepstream-shell-demo| Input Source | Command | Applicable Scenario |
|---|---|---|
| Local video | ./run_video.sh parallel | First verification of the four-pane layout and each model's effect, with MP4 recording |
| USB camera | ./run_usb.sh parallel | Real-time input from a UVC/V4L2 USB camera, display only |
| CSI camera | ./run_csi.sh parallel | Real-time input from a MIPI CSI camera, display only |
All three entry points display the four-pane layout in real time; only the local video entry records an MP4, while the USB/CSI entries do not write video files.
4. Local Video Verification
For the first run, it is recommended to run the local video first to confirm that the models, the four-pane layout, and the display environment are correct:
xxxxxxxxxxcd ~/DeepStream/deepstream-shell-demo./run_video.sh parallel
5. Real-Time Verification with a USB Camera
5.1 Direct Run
Note: the bundled factory USB camera has been adapted by default to the MJPEG 1280×720@60 mode
xxxxxxxxxxcd ~/DeepStream/deepstream-shell-demo./run_usb.sh parallel5.2 Common Parameters
Before running, you can check the device nodes and the formats, resolutions, and frame rates supported by the camera:
xxxxxxxxxxv4l2-ctl --list-devicesv4l2-ctl -d /dev/video0 --list-formats-ext
Then pass parameters according to the actual device.
xxxxxxxxxx# View all parameters./run_usb.sh parallel --help
# Use another video device./run_usb.sh parallel --device /dev/video1
# Choose among the supported configurations listed above; here 1920×1080, 30 FPS is used; without the suffix, this is the YUY2 stream./run_usb.sh parallel --width 1920 --height 1080 --fps 6
# Add the mjpeg parameter to use MJPG mode./run_usb.sh parallel --width 1920 --height 1080 --fps 30 --mjpeg6. Real-Time Verification with a CSI Camera
6.1 Direct Run
xxxxxxxxxxcd ~/DeepStream/deepstream-shell-demo./run_csi.sh parallel6.2 Common Parameters
Before running, you can check the formats, resolutions, and frame rates supported by the camera; this needs to be run in a graphical interface:
xxxxxxxxxxgst-launch-1.0 nvarguscamerasrc
xxxxxxxxxx# View all parameters./run_csi.sh parallel --help
# Use the second CSI camera./run_csi.sh parallel --sensor-id 1
# Adjust the input size and frame rate./run_csi.sh parallel --width 1920 --height 1080 --fps 307. Traffic Tracking and Line-Crossing Counting
The processing chain at the top right is:
xxxxxxxxxxYOLO detection boxes → NVTracker → line-crossing counting → TrafficCamNetYOLO finds vehicles, persons, and other targets and generates detection boxes.
NVTracker assigns a stable ID to the same target across consecutive frames, avoiding repeated per-frame counting of the same vehicle.
The line-crossing counting accumulates once when a tracked target crosses the manually configured counting line.
TrafficCamNet is a traffic scene detection model, used for the parallel traffic model demonstration.
The green line, Entry, and Exit are manually defined business rules and count displays, not obtained by the model automatically recognizing the road. Before real deployment, the counting line must be re-calibrated according to the road, lanes, or entrances and exits in the actual camera view; the default line in the tutorial is only for demonstration.
8. Output, Stop, and Checks
The real-time window is displayed on the Jetson local desktop or the VNC desktop. The video output location of the local video version is fixed at:
xxxxxxxxxx~/DeepStream/src/apps/reference_apps/deepstream_parallel_inference_app/tritonclient/sample/out.mp4Press Ctrl+C once in the terminal where the run command was executed to stop normally. The program sends EOS and finishes writing the MP4 index; do not use kill -9, otherwise the video may become unplayable.
After stopping, you can play or inspect:
xxxxxxxxxxtotem ~/DeepStream/src/apps/reference_apps/deepstream_parallel_inference_app/tritonclient/sample/out.mp4
gst-discoverer-1.0 \ ~/DeepStream/src/apps/reference_apps/deepstream_parallel_inference_app/tritonclient/sample/out.mp49. Common Symptoms
| Symptom | Handling |
|---|---|
| The video version runs, but the USB version does not start | First run v4l2-ctl --list-formats-ext, and verify the device number, resolution, and frame rate; MJPEG input requires adding --mjpeg. |
| CSI cannot be opened | Confirm the sensor-id, the camera wiring, and whether the basic CSI example works normally. |
A fixed counting line or Exit text appears at the top right | This is the configuration overlay of nvdsanalytics; it only has business meaning after re-calibration according to the actual road. |
| The MP4 is 0 bytes or unplayable | Do not force-kill the process; run again and press Ctrl+C once, and wait for the program to exit completely. |