yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

03.Multimodal video analysis application

Multimodal Video Analysis Application

Multimodal Video Analysis Application1. Concept Introduction1.1 What Is "Video Analysis"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer and Frame Extraction (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Input Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The message "Video file not found" or "Unable to extract keyframes from the video" is displayed.Problem 2: Analyzing a long video takes a very long time.

Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.


1. Concept Introduction

1.1 What Is "Video Analysis"?

In the largemodel project, the multimodal video analysis feature means enabling the robot to process a video and summarize its core content in natural language, describe key events, or answer specific questions about the video. This elevates the robot from understanding only static images to understanding a dynamic world with temporal relationships.

The core tool of this feature is analyze_video. When the user provides a video file and asks a question (for example "Summarize what this video is about"), the system calls this tool to process and analyze the video, and returns the AI's text answer.

1.2 Implementation Principles Overview

The core challenge of offline video analysis is how to enable the model to efficiently process video data containing hundreds or thousands of frames. One mainstream implementation principle is as follows:

  1. Keyframe Extraction: First, the system does not process every frame of the video, but extracts the most representative keyframes through algorithms (such as scene boundary detection or fixed-interval sampling), which greatly reduces the amount of data that needs to be processed.

  2. Image Encoding: Each extracted keyframe is fed, just as in the visual understanding application, into a vision encoder and converted into a digital vector containing the image information.

  3. Temporal Information Fusion: This is the biggest difference from single-image understanding. The model needs to understand the temporal order among these keyframes. Usually a recurrent neural network (RNN) or Transformer model is used to fuse the vectors of all keyframes into a "memory" vector that can represent the dynamic content of the entire video.

  4. Question Answering and Generation: The user's text question is encoded and then cross-modally fused with this "memory" vector. Finally, the language model part generates a summary of the entire video or the answer to a specific question based on this fused information.

Simply put, it means condensing a video into a few key pictures and their order, then understanding the whole story like reading a comic strip, and answering related questions.


2. Code Analysis

Key Code

1. Tool Layer Entry (largemodel/utils/tools_manager.py)

The analyze_video function in this file defines the execution flow of the tool.

# From largemodel/utils/tools_manager.py
class ToolsManager:
    # ...
    def analyze_video(self, args):
        """
        Analyze video file and provide content description.
        
        :param args: Arguments containing video path.
        :return: Dictionary with video description and path.
        """
        self.node.get_logger().info(f"Executing analyze_video() tool with args: {args}")
        try:
            video_path = args.get("video_path")
            # ... (intelligent path fallback mechanism)
            
            if video_path and os.path.exists(video_path):
                # ... (build the prompt)
                
                # Use a fully isolated, one-time context for video analysis to ensure a plain text description.
                simple_context = [{
                    "role": "system",
                    "content": "You are a video description assistant. ..."
                }]
                
                result = self.node.model_client.infer_with_video(video_path, prompt, message=simple_context)
                
                # ... (process the result)
                return {
                    "description": description,
                    "video_path": video_path
                }
            # ... (error handling)

2. Model Interface Layer and Frame Extraction (largemodel/utils/large_model_interface.py)

The functions in this file are responsible for processing the video file and passing it to the underlying model.

​x
# From largemodel/utils/large_model_interface.py

class model_interface:
    # ...
    def infer_with_video(self, video_path, text=None, message=None):
        """Unified video inference interface."""
        # ... (prepare messages)
        try:
            # Decide which specific implementation to call based on self.llm_platform
            if self.llm_platform == 'ollama':
                response_content = self.ollama_infer(self.messages, video_path=video_path)
            # ... (logic for other online platforms)
        # ...
        return {'response': response_content, 'messages': self.messages.copy()}

    def _extract_video_frames(self, video_path, max_frames=5):
        """Extract keyframes from a video for analysis."""
        try:
            import cv2
            # ... (video reading and frame interval calculation)
            while extracted_count < max_frames:
                # ... (loop to read video frames)
                if frame_count % frame_interval == 0:
                    # ... (save the frame as a temporary image)
                    frame_base64 = self.encode_file_to_base64(temp_path)
                    frame_images.append(frame_base64)
            # ...
            return frame_images
        # ... (exception handling)

Code Analysis

The implementation of the video analysis feature is one step more complex than image analysis: it requires a key preprocessing step at the model interface layer, namely frame extraction.

  1. Tool Layer (tools_manager.py):

    • The analyze_video function is the business entry point of video analysis. Its responsibility is clear: receive a video file path and build an instruction (prompt) for requesting a description of the video content.

    • It starts the analysis flow by calling the self.node.model_client.infer_with_video method. Like the visual understanding tool, it is completely unconcerned with the underlying model details and is only responsible for passing the "video file" and the "analysis instruction".

  2. Model Interface Layer (large_model_interface.py):

    • The infer_with_video function is the dispatcher connecting the upper-layer tools and the underlying model. It distributes the task to the corresponding specific implementation function according to the platform configuration (self.llm_platform).

    • Unlike processing a single image, processing a video requires an extra step. The _extract_video_frames method shows the general logic for implementing this step: it uses the cv2 library to read the video and extracts several keyframes from it (5 by default).

    • Each extracted frame is treated as an independent image and is usually encoded as a Base64 string.

    • Finally, a request containing the data of multiple frame images and the analysis instruction is sent to the large model. The model comprehensively analyzes these consecutive images to generate a description of the entire video content.

    • This "video to multiple images" preprocessing process is fully encapsulated in the model interface layer and is transparent to the tool layer.

In summary, the general flow of video analysis is: ToolsManager initiates the analysis request -> model_interface intercepts the request and calls _extract_video_frames to decompose the video file into multiple keyframe images -> model_interface sends these images together with the analysis instruction to the corresponding model platform according to the configuration -> the model returns a comprehensive description of the video -> the result is finally returned to ToolsManager. This design ensures the stability and universality of the upper-layer application.

3. Practical Operations

3.1 Configuring the Large Model

This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.

3.1.1 Configuring the LLM Platform (yahboom.yaml)

Open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

Set llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:

xxxxxxxxxx
model_service:
  ros__parameters:
    language: 'zh'
    llm_platform: 'ollama'  # Offline large model; change to 'tongyi' when using an online model
    regional_setting: "China"
  • Offline Ollama: set to llm_platform: 'ollama'.

  • Online Tongyi Qianwen: set to llm_platform: 'tongyi'.

If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.

3.1.2 Configuring the Model Interface (large_model_interface.yaml)

Open the model interface configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

According to the backend selected in the previous step, fill in only the corresponding group of parameters.

Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:

xxxxxxxxxx
qianwen_api_key: "Replace with your API-Key"
qianwen_model: "qwen-vl-max-latest"

Replace the placeholder with your own API Key. Keyframes are extracted from the video, so the selected model needs to support image input. Do not disclose real API Keys.

Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "llava"  # Replace with the model name in ollama list

The model name must be consistent with the output of ollama list, and the model needs to support image input.

After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.

The video analysis speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the video duration, the number of extracted frames, and the model size also affect the processing time.

3.2 Starting and Testing the Feature (Text Input Mode)

  1. Prepare the video file:

    Place a video file to be tested in the following path: /home/jetson/yahboom_ws/src/largemodel/resources_file/analyze_video

    Then name the video test_video.mp4.

  2. Start the largemodel main program (text mode):

    Open a terminal and run the following command:

    xxxxxxxxxx
    ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=true
  3. Send a text command: Open another terminal again and run the following command,

    xxxxxxxxxx
    ros2 run text_chat text_chat

    Then start typing the text: "Analyze this video".

  4. Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the analyze_video tool, extracted keyframes, and finally printed out the AI's summary of the video content.


3.3 Voice Interaction

Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:

xxxxxxxxxx
model_service:
  ros__parameters:
    use_oline_asr: False  # True: online ASR; False: offline ASR
    useolinetts: False    # True: online TTS; False: offline TTS

Complete the corresponding configuration according to the ASR/TTS tutorials, confirm that the video is in the directory specified above and named test_video.mp4, and then start:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking the device, say "Analyze the video content". Check the ASR recognized text, the video analysis tool call log, and the model result, and confirm the speaker announcement. Video analysis needs to process multiple frames, so offline inference takes longer.

4. Common Problems and Solutions

Problem 1: The message "Video file not found" or "Unable to extract keyframes from the video" is displayed.

Solution:

  1. Check the path: Make sure you have placed the video file in the specified path, that it is named test_video.mp4, and that you have permission to read the file.

  2. Video format/encoding: Make sure the video file is not damaged and is in .mp4 format.

Problem 2: Analyzing a long video takes a very long time.

Solution:

  1. Use a lighter model: Hardware resources are limited, so you can switch to a video understanding model with fewer parameters.

Multimodal text image application
Multi module visual position application

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文