Welcome to Jetson Orin NANO Developer Kit repository
03.Multimodal video analysis application
Multimodal Video Analysis Application
Multimodal Video Analysis Application1. Concept Introduction1.1 What Is "Video Analysis"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer and Frame Extraction (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Input Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The message "Video file not found" or "Unable to extract keyframes from the video" is displayed.Problem 2: Analyzing a long video takes a very long time.
Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.
1. Concept Introduction
1.1 What Is "Video Analysis"?
In the largemodel project, the multimodal video analysis feature means enabling the robot to process a video and summarize its core content in natural language, describe key events, or answer specific questions about the video. This elevates the robot from understanding only static images to understanding a dynamic world with temporal relationships.
The core tool of this feature is analyze_video. When the user provides a video file and asks a question (for example "Summarize what this video is about"), the system calls this tool to process and analyze the video, and returns the AI's text answer.
1.2 Implementation Principles Overview
The core challenge of offline video analysis is how to enable the model to efficiently process video data containing hundreds or thousands of frames. One mainstream implementation principle is as follows:
Keyframe Extraction: First, the system does not process every frame of the video, but extracts the most representative keyframes through algorithms (such as scene boundary detection or fixed-interval sampling), which greatly reduces the amount of data that needs to be processed.
Image Encoding: Each extracted keyframe is fed, just as in the visual understanding application, into a vision encoder and converted into a digital vector containing the image information.
Temporal Information Fusion: This is the biggest difference from single-image understanding. The model needs to understand the temporal order among these keyframes. Usually a recurrent neural network (RNN) or Transformer model is used to fuse the vectors of all keyframes into a "memory" vector that can represent the dynamic content of the entire video.
Question Answering and Generation: The user's text question is encoded and then cross-modally fused with this "memory" vector. Finally, the language model part generates a summary of the entire video or the answer to a specific question based on this fused information.
Simply put, it means condensing a video into a few key pictures and their order, then understanding the whole story like reading a comic strip, and answering related questions.
2. Code Analysis
Key Code
1. Tool Layer Entry (largemodel/utils/tools_manager.py)
The analyze_video function in this file defines the execution flow of the tool.
# From largemodel/utils/tools_manager.pyclass ToolsManager: # ... def analyze_video(self, args): """ Analyze video file and provide content description. :param args: Arguments containing video path. :return: Dictionary with video description and path. """ self.node.get_logger().info(f"Executing analyze_video() tool with args: {args}") try: video_path = args.get("video_path") # ... (intelligent path fallback mechanism) if video_path and os.path.exists(video_path): # ... (build the prompt) # Use a fully isolated, one-time context for video analysis to ensure a plain text description. simple_context = [{ "role": "system", "content": "You are a video description assistant. ..." }] result = self.node.model_client.infer_with_video(video_path, prompt, message=simple_context) # ... (process the result) return { "description": description, "video_path": video_path } # ... (error handling)2. Model Interface Layer and Frame Extraction (largemodel/utils/large_model_interface.py)
The functions in this file are responsible for processing the video file and passing it to the underlying model.
x# From largemodel/utils/large_model_interface.py
class model_interface: # ... def infer_with_video(self, video_path, text=None, message=None): """Unified video inference interface.""" # ... (prepare messages) try: # Decide which specific implementation to call based on self.llm_platform if self.llm_platform == 'ollama': response_content = self.ollama_infer(self.messages, video_path=video_path) # ... (logic for other online platforms) # ... return {'response': response_content, 'messages': self.messages.copy()}
def _extract_video_frames(self, video_path, max_frames=5): """Extract keyframes from a video for analysis.""" try: import cv2 # ... (video reading and frame interval calculation) while extracted_count < max_frames: # ... (loop to read video frames) if frame_count % frame_interval == 0: # ... (save the frame as a temporary image) frame_base64 = self.encode_file_to_base64(temp_path) frame_images.append(frame_base64) # ... return frame_images # ... (exception handling)Code Analysis
The implementation of the video analysis feature is one step more complex than image analysis: it requires a key preprocessing step at the model interface layer, namely frame extraction.
Tool Layer (
tools_manager.py):The
analyze_videofunction is the business entry point of video analysis. Its responsibility is clear: receive a video file path and build an instruction (prompt) for requesting a description of the video content.It starts the analysis flow by calling the
self.node.model_client.infer_with_videomethod. Like the visual understanding tool, it is completely unconcerned with the underlying model details and is only responsible for passing the "video file" and the "analysis instruction".
Model Interface Layer (
large_model_interface.py):The
infer_with_videofunction is the dispatcher connecting the upper-layer tools and the underlying model. It distributes the task to the corresponding specific implementation function according to the platform configuration (self.llm_platform).Unlike processing a single image, processing a video requires an extra step. The
_extract_video_framesmethod shows the general logic for implementing this step: it uses thecv2library to read the video and extracts several keyframes from it (5 by default).Each extracted frame is treated as an independent image and is usually encoded as a Base64 string.
Finally, a request containing the data of multiple frame images and the analysis instruction is sent to the large model. The model comprehensively analyzes these consecutive images to generate a description of the entire video content.
This "video to multiple images" preprocessing process is fully encapsulated in the model interface layer and is transparent to the tool layer.
In summary, the general flow of video analysis is: ToolsManager initiates the analysis request -> model_interface intercepts the request and calls _extract_video_frames to decompose the video file into multiple keyframe images -> model_interface sends these images together with the analysis instruction to the corresponding model platform according to the configuration -> the model returns a comprehensive description of the video -> the result is finally returned to ToolsManager. This design ensures the stability and universality of the upper-layer application.
3. Practical Operations
3.1 Configuring the Large Model
This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.
3.1.1 Configuring the LLM Platform (yahboom.yaml)
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlSet llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:
xxxxxxxxxxmodel_service ros__parameters language'zh' llm_platform'ollama' # Offline large model; change to 'tongyi' when using an online model regional_setting"China"Offline Ollama: set to
llm_platform: 'ollama'.Online Tongyi Qianwen: set to
llm_platform: 'tongyi'.
If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.
3.1.2 Configuring the Model Interface (large_model_interface.yaml)
Open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlAccording to the backend selected in the previous step, fill in only the corresponding group of parameters.
Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:
xxxxxxxxxxqianwen_api_key"Replace with your API-Key"qianwen_model"qwen-vl-max-latest"Replace the placeholder with your own API Key. Keyframes are extracted from the video, so the selected model needs to support image input. Do not disclose real API Keys.
Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"llava" # Replace with the model name in ollama listThe model name must be consistent with the output of ollama list, and the model needs to support image input.
After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.
The video analysis speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the video duration, the number of extracted frames, and the model size also affect the processing time.
3.2 Starting and Testing the Feature (Text Input Mode)
Prepare the video file:
Place a video file to be tested in the following path:
/home/jetson/yahboom_ws/src/largemodel/resources_file/analyze_videoThen name the video
test_video.mp4.Start the
largemodelmain program (text mode):Open a terminal and run the following command:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.py text_chat_mode:=trueSend a text command: Open another terminal again and run the following command,
xxxxxxxxxxros2 run text_chat text_chatThen start typing the text: "Analyze this video".
Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the
analyze_videotool, extracted keyframes, and finally printed out the AI's summary of the video content.
3.3 Voice Interaction
Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:
xxxxxxxxxxmodel_service ros__parameters use_oline_asrFalse # True: online ASR; False: offline ASR useolinettsFalse # True: online TTS; False: offline TTSComplete the corresponding configuration according to the ASR/TTS tutorials, confirm that the video is in the directory specified above and named test_video.mp4, and then start:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyAfter waking the device, say "Analyze the video content". Check the ASR recognized text, the video analysis tool call log, and the model result, and confirm the speaker announcement. Video analysis needs to process multiple frames, so offline inference takes longer.
4. Common Problems and Solutions
Problem 1: The message "Video file not found" or "Unable to extract keyframes from the video" is displayed.
Solution:
Check the path: Make sure you have placed the video file in the specified path, that it is named test_video.mp4, and that you have permission to read the file.
Video format/encoding: Make sure the video file is not damaged and is in .mp4 format.
Problem 2: Analyzing a long video takes a very long time.
Solution:
Use a lighter model: Hardware resources are limited, so you can switch to a video understanding model with fewer parameters.