yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

01.Multimodal visual understand application

Multimodal Visual Understanding Application

Multimodal Visual Understanding Application1. Function Description1.1 Execution Flow1.2 Running Conditions2. Configuring the Large Model Backend2.1 Online Large Model: Tongyi Qianwen2.2 Offline Large Model: Ollama3. Text Interaction3.1 Starting the Program3.2 Initiating a Visual Question4. Voice Interaction4.1 Starting and Testing5. Acceptance and Troubleshooting5.1 Acceptance Steps5.2 Common Problems

Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.


1. Function Description

The visual understanding feature acquires the current frame through the camera, then sends the image and a prompt to a multimodal large model that supports image input, and finally returns the analysis result as text or voice. The course program completes photographing and image analysis through ToolsManager.seewhat(), and image inference is performed by model_interface.infer_with_image() calling the corresponding platform according to the llm_platform configuration.

This article puts online/offline large models and text/voice interaction into the same set of operations. The differences between the four combinations are as follows:

Large ModelText InteractionVoice Interaction
Online (example: Tongyi Qianwen)text_chat_mode:=true, input with text_chatStart the voice pipeline; llm_platform: 'tongyi', ASR/TTS can each be set to online or offline
Offline (Ollama)text_chat_mode:=true, input with text_chatStart the voice pipeline; llm_platform: 'ollama', ASR/TTS can each be set to online or offline

Online/offline describes the large model backend; ASR (speech-to-text) and TTS (text-to-speech) in voice interaction are configured independently. Text interaction does not use the microphone or speaker.

1.1 Execution Flow

  1. seewhat requests the camera to capture one frame.

  2. The program passes the image and the scene description prompt to infer_with_image().

  3. The model returns the image analysis text.

  4. Text mode displays the result in the terminal; voice mode passes the result to TTS for announcement.

Note: seewhat is a tool call. Whether it can be triggered automatically by a natural language question in text mode depends on the current model and the tool call logic of the course program; supporting image input does not mean the model also supports tool calls. In voice mode, the course deployment version can directly trigger photographing and analysis by recognizing visual keywords in the text; if the source version you use does not have that logic, the model also needs to initiate the tool call correctly. When testing, check whether Executing seewhat() appears in the logs; do not judge that the camera pipeline succeeded solely from the model's text answer.

1.2 Running Conditions

  • The largemodel workspace, camera, and voice interaction environment have been configured according to the prerequisite tutorials.

  • The camera is connected and the current user has access permission; the model must be able to receive image input.

  • An online large model requires an available network, valid platform credentials, and a model that supports image input. An offline large model requires the Ollama service to be running, a downloaded model that supports image input, and sufficient memory.

  • Voice mode also requires an available microphone, speaker, ASR/TTS configuration, and the corresponding service or local models.


2. Configuring the Large Model Backend

The configuration files are located at:

~/yahboom_ws/src/largemodel/config/yahboom.yaml
~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

After switching the backend each time, save the configuration and restart largemodel. Do not configure llm_platform with multiple values at the same time.

2.1 Online Large Model: Tongyi Qianwen

The following uses Tongyi Qianwen as an example. First apply for and confirm that the account has available quota according to the online large model basic tutorial, then open the model interface configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

Find the Tongyi Qianwen configuration item, fill in your own API Key, and select a vision model that supports image input. Do not keep the example placeholder as is:

xxxxxxxxxx
qianwen_api_key: "Replace with your API-Key"
qianwen_model: "qwen-vl-max-latest"

After saving the model interface configuration, open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

In the existing model_service.ros__parameters parameter block, set llm_platform to tongyi. If this parameter already exists, simply modify the original value; do not add duplicate YAML keys with the same name:

xxxxxxxxxx
model_service:
  ros__parameters:
    llm_platform: 'tongyi'

After saving both configuration files, restart largemodel for the settings to take effect. Running requires that the Jetson can access the online service, and the selected model must support image input. Do not publish real API Keys to public code repositories or screenshots; the names, credentials, and vision model configurations of other online platforms are subject to the tutorials of the corresponding platforms.

2.2 Offline Large Model: Ollama

Make sure the Ollama service has started and that a model supporting image input has been downloaded. Edit large_model_interface.yaml:

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "qwen3-vl:2b"  # Example; replace with a vision model already downloaded on this machine

Select Ollama in yahboom.yaml:

xxxxxxxxxx
model_service:
  ros__parameters:
    llm_platform: 'ollama'

Check the local Ollama models and service:

xxxxxxxxxx
ollama list
systemctl status ollama --no-pager

The model name must be consistent with the name shown by ollama list. Different models have different support for image input and tool calling; if the photographing tool is not triggered in text mode, first confirm the model's tool calling capability and the program logs. If the voice pipeline is configured with a visual keyword passthrough, you can use an explicit visual request to verify that branch.


The inference speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed also depends on the model size and image resolution.

3. Text Interaction

Text interaction uses the text_chat node to input questions and does not require starting ASR/TTS. First select an online or offline large model backend according to Section 2.

3.1 Starting the Program

Open the first terminal, enter a terminal where the ROS 2 Jazzy and workspace environments have been configured, and then start:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=true

Open a second terminal, likewise load the ROS 2 and workspace environments, and then run:

xxxxxxxxxx
ros2 run text_chat text_chat

3.2 Initiating a Visual Question

In the text_chat terminal, enter a clear visual request, for example:

xxxxxxxxxx
Please call seewhat to capture the current frame and tell me what is in the picture.

Observe the largemodel terminal: if Executing seewhat() appears in the logs, the program has entered the photographing and image analysis flow; a scene description returned by the model should follow. After testing, press Ctrl+C in the two terminals respectively to close the text client and the main program.

If the model only answers in text and there is no Executing seewhat(), it means the tool was not triggered, which does not mean the camera capture succeeded. Try a model that supports tool calling, or switch to voice mode to verify the visual keyword trigger logic in the course version.


4. Voice Interaction

Voice interaction receives requests through the microphone, triggers seewhat after recognizing a visual question, and then announces the analysis result through TTS. First select the large model backend according to Section 2, then check the voice options of model_service.ros__parameters in yahboom.yaml:

xxxxxxxxxx
model_service:
  ros__parameters:
    llm_platform: 'ollama'  # Change to 'tongyi' for an online large model
    use_oline_asr: False    # True: online ASR; False: offline ASR
    useolinetts: False      # True: online TTS; False: offline TTS
    regional_setting: "China"

The example above uses offline ASR and offline TTS. If you want speech recognition and speech synthesis to also use online services, set use_oline_asr and useolinetts to True respectively, and configure the service credentials and network according to the corresponding ASR/TTS tutorials. Whichever voice service option is chosen, the LLM backend is determined solely by llm_platform. For example, for an online LLM with offline voice front-end and back-end, set tongyi and set both voice parameters to False; for an offline LLM with online ASR/TTS, set ollama and set both voice parameters to True.

4.1 Starting and Testing

Confirm that the microphone, speaker, and voice-related services are ready. Open a terminal, load the ROS 2 and workspace environments, and then start:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking the device, ask a clear visual question, for example "What do you see?" or "Please describe the current picture." Check the ASR recognized text and the Executing seewhat() log in the startup terminal; after a moment you should hear the analysis result announced by TTS. Press Ctrl+C to stop the program.


5. Acceptance and Troubleshooting

5.1 Acceptance Steps

It is recommended to verify the basic pipelines separately first, and then test the complete combination:

  1. Use ollama list and systemctl status ollama to check the offline service; in online mode, confirm that the API Key, model name, and network are available.

  2. Confirm that the camera device is visible:

    xxxxxxxxxx
    ls /dev/video*
  3. Issue a clear photographing and analysis request in text mode or voice mode, and confirm that Executing seewhat() appears in the logs.

  4. Confirm that the image model returns content; in voice mode, also confirm that the speaker announces the result.

5.2 Common Problems

SymptomCheck Direction
Online model authentication fails or the request times outCheck qianwen_api_key, the model name, the network connection, and the platform account permissions; confirm that the selected model supports image input.
The Ollama connection is refused or the model cannot be foundCheck the Ollama service status and the spelling of ollama_host and ollama_model, and confirm that the model has been downloaded.
A text answer is received, but there is no photographing logThe model may not have called seewhat. Confirm the model's tool calling capability; in text mode, do not regard a natural language answer as proof that the camera pipeline has run.
seewhat reports that the camera cannot be opened or photographing failsCheck the camera connection, the /dev/video* device node, device occupation, and the current user's access permission.
ASR has no recognition resultCheck the microphone connection, the use_oline_asr selection, the ASR credentials or offline model, and the voice device parameters.
There is a text result but no voice announcementCheck useolinetts, the TTS service/model, the speaker, and the audio output configuration.
Offline inference is very slow and the program is terminated by the systemCheck the available memory, first verify separately with a smaller vision model and text mode, and close other high-usage programs.
AI large model offline voice assistant
Multimodal text image application

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文