Welcome to Jetson Orin NANO Developer Kit repository
01.Multimodal visual understand application
Multimodal Visual Understanding Application
Multimodal Visual Understanding Application1. Function Description1.1 Execution Flow1.2 Running Conditions2. Configuring the Large Model Backend2.1 Online Large Model: Tongyi Qianwen2.2 Offline Large Model: Ollama3. Text Interaction3.1 Starting the Program3.2 Initiating a Visual Question4. Voice Interaction4.1 Starting and Testing5. Acceptance and Troubleshooting5.1 Acceptance Steps5.2 Common Problems
Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.
1. Function Description
The visual understanding feature acquires the current frame through the camera, then sends the image and a prompt to a multimodal large model that supports image input, and finally returns the analysis result as text or voice. The course program completes photographing and image analysis through ToolsManager.seewhat(), and image inference is performed by model_interface.infer_with_image() calling the corresponding platform according to the llm_platform configuration.
This article puts online/offline large models and text/voice interaction into the same set of operations. The differences between the four combinations are as follows:
| Large Model | Text Interaction | Voice Interaction |
|---|---|---|
| Online (example: Tongyi Qianwen) | text_chat_mode:=true, input with text_chat | Start the voice pipeline; llm_platform: 'tongyi', ASR/TTS can each be set to online or offline |
| Offline (Ollama) | text_chat_mode:=true, input with text_chat | Start the voice pipeline; llm_platform: 'ollama', ASR/TTS can each be set to online or offline |
Online/offline describes the large model backend; ASR (speech-to-text) and TTS (text-to-speech) in voice interaction are configured independently. Text interaction does not use the microphone or speaker.
1.1 Execution Flow
seewhatrequests the camera to capture one frame.The program passes the image and the scene description prompt to
infer_with_image().The model returns the image analysis text.
Text mode displays the result in the terminal; voice mode passes the result to TTS for announcement.
Note:
seewhatis a tool call. Whether it can be triggered automatically by a natural language question in text mode depends on the current model and the tool call logic of the course program; supporting image input does not mean the model also supports tool calls. In voice mode, the course deployment version can directly trigger photographing and analysis by recognizing visual keywords in the text; if the source version you use does not have that logic, the model also needs to initiate the tool call correctly. When testing, check whetherExecuting seewhat()appears in the logs; do not judge that the camera pipeline succeeded solely from the model's text answer.
1.2 Running Conditions
The
largemodelworkspace, camera, and voice interaction environment have been configured according to the prerequisite tutorials.The camera is connected and the current user has access permission; the model must be able to receive image input.
An online large model requires an available network, valid platform credentials, and a model that supports image input. An offline large model requires the Ollama service to be running, a downloaded model that supports image input, and sufficient memory.
Voice mode also requires an available microphone, speaker, ASR/TTS configuration, and the corresponding service or local models.
2. Configuring the Large Model Backend
The configuration files are located at:
~/yahboom_ws/src/largemodel/config/yahboom.yaml~/yahboom_ws/src/largemodel/config/large_model_interface.yamlAfter switching the backend each time, save the configuration and restart largemodel. Do not configure llm_platform with multiple values at the same time.
2.1 Online Large Model: Tongyi Qianwen
The following uses Tongyi Qianwen as an example. First apply for and confirm that the account has available quota according to the online large model basic tutorial, then open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlFind the Tongyi Qianwen configuration item, fill in your own API Key, and select a vision model that supports image input. Do not keep the example placeholder as is:
xxxxxxxxxxqianwen_api_key"Replace with your API-Key"qianwen_model"qwen-vl-max-latest"After saving the model interface configuration, open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlIn the existing model_service.ros__parameters parameter block, set llm_platform to tongyi. If this parameter already exists, simply modify the original value; do not add duplicate YAML keys with the same name:
xxxxxxxxxxmodel_service ros__parameters llm_platform'tongyi'After saving both configuration files, restart largemodel for the settings to take effect. Running requires that the Jetson can access the online service, and the selected model must support image input. Do not publish real API Keys to public code repositories or screenshots; the names, credentials, and vision model configurations of other online platforms are subject to the tutorials of the corresponding platforms.
2.2 Offline Large Model: Ollama
Make sure the Ollama service has started and that a model supporting image input has been downloaded. Edit large_model_interface.yaml:
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"qwen3-vl:2b" # Example; replace with a vision model already downloaded on this machineSelect Ollama in yahboom.yaml:
xxxxxxxxxxmodel_service ros__parameters llm_platform'ollama'Check the local Ollama models and service:
xxxxxxxxxxollama listsystemctl status ollama --no-pagerThe model name must be consistent with the name shown by ollama list. Different models have different support for image input and tool calling; if the photographing tool is not triggered in text mode, first confirm the model's tool calling capability and the program logs. If the voice pipeline is configured with a visual keyword passthrough, you can use an explicit visual request to verify that branch.
The inference speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed also depends on the model size and image resolution.
3. Text Interaction
Text interaction uses the text_chat node to input questions and does not require starting ASR/TTS. First select an online or offline large model backend according to Section 2.
3.1 Starting the Program
Open the first terminal, enter a terminal where the ROS 2 Jazzy and workspace environments have been configured, and then start:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.py text_chat_mode:=trueOpen a second terminal, likewise load the ROS 2 and workspace environments, and then run:
xxxxxxxxxxros2 run text_chat text_chat3.2 Initiating a Visual Question
In the text_chat terminal, enter a clear visual request, for example:
xxxxxxxxxxPlease call seewhat to capture the current frame and tell me what is in the picture.Observe the largemodel terminal: if Executing seewhat() appears in the logs, the program has entered the photographing and image analysis flow; a scene description returned by the model should follow. After testing, press Ctrl+C in the two terminals respectively to close the text client and the main program.
If the model only answers in text and there is no Executing seewhat(), it means the tool was not triggered, which does not mean the camera capture succeeded. Try a model that supports tool calling, or switch to voice mode to verify the visual keyword trigger logic in the course version.
4. Voice Interaction
Voice interaction receives requests through the microphone, triggers seewhat after recognizing a visual question, and then announces the analysis result through TTS. First select the large model backend according to Section 2, then check the voice options of model_service.ros__parameters in yahboom.yaml:
xxxxxxxxxxmodel_service ros__parameters llm_platform'ollama' # Change to 'tongyi' for an online large model use_oline_asrFalse # True: online ASR; False: offline ASR useolinettsFalse # True: online TTS; False: offline TTS regional_setting"China"The example above uses offline ASR and offline TTS. If you want speech recognition and speech synthesis to also use online services, set use_oline_asr and useolinetts to True respectively, and configure the service credentials and network according to the corresponding ASR/TTS tutorials. Whichever voice service option is chosen, the LLM backend is determined solely by llm_platform. For example, for an online LLM with offline voice front-end and back-end, set tongyi and set both voice parameters to False; for an offline LLM with online ASR/TTS, set ollama and set both voice parameters to True.
4.1 Starting and Testing
Confirm that the microphone, speaker, and voice-related services are ready. Open a terminal, load the ROS 2 and workspace environments, and then start:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyAfter waking the device, ask a clear visual question, for example "What do you see?" or "Please describe the current picture." Check the ASR recognized text and the Executing seewhat() log in the startup terminal; after a moment you should hear the analysis result announced by TTS. Press Ctrl+C to stop the program.
5. Acceptance and Troubleshooting
5.1 Acceptance Steps
It is recommended to verify the basic pipelines separately first, and then test the complete combination:
Use
ollama listandsystemctl status ollamato check the offline service; in online mode, confirm that the API Key, model name, and network are available.Confirm that the camera device is visible:
xxxxxxxxxxls /dev/video*Issue a clear photographing and analysis request in text mode or voice mode, and confirm that
Executing seewhat()appears in the logs.Confirm that the image model returns content; in voice mode, also confirm that the speaker announces the result.
5.2 Common Problems
| Symptom | Check Direction |
|---|---|
| Online model authentication fails or the request times out | Check qianwen_api_key, the model name, the network connection, and the platform account permissions; confirm that the selected model supports image input. |
| The Ollama connection is refused or the model cannot be found | Check the Ollama service status and the spelling of ollama_host and ollama_model, and confirm that the model has been downloaded. |
| A text answer is received, but there is no photographing log | The model may not have called seewhat. Confirm the model's tool calling capability; in text mode, do not regard a natural language answer as proof that the camera pipeline has run. |
seewhat reports that the camera cannot be opened or photographing fails | Check the camera connection, the /dev/video* device node, device occupation, and the current user's access permission. |
| ASR has no recognition result | Check the microphone connection, the use_oline_asr selection, the ASR credentials or offline model, and the voice device parameters. |
| There is a text result but no voice announcement | Check useolinetts, the TTS service/model, the speaker, and the audio output configuration. |
| Offline inference is very slow and the program is terminated by the system | Check the available memory, first verify separately with a smaller vision model and text mode, and close other high-usage programs. |