yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

04.Multi module visual position application

Multimodal Visual Localization Application

Multimodal Visual Localization Application1. Concept Introduction1.1 What Is "Multimodal Visual Localization"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The message "Image file not found" is displayed.

Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.


1. Concept Introduction

1.1 What Is "Multimodal Visual Localization"?

Multimodal visual localization is a technology that combines multiple sensor inputs (such as cameras, depth sensors, IMUs, and so on) with algorithmic processing to achieve recognition and tracking of the precise position and pose of a device or user in the environment. This technology does not rely on a single type of sensor data, but integrates information from different perception modalities, thereby improving the accuracy and robustness of localization.

1.2 Implementation Principles Overview

  1. Cross-modal representation learning: To give the LLM the ability to process visual information, a mechanism needs to be developed to convert visual signals into a form the model can understand. This may involve using convolutional neural networks (CNNs) or other architectures suitable for image processing to extract features and map them into the same embedding space as text.

  2. Joint training: By designing an appropriate loss function, text and visual data can be trained simultaneously within the same framework, so that the model learns to establish connections between the two modalities. For example, in a question answering system, an answer can be given based either on the provided text question or on the related image content.

  3. Vision-guided language generation/understanding: Once an effective cross-modal representation has been established, visual information can be used to enhance the capabilities of the language model. For example, when given a photo, the model can not only describe what is happening in the picture, but also answer specific questions about the scene, and even execute instructions based on visual cues (such as navigating to a certain location).


2. Code Analysis

Key Code

1. Tool Layer Entry (largemodel/utils/tools_manager.py)

The visual_positioning function in this file defines the execution flow of the tool, in particular how it builds a prompt containing the target object name and the format requirement.

# From largemodel/utils/tools_manager.py
class ToolsManager:
    # ...
    def visual_positioning(self, args):
        """
        Locate object coordinates in image and save results to MD file.
        
        :param args: Arguments containing image path and object name.
        :return: Dictionary with file path and coordinate data.
        """
        self.node.get_logger().info(f"Executing visual_positioning() tool with args: {args}")
        try:
            image_path = args.get("image_path")
            object_name = args.get("object_name")
            # ... (path fallback mechanism and parameter check)
            
            # Construct a prompt asking the large model to identify the coordinates of the specified object.
            if self.node.language == 'zh':
                prompt = f"Please carefully analyze this image and use a box to locate each {object_name} in the image..."
            else:
                prompt = f"Please carefully analyze this image and find the position of all {object_name}..."
            
            # ... (build an independent message context)
            
            result = self.node.model_client.infer_with_image(image_path, prompt, message=message_to_use)
            
            # ... (process and parse the returned coordinate text)
            
            return {
                "file_path": md_file_path,
                "coordinates_content": coordinates_content,
                "explanation_content": explanation_content
            }
        # ... (error handling)

2. Model Interface Layer (largemodel/utils/large_model_interface.py)

The infer_with_image function in this file is the unified entry point for all image-related tasks.

​x
# From largemodel/utils/large_model_interface.py

class model_interface:
    # ...
    def infer_with_image(self, image_path, text=None, message=None):
        """Unified image inference interface."""
        # ... (prepare messages)
        try:
            # Decide which specific implementation to call based on the value of self.llm_platform
            if self.llm_platform == 'ollama':
                response_content = self.ollama_infer(self.messages, image_path=image_path)
            elif self.llm_platform == 'tongyi':
                # ... logic for calling the Tongyi model
                pass
            # ... (logic for other platforms)
        # ...
        return {'response': response_content, 'messages': self.messages.copy()}

Code Analysis

The core of the visual localization feature lies in guiding the large model to output structured data through precise instructions. It likewise follows the layered design of the tool layer and the model interface layer.

  1. Tool Layer (tools_manager.py):

    • The visual_positioning function is the business core of this feature. It receives two key parameters: image_path (the image path) and object_name (the name of the object to be located).

    • The most critical operation of this function is to build a highly customized prompt. It does not simply ask the model to describe the image; instead, it embeds object_name into a carefully designed template, explicitly requiring the model to "locate the position of every {object_name} in the image", and implicitly or explicitly requiring the result to be returned in a specific format (such as an array of coordinates).

    • After building the prompt, it calls the infer_with_image method of the model interface layer, passing the image together with this customized instruction.

    • After obtaining the returned text from the model interface layer, it also needs to perform post-processing: using methods such as regular expressions to parse the precise coordinate data from the model's natural language reply.

    • Finally, it returns the parsed structured coordinate data to the upper-layer application.

  2. Model Interface Layer (large_model_interface.py):

    • The infer_with_image function still plays the role of the "dispatching center". It receives the image and prompt from visual_positioning and distributes the task to the correct backend model implementation according to the current configuration (self.llm_platform).

    • For visual localization tasks, the responsibility of the model interface layer is basically the same as for visual understanding tasks: package the image data and text instruction correctly, send them to the selected model platform, and then return the returned text result to the tool layer unchanged. All platform-specific implementation details are encapsulated in this layer.

In summary, the general flow of visual localization is: ToolsManager receives the target object name and builds a precise prompt requiring coordinates to be returned -> ToolsManager calls the model interface -> model_interface packages the image and the prompt together and sends them to the corresponding model platform according to the configuration -> the model returns text containing the coordinate information -> model_interface returns the text to ToolsManager -> ToolsManager parses the text, extracts the structured coordinate data, and returns it. This process demonstrates how prompt engineering techniques can be used to make a general-purpose vision large model complete a more specific and more structured task.

3. Practical Operations

3.1 Configuring the Large Model

This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.

3.1.1 Configuring the LLM Platform (yahboom.yaml)

Open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

Set llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:

xxxxxxxxxx
model_service:
  ros__parameters:
    language: 'zh'
    llm_platform: 'ollama'  # Offline large model; change to 'tongyi' when using an online model
    regional_setting: "China"
  • Offline Ollama: set to llm_platform: 'ollama'.

  • Online Tongyi Qianwen: set to llm_platform: 'tongyi'.

If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.

3.1.2 Configuring the Model Interface (large_model_interface.yaml)

Open the model interface configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

According to the backend selected in the previous step, fill in only the corresponding group of parameters.

Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:

xxxxxxxxxx
qianwen_api_key: "Replace with your API-Key"
qianwen_model: "qwen-vl-max-latest"

Replace the placeholder with your own API Key. Visual localization requires the model to understand the image content and to return the target position or coordinate information as prompted. Do not disclose real API Keys.

Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "llava"  # Replace with the model name in ollama list

The model name must be consistent with the output of ollama list, and the model needs to support image input.

After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.

The visual localization inference speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed is also related to the model size and the input image.

3.2 Starting and Testing the Feature (Text Mode)

  1. Prepare the image file:

    Place an image file to be tested in the following path: /home/jetson/yahboom_ws/src/largemodel/resources_file/visual_positioning

    Then name the image test_image.jpg

  2. Start the largemodel main program (text mode):

    Open a terminal and run the following command:

    xxxxxxxxxx
    ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=true
  3. Send a text command: Open another terminal again and run the following command,

    xxxxxxxxxx
    ros2 run text_chat text_chat

    Then start typing the text: "Locate the position of the dinosaur in the image".

  4. Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the visual_positioning tool, that visual_positioning has finished executing, and that the coordinates were saved to a document.

    We can find this document in the path ~/yahboom_ws/src/largemodel/resources_file/visual_positioning.


3.3 Voice Interaction

Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:

xxxxxxxxxx
model_service:
  ros__parameters:
    use_oline_asr: False  # True: online ASR; False: offline ASR
    useolinetts: False    # True: online TTS; False: offline TTS

Complete the corresponding configuration according to the ASR/TTS tutorials. Place the image in the directory specified above and name it test_image.jpg, then start:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking the device, say "Locate the position of the dinosaur in the image". Check the ASR recognized text, the visual_positioning tool call log, and the output document. Voice tool calling depends on the current program version and the model capability; if there is no tool call log, first verify in text mode.

4. Common Problems and Solutions

Problem 1: The message "Image file not found" is displayed.

Solution:

  1. Check the path: Make sure you have placed the image file in the specified path, that it is named test_image.jpg, and that you have permission to read the file.

  2. Image format: Make sure the image file is not damaged and is in .jpg format.

Multimodal video analysis application
Multimodal table scanning application

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文