Welcome to Jetson Orin NANO Developer Kit repository
04.Multi module visual position application
Multimodal Visual Localization Application
Multimodal Visual Localization Application1. Concept Introduction1.1 What Is "Multimodal Visual Localization"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The message "Image file not found" is displayed.
Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.
1. Concept Introduction
1.1 What Is "Multimodal Visual Localization"?
Multimodal visual localization is a technology that combines multiple sensor inputs (such as cameras, depth sensors, IMUs, and so on) with algorithmic processing to achieve recognition and tracking of the precise position and pose of a device or user in the environment. This technology does not rely on a single type of sensor data, but integrates information from different perception modalities, thereby improving the accuracy and robustness of localization.
1.2 Implementation Principles Overview
Cross-modal representation learning: To give the LLM the ability to process visual information, a mechanism needs to be developed to convert visual signals into a form the model can understand. This may involve using convolutional neural networks (CNNs) or other architectures suitable for image processing to extract features and map them into the same embedding space as text.
Joint training: By designing an appropriate loss function, text and visual data can be trained simultaneously within the same framework, so that the model learns to establish connections between the two modalities. For example, in a question answering system, an answer can be given based either on the provided text question or on the related image content.
Vision-guided language generation/understanding: Once an effective cross-modal representation has been established, visual information can be used to enhance the capabilities of the language model. For example, when given a photo, the model can not only describe what is happening in the picture, but also answer specific questions about the scene, and even execute instructions based on visual cues (such as navigating to a certain location).
2. Code Analysis
Key Code
1. Tool Layer Entry (largemodel/utils/tools_manager.py)
The visual_positioning function in this file defines the execution flow of the tool, in particular how it builds a prompt containing the target object name and the format requirement.
# From largemodel/utils/tools_manager.pyclass ToolsManager: # ... def visual_positioning(self, args): """ Locate object coordinates in image and save results to MD file. :param args: Arguments containing image path and object name. :return: Dictionary with file path and coordinate data. """ self.node.get_logger().info(f"Executing visual_positioning() tool with args: {args}") try: image_path = args.get("image_path") object_name = args.get("object_name") # ... (path fallback mechanism and parameter check) # Construct a prompt asking the large model to identify the coordinates of the specified object. if self.node.language == 'zh': prompt = f"Please carefully analyze this image and use a box to locate each {object_name} in the image..." else: prompt = f"Please carefully analyze this image and find the position of all {object_name}..." # ... (build an independent message context) result = self.node.model_client.infer_with_image(image_path, prompt, message=message_to_use) # ... (process and parse the returned coordinate text) return { "file_path": md_file_path, "coordinates_content": coordinates_content, "explanation_content": explanation_content } # ... (error handling)2. Model Interface Layer (largemodel/utils/large_model_interface.py)
The infer_with_image function in this file is the unified entry point for all image-related tasks.
x# From largemodel/utils/large_model_interface.py
class model_interface: # ... def infer_with_image(self, image_path, text=None, message=None): """Unified image inference interface.""" # ... (prepare messages) try: # Decide which specific implementation to call based on the value of self.llm_platform if self.llm_platform == 'ollama': response_content = self.ollama_infer(self.messages, image_path=image_path) elif self.llm_platform == 'tongyi': # ... logic for calling the Tongyi model pass # ... (logic for other platforms) # ... return {'response': response_content, 'messages': self.messages.copy()}Code Analysis
The core of the visual localization feature lies in guiding the large model to output structured data through precise instructions. It likewise follows the layered design of the tool layer and the model interface layer.
Tool Layer (
tools_manager.py):The
visual_positioningfunction is the business core of this feature. It receives two key parameters:image_path(the image path) andobject_name(the name of the object to be located).The most critical operation of this function is to build a highly customized prompt. It does not simply ask the model to describe the image; instead, it embeds
object_nameinto a carefully designed template, explicitly requiring the model to "locate the position of every {object_name} in the image", and implicitly or explicitly requiring the result to be returned in a specific format (such as an array of coordinates).After building the prompt, it calls the
infer_with_imagemethod of the model interface layer, passing the image together with this customized instruction.After obtaining the returned text from the model interface layer, it also needs to perform post-processing: using methods such as regular expressions to parse the precise coordinate data from the model's natural language reply.
Finally, it returns the parsed structured coordinate data to the upper-layer application.
Model Interface Layer (
large_model_interface.py):The
infer_with_imagefunction still plays the role of the "dispatching center". It receives the image and prompt fromvisual_positioningand distributes the task to the correct backend model implementation according to the current configuration (self.llm_platform).For visual localization tasks, the responsibility of the model interface layer is basically the same as for visual understanding tasks: package the image data and text instruction correctly, send them to the selected model platform, and then return the returned text result to the tool layer unchanged. All platform-specific implementation details are encapsulated in this layer.
In summary, the general flow of visual localization is: ToolsManager receives the target object name and builds a precise prompt requiring coordinates to be returned -> ToolsManager calls the model interface -> model_interface packages the image and the prompt together and sends them to the corresponding model platform according to the configuration -> the model returns text containing the coordinate information -> model_interface returns the text to ToolsManager -> ToolsManager parses the text, extracts the structured coordinate data, and returns it. This process demonstrates how prompt engineering techniques can be used to make a general-purpose vision large model complete a more specific and more structured task.
3. Practical Operations
3.1 Configuring the Large Model
This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.
3.1.1 Configuring the LLM Platform (yahboom.yaml)
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlSet llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:
xxxxxxxxxxmodel_service ros__parameters language'zh' llm_platform'ollama' # Offline large model; change to 'tongyi' when using an online model regional_setting"China"Offline Ollama: set to
llm_platform: 'ollama'.Online Tongyi Qianwen: set to
llm_platform: 'tongyi'.
If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.
3.1.2 Configuring the Model Interface (large_model_interface.yaml)
Open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlAccording to the backend selected in the previous step, fill in only the corresponding group of parameters.
Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:
xxxxxxxxxxqianwen_api_key"Replace with your API-Key"qianwen_model"qwen-vl-max-latest"Replace the placeholder with your own API Key. Visual localization requires the model to understand the image content and to return the target position or coordinate information as prompted. Do not disclose real API Keys.
Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"llava" # Replace with the model name in ollama listThe model name must be consistent with the output of ollama list, and the model needs to support image input.
After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.
The visual localization inference speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed is also related to the model size and the input image.
3.2 Starting and Testing the Feature (Text Mode)
Prepare the image file:
Place an image file to be tested in the following path:
/home/jetson/yahboom_ws/src/largemodel/resources_file/visual_positioningThen name the image
test_image.jpgStart the
largemodelmain program (text mode):Open a terminal and run the following command:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.py text_chat_mode:=trueSend a text command: Open another terminal again and run the following command,
xxxxxxxxxxros2 run text_chat text_chatThen start typing the text: "Locate the position of the dinosaur in the image".
Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the
visual_positioningtool, that visual_positioning has finished executing, and that the coordinates were saved to a document.We can find this document in the path ~/yahboom_ws/src/largemodel/resources_file/visual_positioning.
3.3 Voice Interaction
Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:
xxxxxxxxxxmodel_service ros__parameters use_oline_asrFalse # True: online ASR; False: offline ASR useolinettsFalse # True: online TTS; False: offline TTSComplete the corresponding configuration according to the ASR/TTS tutorials. Place the image in the directory specified above and name it test_image.jpg, then start:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyAfter waking the device, say "Locate the position of the dinosaur in the image". Check the ASR recognized text, the visual_positioning tool call log, and the output document. Voice tool calling depends on the current program version and the model capability; if there is no tool call log, first verify in text mode.
4. Common Problems and Solutions
Problem 1: The message "Image file not found" is displayed.
Solution:
Check the path: Make sure you have placed the image file in the specified path, that it is named
test_image.jpg, and that you have permission to read the file.Image format: Make sure the image file is not damaged and is in .jpg format.