Multimodal Visual Understanding Application

Note: Due to performance limitations, the Raspberry Pi 5 1GB, 2GB, and 4GB versions cannot run offline. You can refer to online large models.

1. Concept Introduction

1.1 What is "Visual Understanding"?

In the largemodel project, the multimodal visual understanding feature enables robots to go beyond simply "seeing" a matrix of pixels and truly "understand" the content, objects, scenes, and relationships within an image. This is like giving robots a pair of thinking eyes.

The core tool of this feature is seewhat. When a user issues a command such as "see what's here," the system invokes this tool, triggering a series of background operations that ultimately provide the user with natural language feedback on the AI's analysis of the real-time image.

1.2 Implementation Principles

The basic principle is to feed two different types of information—images (visual information) and text (linguistic information)—into a powerful multimodal large model (such as LLaVA).

1. Image Encoding: The model first uses a visual encoder to convert the input image into a computer-interpretable numerical vector that captures features such as color, shape, and texture.

2. Text Encoding: Simultaneously, the user's question (e.g., "What's on the table?") is also converted into a text vector.

3. Cross-modal Fusion: The most critical step is to fuse the image and text vectors in a special "attention layer." Here, the model learns to "focus" on the parts of the image relevant to the question. For example, when asked about the word "table," the model will pay more attention to areas of the image that match the characteristics of a table.

4. Answer Generation: Finally, a large language model (LLM) generates a descriptive text answer based on this fused information.

Simply put, it uses text to highlight corresponding parts of the image, and then describes the highlighted parts verbally.

2. Code Analysis

Key Code

1. Tool Layer Entry (largemodel/utils/tools_manager.py)

The seewhat function in this file defines the tool's execution flow.

​x
# From largemodel/utils/tools_manager.py

class ToolsManager:
    # ...

    def seewhat(self):
        """
        Capture camera frame and analyze environment with AI model.
   
        
        :return: Dictionary with scene description and image path, or None if failed.
        """
        self.node.get_logger().info("Executing seewhat() tool")
        image_path = self.capture_frame()
        if image_path:
            # Use isolated context for image analysis. 
            analysis_text = self._get_actual_scene_description(image_path)

            # Return structured data for the tool chain. 
            return {
                "description": analysis_text,
                "image_path": image_path
            }
        else:
            # ... (Error handling)
            return None

    def _get_actual_scene_description(self, image_path, message_context=None):
        """
        Get AI-generated scene description for captured image.
        
        :param image_path: Path to captured image file.
        :return: Plain text description of scene.
        """
        try:
            # ... (Building Prompt)
            result = self.node.model_client.infer_with_image(image_path, scene_prompt, message=simple_context)
            # ... (Processing results)
            return description
        except Exception as e:
            # ...

2. Model Interface Layer (largemodel/utils/large_model_interface.py)

The infer_with_image function in this file is the unified entry point for all image understanding tasks. It is responsible for calling the specific model implementation based on the configuration.

 MiniCPM-V Visual Multimodal Large Model
Multimodal Text-Based Image Application