yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

06.Multimodal autonomous agent application

Multimodal Autonomous Agent Application

Multimodal Autonomous Agent Application1. Concept Introduction1.1 What Is an "Autonomous Agent"?1.2 Implementation Principles Overview 2. Code AnalysisKey Code1. Agent Core Workflow (largemodel/utils/ai_agent.py)2. Task Planning and LLM Interaction (largemodel/utils/ai_agent.py)3. Parameter Processing and Data Flow Implementation (largemodel/utils/ai_agent.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Input Mode)3.3 Voice Interaction4. Common Problems and Solutions4.1 Abnormal Agent BehaviorProblem 1: The agent falls into an infinite loop or repeatedly executes the same tool.4.2 Tool Call FailureProblem 2: The agent plans the actions correctly, but tool execution fails.

Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.


1. Concept Introduction

1.1 What Is an "Autonomous Agent"?

In the largemodel project, the multimodal autonomous agent is the most advanced form of intelligence. It no longer simply makes one response to one user instruction; instead, it is able to think, plan, and continuously call multiple tools autonomously to complete a task in order to achieve a complex goal.

The core of this feature is the agent_call tool or the tool chain manager (ToolChainManager) behind it. When the user raises a complex request that cannot be completed through a single tool call, the autonomous agent starts.

1.2 Implementation Principles Overview

The implementation of the autonomous agent in largemodel follows the industry-mainstream ReAct (Reason + Act) paradigm. Its core idea is to imitate the human problem-solving process, cycling between "thinking" and "acting".

  1. Reason: When the agent receives a complex goal, it first calls a powerful language model (LLM) to "think". It asks itself: "To accomplish this goal, what should my first step be? Which tool should I use?" The LLM's output is not the final answer, but an action plan.

  2. Act: Based on the LLM's thinking result, the agent executes the corresponding action, calling ToolsManager to run the specified tool (such as visual_positioning).

  3. Observe: The agent obtains the result of the previous action (the "observation"), for example {"result": "Found the cup, position at [120, 300, 180, 360]"}.

  4. Reason Again: The agent submits this observation result, together with the original goal, to the LLM for a second round of "thinking". It asks itself: "I have found the position of the cup; what should I do next to know its color?" The LLM may generate a new action plan, for example {"thought": "I need to analyze the image of the region where the cup is located to determine its color", "action": "seewhat", "args": {"crop_area": [120, 300, 180, 360]}}.

This Reason -> Act -> Observe loop continues until the original goal is achieved, and only then does the agent generate and output the final answer.

2. Code Analysis

Key Code

1. Agent Core Workflow (largemodel/utils/ai_agent.py)

The _execute_agent_workflow function is the agent's main execution loop; it defines the core flow of "plan -> execute".

​x
# From largemodel/utils/ai_agent.py

class AIAgent:
    # ...

    def _execute_agent_workflow(self, task_description: str) -> Dict[str, Any]:
        """
        Executes the agent workflow: Plan -> Execute.
        """
        try:
            # Step 1: task planning
            self.node.get_logger().info("AI Agent starting task planning phase")
            plan_result = self._plan_task(task_description)
            
            # ... (return early if planning fails)

            self.task_steps = plan_result["steps"]

            # Step 2: execute all steps in order
            execution_results = []
            tool_outputs = [] 

            for i, step in enumerate(self.task_steps):
                # 2.1. Before execution, process the data references in the parameters
                processed_parameters = self._process_step_parameters(step.get("parameters", {}), tool_outputs)
                step["parameters"] = processed_parameters

                # 2.2. Execute a single step
                step_result = self._execute_step(step, tool_outputs)
                execution_results.append(step_result)

                # 2.3. If the step succeeds, save its output for reference by subsequent steps
                if step_result.get("success") and step_result.get("tool_output"):
                    tool_outputs.append(step_result["tool_output"])
                else:
                    # If any step fails, abort the entire task
                    return { "success": False, "message": f"Task terminated because step '{step['description']}' failed." }
            
            # ... summarize and return the final result
            summary = self._summarize_execution(task_description, execution_results)
            return { "success": True, "message": summary, "results": execution_results }

        # ... (exception handling)

2. Task Planning and LLM Interaction (largemodel/utils/ai_agent.py)

The core of the _plan_task function is to build a sophisticated prompt and use the large model's own reasoning capability to generate a structured execution plan.

xxxxxxxxxx
# From largemodel/utils/ai_agent.py

class AIAgent:
    # ...
    def _plan_task(self, task_description: str) -> Dict[str, Any]:
        """
        Uses the large model for task planning and decomposition.
        """
        # Dynamically generate the list of available tools and their descriptions
        tool_descriptions = []
        for name, adapter in self.tools_manager.tool_chain_manager.tools.items():
            # ... (get the tool description from adapter.input_schema)
            tool_descriptions.append(f"- {name}({params}): {description}")
        available_tools_str = "\\n".join(tool_descriptions)

        # Build a highly structured planning prompt
        planning_prompt = f"""
As a professional task planning Agent, break the user task down into a series of concrete, executable JSON steps.

**# Available Tools:**
{available_tools_str}

**# Core Rules:**
1.  **Data Passing**: When a subsequent step needs to use the output of a previous step, it **must** be referenced using `{{{{steps.N.outputs.KEY}}}}` format.
    - `N` is the step ID (starting from 1).
    - `KEY` is the specific field name in the output data of the previous step.
2.  **JSON Format**: Must strictly return a JSON object.

**# User Task:**
{task_description}
"""
        
        # Call the large model for planning
        messages_to_use = [{"role": "user", "content": planning_prompt}]
        # Note that the general text inference interface is called here
        result = self.node.model_client.infer_with_text("", message=messages_to_use)
        
        # ... (parse the JSON response and return the step list)

3. Parameter Processing and Data Flow Implementation (largemodel/utils/ai_agent.py)

The _process_step_parameters function is responsible for parsing placeholders and realizing data flow between steps.

xxxxxxxxxx
# From largemodel/utils/ai_agent.py

class AIAgent:
    # ...
    def _process_step_parameters(self, parameters: Dict[str, Any], previous_outputs: List[Any]) -> Dict[str, Any]:
        """
        Parses parameter dictionary, finds and replaces all {{...}} references.
        """
        processed_params = parameters.copy()
        # Regular expression to match placeholders in the {{steps.N.outputs.KEY}} format
        pattern = re.compile(r"\\{\\{steps\\.(\\d+)\\.outputs\\.(.+?)\\}\\}")

        for key, value in processed_params.items():
            if isinstance(value, str) and pattern.search(value):
                # Use re.sub with a replacer function to handle all found placeholders
                # The replacer function looks up and returns the value from the previous_outputs list
                processed_params[key] = pattern.sub(replacer_function, value)
        
        return processed_params

Code Analysis

The AI Agent is the "central brain" of the system; it turns the high-level, sometimes even vague tasks raised by the user into a series of precise, ordered tool calls. Its implementation does not depend on any specific model platform, but is built on a general, extensible architecture.

  1. Dynamic Task Planning: The core capability of the agent lies in the _plan_task function. It does not rely on hard-coded logic, but interacts with the large model to dynamically generate a task plan.

    • Self-awareness and prompt construction: At the start of planning, the agent first checks all currently available tools and their descriptions. It then packages this tool information, the user task, and strict rules (such as the data passing format) into a highly structured planning_prompt.

    • The model as the planner: This prompt is sent to a general text large model. The model reasons based on the provided context and returns an action plan containing multiple steps in JSON format. This design is highly extensible: when tools are added or modified in the system, the agent's planning capability updates automatically without modifying the agent's own code.

  2. Tool Chain and Data Flow: Real-world tasks often require multiple tools to cooperate; for example, "take a photo and describe it" requires the output of the "take a photo" tool (the image path) as the input of the "describe" tool. The AI Agent implements this elegantly through the _process_step_parameters function.

    • Data reference placeholders: During the planning stage, the large model embeds special placeholders, such as {{steps.1.outputs.data}}, into the parameter values that need to pass data.

    • Real-time parameter replacement: In the main loop of _execute_agent_workflow, before executing each step, _process_step_parameters is called. It uses regular expressions to scan all parameters of the current step; once a placeholder is found, it locates the corresponding data in the output list of the previous steps and replaces it in real time. This mechanism is the key to automating complex tasks.

  3. Supervised Execution and Fault Tolerance: _execute_agent_workflow constitutes the agent's main execution loop. It executes each action in strict accordance with the planned step order and ensures that data is passed correctly between steps.

    • Atomic steps: Each step is treated as an independent "atomic operation". If any step fails to execute, the entire task chain is terminated immediately and an error is reported. This ensures the stability and predictability of the system and avoids continuing execution in an error state.

In summary, the general implementation of the AI Agent demonstrates an advanced software architecture: it does not solve problems directly, but builds a framework that lets an external, general reasoning engine (the large model) solve the problem. Through the two core mechanisms of "dynamic planning" and "data flow management", the agent can orchestrate a series of independent tools into a complex workflow capable of completing high-level tasks.

3. Practical Operations

3.1 Configuring the Large Model

This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.

3.1.1 Configuring the LLM Platform (yahboom.yaml)

Open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

Set llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:

xxxxxxxxxx
model_service:
  ros__parameters:
    language: 'zh'
    llm_platform: 'ollama'  # Offline large model; change to 'tongyi' when using an online model
    regional_setting: "China"
  • Offline Ollama: set to llm_platform: 'ollama'.

  • Online Tongyi Qianwen: set to llm_platform: 'tongyi'.

If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.

3.1.2 Configuring the Model Interface (large_model_interface.yaml)

Open the model interface configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

According to the backend selected in the previous step, fill in only the corresponding group of parameters.

Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:

xxxxxxxxxx
qianwen_api_key: "Replace with your API-Key"
qianwen_model: "qwen-vl-max-latest"

Replace the placeholder with your own API Key. The autonomous agent requires the model to understand the task and generate an executable tool call plan as prompted; if the task includes image analysis, the vision tool being called also requires the model to support image input. Do not disclose real API Keys.

Offline Ollama: Make sure the Ollama service has started and that a model able to follow instructions and return a structured task plan as required has been downloaded; if the agent task needs to analyze images, that model must also support image input. Then fill in the service address and the local model name:

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "llava"  # Configuration example; replace with a model in ollama list suitable for agent tasks

The model name must be consistent with the output of ollama list. llava is only a configuration example and is not guaranteed to generate the structured plan required by the agent stably; select a model according to the actual task, and confirm through subsequent tests that both planning and tool calling succeed.

After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.

The task reasoning speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the number of call steps and tools also affects the task duration.

3.2 Starting and Testing the Feature (Text Input Mode)

  1. Start the largemodel main program (text mode): Open a terminal and run the following command:

    xxxxxxxxxx
    ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=true
  2. Send a text command: Open another terminal again and run the following command,

    xxxxxxxxxx
    ros2 run text_chat text_chat

    Then start typing the text: "Based on the current environment, save the generated environment description as a txt document".

  3. Observe the result:

    In the first terminal running the main program, you will see log output indicating that the system received the text command, called the aiagent tool, and then provided a prompt to the LLM; the LLM analyzes the detailed tool call steps. For the current question, for example, the seewhat tool will be called to obtain the frame, then the frame will be provided to the LLM for parsing, and the parsed text will be saved under the ~/yahboom_ws/src/largemodel/resources_file/documents folder.


3.3 Voice Interaction

The AI Agent can hand the task obtained from speech recognition to the LLM for planning, and then call tools according to the plan. The LLM backend is selected by llm_platform; ASR/TTS use independent configurations:

xxxxxxxxxx
model_service:
  ros__parameters:
    use_oline_asr: False  # True: online ASR; False: offline ASR
    useolinetts: False    # True: online TTS; False: offline TTS

Complete the service configuration according to the ASR/TTS tutorials. Set tongyi for an online LLM and ollama for an offline LLM. Start the voice program:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking the device, say "Based on the current environment, save the generated environment description as a txt document". Check the ASR recognized text, the aiagent planning and tool execution logs, and confirm that the result file is written to ~/yahboom_ws/src/largemodel/resources_file/documents, and also confirm the speaker announcement. Complex tasks place high demands on the model's instruction-following and tool calling capabilities; if no tool execution log is seen, troubleshoot in text mode first.

4. Common Problems and Solutions

4.1 Abnormal Agent Behavior

Problem 1: The agent falls into an infinite loop or repeatedly executes the same tool.

Solution:

  1. Strengthen the prompt: Add stronger restrictions to the ReAct prompt, for example: "Do not repeat the same operation" and "If the observation result is the same twice in a row, try a different tool or end the task".

  2. Add an iteration limit: As with max_turns in the code example, set a maximum number of loops to prevent infinite loops.

  3. Switch to a more powerful LLM: The agent's logical capability depends to a large extent on the "intelligence" of the backend LLM. A more powerful model can understand the task better and make plans.

4.2 Tool Call Failure

Problem 2: The agent plans the actions correctly, but tool execution fails.

Solution:

  1. Check the individual tool: This is usually not a problem with the agent logic, but a problem with the tool being called (such as seewhat or visual_positioning) itself. Please troubleshoot according to the tutorial of the corresponding tool.

  2. Error handling: In the loop of ToolChainManager, a well-established try...except mechanism is needed to catch exceptions from tool execution and feed the error information back to the LLM as an "observation result", so that it knows the previous step failed and re-plans.

Multimodal table scanning application
Build Isaac ROS environment

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文