Welcome to Jetson Orin NANO Developer Kit repository
06.Multimodal autonomous agent application
Multimodal Autonomous Agent Application
Multimodal Autonomous Agent Application1. Concept Introduction1.1 What Is an "Autonomous Agent"?1.2 Implementation Principles Overview 2. Code AnalysisKey Code1. Agent Core Workflow (largemodel/utils/ai_agent.py)2. Task Planning and LLM Interaction (largemodel/utils/ai_agent.py)3. Parameter Processing and Data Flow Implementation (largemodel/utils/ai_agent.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Input Mode)3.3 Voice Interaction4. Common Problems and Solutions4.1 Abnormal Agent BehaviorProblem 1: The agent falls into an infinite loop or repeatedly executes the same tool.4.2 Tool Call FailureProblem 2: The agent plans the actions correctly, but tool execution fails.
Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.
1. Concept Introduction
1.1 What Is an "Autonomous Agent"?
In the largemodel project, the multimodal autonomous agent is the most advanced form of intelligence. It no longer simply makes one response to one user instruction; instead, it is able to think, plan, and continuously call multiple tools autonomously to complete a task in order to achieve a complex goal.
The core of this feature is the agent_call tool or the tool chain manager (ToolChainManager) behind it. When the user raises a complex request that cannot be completed through a single tool call, the autonomous agent starts.
1.2 Implementation Principles Overview
The implementation of the autonomous agent in largemodel follows the industry-mainstream ReAct (Reason + Act) paradigm. Its core idea is to imitate the human problem-solving process, cycling between "thinking" and "acting".
Reason: When the agent receives a complex goal, it first calls a powerful language model (LLM) to "think". It asks itself: "To accomplish this goal, what should my first step be? Which tool should I use?" The LLM's output is not the final answer, but an action plan.
Act: Based on the LLM's thinking result, the agent executes the corresponding action, calling
ToolsManagerto run the specified tool (such asvisual_positioning).Observe: The agent obtains the result of the previous action (the "observation"), for example
{"result": "Found the cup, position at [120, 300, 180, 360]"}.Reason Again: The agent submits this observation result, together with the original goal, to the LLM for a second round of "thinking". It asks itself: "I have found the position of the cup; what should I do next to know its color?" The LLM may generate a new action plan, for example
{"thought": "I need to analyze the image of the region where the cup is located to determine its color", "action": "seewhat", "args": {"crop_area": [120, 300, 180, 360]}}.
This Reason -> Act -> Observe loop continues until the original goal is achieved, and only then does the agent generate and output the final answer.
2. Code Analysis
Key Code
1. Agent Core Workflow (largemodel/utils/ai_agent.py)
The _execute_agent_workflow function is the agent's main execution loop; it defines the core flow of "plan -> execute".
x# From largemodel/utils/ai_agent.py
class AIAgent: # ...
def _execute_agent_workflow(self, task_description: str) -> Dict[str, Any]: """ Executes the agent workflow: Plan -> Execute. """ try: # Step 1: task planning self.node.get_logger().info("AI Agent starting task planning phase") plan_result = self._plan_task(task_description) # ... (return early if planning fails)
self.task_steps = plan_result["steps"]
# Step 2: execute all steps in order execution_results = [] tool_outputs = []
for i, step in enumerate(self.task_steps): # 2.1. Before execution, process the data references in the parameters processed_parameters = self._process_step_parameters(step.get("parameters", {}), tool_outputs) step["parameters"] = processed_parameters
# 2.2. Execute a single step step_result = self._execute_step(step, tool_outputs) execution_results.append(step_result)
# 2.3. If the step succeeds, save its output for reference by subsequent steps if step_result.get("success") and step_result.get("tool_output"): tool_outputs.append(step_result["tool_output"]) else: # If any step fails, abort the entire task return { "success": False, "message": f"Task terminated because step '{step['description']}' failed." } # ... summarize and return the final result summary = self._summarize_execution(task_description, execution_results) return { "success": True, "message": summary, "results": execution_results }
# ... (exception handling)2. Task Planning and LLM Interaction (largemodel/utils/ai_agent.py)
The core of the _plan_task function is to build a sophisticated prompt and use the large model's own reasoning capability to generate a structured execution plan.
xxxxxxxxxx# From largemodel/utils/ai_agent.py
class AIAgent: # ... def _plan_task(self, task_description: str) -> Dict[str, Any]: """ Uses the large model for task planning and decomposition. """ # Dynamically generate the list of available tools and their descriptions tool_descriptions = [] for name, adapter in self.tools_manager.tool_chain_manager.tools.items(): # ... (get the tool description from adapter.input_schema) tool_descriptions.append(f"- {name}({params}): {description}") available_tools_str = "\\n".join(tool_descriptions)
# Build a highly structured planning prompt planning_prompt = f"""As a professional task planning Agent, break the user task down into a series of concrete, executable JSON steps.
**# Available Tools:**{available_tools_str}
**# Core Rules:**1. **Data Passing**: When a subsequent step needs to use the output of a previous step, it **must** be referenced using `{{{{steps.N.outputs.KEY}}}}` format. - `N` is the step ID (starting from 1). - `KEY` is the specific field name in the output data of the previous step.2. **JSON Format**: Must strictly return a JSON object.
**# User Task:**{task_description}""" # Call the large model for planning messages_to_use = [{"role": "user", "content": planning_prompt}] # Note that the general text inference interface is called here result = self.node.model_client.infer_with_text("", message=messages_to_use) # ... (parse the JSON response and return the step list)3. Parameter Processing and Data Flow Implementation (largemodel/utils/ai_agent.py)
The _process_step_parameters function is responsible for parsing placeholders and realizing data flow between steps.
xxxxxxxxxx# From largemodel/utils/ai_agent.py
class AIAgent: # ... def _process_step_parameters(self, parameters: Dict[str, Any], previous_outputs: List[Any]) -> Dict[str, Any]: """ Parses parameter dictionary, finds and replaces all {{...}} references. """ processed_params = parameters.copy() # Regular expression to match placeholders in the {{steps.N.outputs.KEY}} format pattern = re.compile(r"\\{\\{steps\\.(\\d+)\\.outputs\\.(.+?)\\}\\}")
for key, value in processed_params.items(): if isinstance(value, str) and pattern.search(value): # Use re.sub with a replacer function to handle all found placeholders # The replacer function looks up and returns the value from the previous_outputs list processed_params[key] = pattern.sub(replacer_function, value) return processed_paramsCode Analysis
The AI Agent is the "central brain" of the system; it turns the high-level, sometimes even vague tasks raised by the user into a series of precise, ordered tool calls. Its implementation does not depend on any specific model platform, but is built on a general, extensible architecture.
Dynamic Task Planning: The core capability of the agent lies in the
_plan_taskfunction. It does not rely on hard-coded logic, but interacts with the large model to dynamically generate a task plan.Self-awareness and prompt construction: At the start of planning, the agent first checks all currently available tools and their descriptions. It then packages this tool information, the user task, and strict rules (such as the data passing format) into a highly structured
planning_prompt.The model as the planner: This prompt is sent to a general text large model. The model reasons based on the provided context and returns an action plan containing multiple steps in JSON format. This design is highly extensible: when tools are added or modified in the system, the agent's planning capability updates automatically without modifying the agent's own code.
Tool Chain and Data Flow: Real-world tasks often require multiple tools to cooperate; for example, "take a photo and describe it" requires the output of the "take a photo" tool (the image path) as the input of the "describe" tool. The AI Agent implements this elegantly through the
_process_step_parametersfunction.Data reference placeholders: During the planning stage, the large model embeds special placeholders, such as
{{steps.1.outputs.data}}, into the parameter values that need to pass data.Real-time parameter replacement: In the main loop of
_execute_agent_workflow, before executing each step,_process_step_parametersis called. It uses regular expressions to scan all parameters of the current step; once a placeholder is found, it locates the corresponding data in the output list of the previous steps and replaces it in real time. This mechanism is the key to automating complex tasks.
Supervised Execution and Fault Tolerance:
_execute_agent_workflowconstitutes the agent's main execution loop. It executes each action in strict accordance with the planned step order and ensures that data is passed correctly between steps.Atomic steps: Each step is treated as an independent "atomic operation". If any step fails to execute, the entire task chain is terminated immediately and an error is reported. This ensures the stability and predictability of the system and avoids continuing execution in an error state.
In summary, the general implementation of the AI Agent demonstrates an advanced software architecture: it does not solve problems directly, but builds a framework that lets an external, general reasoning engine (the large model) solve the problem. Through the two core mechanisms of "dynamic planning" and "data flow management", the agent can orchestrate a series of independent tools into a complex workflow capable of completing high-level tasks.
3. Practical Operations
3.1 Configuring the Large Model
This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.
3.1.1 Configuring the LLM Platform (yahboom.yaml)
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlSet llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:
xxxxxxxxxxmodel_service ros__parameters language'zh' llm_platform'ollama' # Offline large model; change to 'tongyi' when using an online model regional_setting"China"Offline Ollama: set to
llm_platform: 'ollama'.Online Tongyi Qianwen: set to
llm_platform: 'tongyi'.
If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.
3.1.2 Configuring the Model Interface (large_model_interface.yaml)
Open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlAccording to the backend selected in the previous step, fill in only the corresponding group of parameters.
Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:
xxxxxxxxxxqianwen_api_key"Replace with your API-Key"qianwen_model"qwen-vl-max-latest"Replace the placeholder with your own API Key. The autonomous agent requires the model to understand the task and generate an executable tool call plan as prompted; if the task includes image analysis, the vision tool being called also requires the model to support image input. Do not disclose real API Keys.
Offline Ollama: Make sure the Ollama service has started and that a model able to follow instructions and return a structured task plan as required has been downloaded; if the agent task needs to analyze images, that model must also support image input. Then fill in the service address and the local model name:
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"llava" # Configuration example; replace with a model in ollama list suitable for agent tasksThe model name must be consistent with the output of ollama list. llava is only a configuration example and is not guaranteed to generate the structured plan required by the agent stably; select a model according to the actual task, and confirm through subsequent tests that both planning and tool calling succeed.
After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.
The task reasoning speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the number of call steps and tools also affects the task duration.
3.2 Starting and Testing the Feature (Text Input Mode)
Start the
largemodelmain program (text mode): Open a terminal and run the following command:xxxxxxxxxxros2 launch largemodel largemodel_control.launch.py text_chat_mode:=trueSend a text command: Open another terminal again and run the following command,
xxxxxxxxxxros2 run text_chat text_chatThen start typing the text: "Based on the current environment, save the generated environment description as a txt document".
Observe the result:
In the first terminal running the main program, you will see log output indicating that the system received the text command, called the
aiagenttool, and then provided a prompt to the LLM; the LLM analyzes the detailed tool call steps. For the current question, for example, the seewhat tool will be called to obtain the frame, then the frame will be provided to the LLM for parsing, and the parsed text will be saved under the ~/yahboom_ws/src/largemodel/resources_file/documents folder.
3.3 Voice Interaction
The AI Agent can hand the task obtained from speech recognition to the LLM for planning, and then call tools according to the plan. The LLM backend is selected by llm_platform; ASR/TTS use independent configurations:
xxxxxxxxxxmodel_service ros__parameters use_oline_asrFalse # True: online ASR; False: offline ASR useolinettsFalse # True: online TTS; False: offline TTSComplete the service configuration according to the ASR/TTS tutorials. Set tongyi for an online LLM and ollama for an offline LLM. Start the voice program:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyAfter waking the device, say "Based on the current environment, save the generated environment description as a txt document". Check the ASR recognized text, the aiagent planning and tool execution logs, and confirm that the result file is written to ~/yahboom_ws/src/largemodel/resources_file/documents, and also confirm the speaker announcement. Complex tasks place high demands on the model's instruction-following and tool calling capabilities; if no tool execution log is seen, troubleshoot in text mode first.
4. Common Problems and Solutions
4.1 Abnormal Agent Behavior
Problem 1: The agent falls into an infinite loop or repeatedly executes the same tool.
Solution:
Strengthen the prompt: Add stronger restrictions to the ReAct prompt, for example: "Do not repeat the same operation" and "If the observation result is the same twice in a row, try a different tool or end the task".
Add an iteration limit: As with
max_turnsin the code example, set a maximum number of loops to prevent infinite loops.Switch to a more powerful LLM: The agent's logical capability depends to a large extent on the "intelligence" of the backend LLM. A more powerful model can understand the task better and make plans.
4.2 Tool Call Failure
Problem 2: The agent plans the actions correctly, but tool execution fails.
Solution:
Check the individual tool: This is usually not a problem with the agent logic, but a problem with the tool being called (such as
seewhatorvisual_positioning) itself. Please troubleshoot according to the tutorial of the corresponding tool.Error handling: In the loop of
ToolChainManager, a well-establishedtry...exceptmechanism is needed to catch exceptions from tool execution and feed the error information back to the LLM as an "observation result", so that it knows the previous step failed and re-plans.