5.Multimodal autonomous proxy application

1. Concept Introduction

1.1 What is an "Autonomous Agent"?

In the largemodel project, multimodal autonomous agents represent the most advanced form of intelligence. Rather than simply responding to a user's command, they are capable of autonomously thinking, planning, and sequentially invoking multiple tools to achieve a complex goal.

The core of this functionality is the **agent_call** tool or its underlying **ToolChainManager. When a user issues a complex request that cannot be accomplished with a single tool call, the autonomous agent is activated.

1.2 Implementation Principles

The autonomous agent implementation in largemodel follows the industry-leading ReAct (Reason + Act) paradigm. Its core concept is to mimic the human problem-solving process, cycling between "thinking" and "acting."

1. Reason: When the agent receives a complex goal, it first "reasons" by invoking a powerful language model (LLM). It asks itself, "What should I do first to achieve this goal? Which tool should I use?" The LLM's output isn't a final answer, but rather an action plan.
2. Act: Based on the LLM's reasoning, the agent takes the appropriate action—calling the ToolsManager to run a specified tool (such as visual_positioning).
3. Observe: The agent obtains the result of the previous action (the "observation"), for example, {"result": "The cup was found at [120, 300, 180, 360]"}.
4. Rethink: The agent submits this observation, along with the original goal, to the LLM for a second round of "reflection." It asks itself, "I've found the location of the cup. What should I do next to learn its color?" The LLM might generate a new action plan, such as {"thought": "I need to analyze the image of the area where the cup is located to determine its color", "action": "seewhat", "args": {"crop_area": ​​[120, 300, 180, 360]}}.

This think -> act -> observe cycle continues until the initial goal is achieved, at which point the agent generates and outputs the final answer.

2. Code Analysis

Key Code

1. Agent Core Workflow (largemodel/utils/ai_agent.py)

The _execute_agent_workflow function is the agent's main execution loop, defining the core "plan -> execute" process.

​x
# From largemodel/utils/ai_agent.py
​
class AIAgent:
    # ...
​
    def _execute_agent_workflow(self, task_description: str) -> Dict[str, Any]:
        """
        Executes the agent workflow: Plan -> Execute. 
        """
        try:
            # Step 1: Task Planning
            self.node.get_logger().info("AI Agent starting task planning phase")
            plan_result = self._plan_task(task_description)
            
            # ... (Return early if planning fails)
​
            self.task_steps = plan_result["steps"]
​
            # Step 2: Execute all steps in order
            execution_results = []
            tool_outputs = [] 
​
            for i, step in enumerate(self.task_steps):
                # 2.1. Processing data references in parameters before execution
                processed_parameters = self._process_step_parameters(step.get("parameters", {}), tool_outputs)
                step["parameters"] = processed_parameters
​
                # 2.2. Executing a single step
                step_result = self._execute_step(step, tool_outputs)
                execution_results.append(step_result)
​
                # 2.3. If the step succeeds, save its output for reference in subsequent steps
                if step_result.get("success") and step_result.get("tool_output"):
                    tool_outputs.append(step_result["tool_output"])
                else:
                    # If any step fails, abort the entire task
                    return { "success": False, "message": f"Task terminated because step '{step['description']}' failed." }
            
            # ... summarize and return the final result
            summary = self._summarize_execution(task_description, execution_results)
            return { "success": True, "message": summary, "results": execution_results }
​
        # ... (Exception handling)

2. Interaction between Task Planning and LLM (largemodel/utils/ai_agent.py)

_plan_taskThe core of the function is to build a precise prompt, using the reasoning capabilities of the large model to generate a structured execution plan.

xxxxxxxxxx
# From largemodel/utils/ai_agent.py
​
class AIAgent:
    # ...
    def _plan_task(self, task_description: str) -> Dict[str, Any]:
        """
        Uses the large model for task planning and decomposition. 
        """
        # Dynamically generate a list of available tools and their descriptions
        tool_descriptions = []
        for name, adapter in self.tools_manager.tool_chain_manager.tools.items():
            # ... (Get the tool description from adapter.input_schema)
            tool_descriptions.append(f"- {name}({params}): {description}")
        available_tools_str = "\\n".join(tool_descriptions)
​
        # Build a highly structured planning prompt
        planning_prompt = f"""
As a professional task planning agent, please break down user tasks into a series of specific, executable JSON steps.
​
**# Available Tools:**
{available_tools_str}
​
**# Core Rules:**
1. **Data Transfer**: When a subsequent step requires the output of a previous step, it must be referenced using the `{{{{steps.N.outputs.KEY}}}}` format.
- `N` is the step ID (starting at 1).
- `KEY` is the specific field name in the output data of the previous step.
2. **JSON Format**: Must strictly return a JSON object.
​
**# User Tasks:**
{task_description}
"""
        
        #Call the large model for planning
        messages_to_use = [{"role": "user", "content": planning_prompt}]
        # Note that the general text reasoning interface is called here
        result = self.node.model_client.infer_with_text("", message=messages_to_use)
        
        # ... (parse the JSON response and return a list of steps)

3. Parameter processing and data flow implementation (largemodel/utils/ai_agent.py)

_process_step_parametersThe function is responsible for parsing placeholders and realizing data flow between steps.

xxxxxxxxxx
# From largemodel/utils/ai_agent.py
​
class AIAgent:
    # ...
    def _process_step_parameters(self, parameters: Dict[str, Any], previous_outputs: List[Any]) -> Dict[str, Any]:
        """
        Parses parameter dictionary, finds and replaces all {{...}} references.
        """
        processed_params = parameters.copy()
        # Regular expression used to match placeholders in the format {{steps.N.outputs.KEY}}
        pattern = re.compile(r"\\{\\{steps\\.(\\d+)\\.outputs\\.(.+?)\\}\\}")
​
        for key, value in processed_params.items():
            if isinstance(value, str) and pattern.search(value):
                # Use re.sub and a replacement function to process all found placeholders
                # The replacement function will look up and return the value from the previous_outputs list
                processed_params[key] = pattern.sub(replacer_function, value)
        
        return processed_params

Code Analysis

The AI ​​Agent is the "brain" of the system, translating high-level, sometimes ambiguous, tasks posed by the user into a precise, ordered series of tool calls. Its implementation is independent of any specific model platform and built on a general, extensible architecture.

1. Dynamic Task Planning: The Agent's core capability lies in the _plan_task function. Rather than relying on hard-coded logic, it dynamically generates task plans by interacting with a larger model.

   • Self-Awareness and Prompt Construction: At the beginning of planning, the Agent first examines all available tools and their descriptions. It then packages this tool information, the user's task, and strict rules (such as data transfer format) into a highly structured planning_prompt.
   • Model as Planner: This prompt is fed into a general text-based model. The model reasoned based on the provided context and returned a multi-step action plan in JSON format. This design is highly scalable: as tools are added or modified in the system, the Agent's planning capabilities are automatically updated without requiring code modifications.

2. Toolchain and Data Flow: Real-world tasks often require the collaboration of multiple tools. For example, "take a picture and describe" requires the output (image