9. Intent Inference + Multimodal Visual Understanding + SLAM Navigation

1. Course Content

Note: This chapter requires completing the map mapping file configuration according to the 【Multimodal Visual Understanding + SLAM Navigation】 chapter.

1. Learn to use the RAG knowledge base to train personal intent understanding.

Course review:

If you need to enhance or reduce the diversity of the large model's replies, refer to 【AI Large Model Development】-【RAG Knowledge Base Deployment】 to modify the knowledge base and sample library.

Using the RAG knowledge base can expand the knowledge scope of the large model. This chapter explains how to extend the RAG knowledge base so that the large model can understand personal intents.

Special note!!!:

1. The personal intent understanding function varies by user, and large model replies are divergent, so the actual debugging results are not exactly the same.
2. Extending the personal intent understanding function must comply with social morality, public order, and laws and regulations. The technical service is not responsible for the final debugging results of this chapter or their impact.

2. Preparation

2.1 Content Description

Add a custom intent inference knowledge base. Since personal intents are divergent, there is no unified content, and the actual debugging results are not exactly the same.

The virtual machine used by default with this product starts the Dify Docker environment automatically at boot, so no additional startup is usually needed.

Open the virtual machine terminal, start the Dify environment:

sh ~/bringup_dify.sh

3. Configuration Documentation

3.1 Preparing the xlsx Document

Open the 【Intent Mapping】 folder in this chapter's folder, and select the corresponding camera 【Intent Mapping_xx.xlsx】 file according to your car configuration.

(You can add multiple files by yourself for more customized information.)

【Intent Mapping】: stores personal intents and the operations the user expects the large model to control the robot to perform.

3.2 Uploading to the RAG Knowledge Base

Some intent inference is already configured at the factory. The following steps are for adding your own new intents.

1. Access the IP of the vehicle and enter the Dify backend.

2. Click 【Knowledge】 and click 【Sample Library】.

3. Click "Add File".

4. Upload your intent inference knowledge base.

5. Click 【Next】, then click 【Save and Process】.

6. Wait for the processing to succeed, refresh the knowledge base, and you can see that the intent mapping file has been added.

7. Return to the 【Studio】 interface and select the application corresponding to your camera model. Here, 【microros-v2-zh】 is used as an example.

8. Select 【Task Routing】, and add your own defined intent expressions in the complex task category, for example, "I'm hungry", "I'm thirsty", and "I'm a little sleepy".

9. Select the 【Decision-Layer Sample Library】 knowledge retrieval node, and confirm that the newly added intent mapping knowledge base has been selected. If it has not been selected, click the Add button on the right to add the just-uploaded intent mapping knowledge base to this node.

10. You can click "Preview" in the upper-right corner, for example, send "I'm hungry" and observe whether it enters the complex task branch, and confirm whether the decision layer can output high-level steps consistent with the intent mapping.

Finally, click 【Publish】 in the upper-right corner to re-publish the corresponding application.

4. Running the Case

4.1 Starting the Program

Open the virtual machine terminal, start the Dify environment (if it is already started, no need to start it again):

xxxxxxxxxx
sh ~/bringup_dify.sh

Open the virtual machine terminal and enter the command to start the underlying proxy, camera topic, and Nav2 navigation:

xxxxxxxxxx
ros2 launch microros_v2_bringup nav2.launch.py

Open another virtual machine terminal and enter the command to start the large model agent node:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then follow the process of starting the navigation function for initial localization. After opening the RViz2 visualization interface, click 2D Pose Estimate in the toolbar at the top, roughly mark the robot's position and orientation on the map. After initial localization, the preparation is complete.

Then open another terminal and start the text dialogue client:

xxxxxxxxxx
ros2 run multi_brains terminal_client

Use the Chinese input method:

• Click the input method icon in the upper-right corner of the virtual machine to switch to Chinese pinyin. You can also press the Shift shortcut key in the terminal to switch.

4.2 Test Cases

Reference test cases are given here. Users can write their own dialogue commands:

• I feel a little thirsty.

4.2.1 Case 1

Enter "I feel a little thirsty." in the terminal. If the intent mapping knowledge base has a high-level task configured for "thirsty", for example, "go to the tea room/kitchen to get water or observe the drink location",

Then the execution-layer large model will execute according to the decision-layer steps:

After the robot completes the task, it enters a waiting state, and the conversation history will continue to be retained. You can input reset in the terminal to reset the conversation context, or input a new natural language instruction to let the latest instruction take over the current task.

5. Source Code Analysis

The key new point in this chapter is not in the new ROS action source code, but in the Dify intent mapping knowledge base and task routing. The ROS side still reuses the navigation, visual understanding, and feedback loop capabilities covered in Chapter 6.

Current relevant source code and configuration paths:

xxxxxxxxxx
~/microros_ws/src/multi_brains/multi_brains/llm_agent_node.py
~/microros_ws/src/multi_brains/multi_brains/services/dify_coordinator.py
~/microros_ws/src/multi_brains/multi_brains/controllers/navigation_controller.py
~/microros_ws/src/multi_brains/config/llm_agent_config.yaml
~/microros_ws/src/multi_brains/config/map_mapping.yaml

5.1 Where the New Capability Is

"Intent inference" does not add new action functions such as hunger() or thirsty(), but adds a layer of personal semantic mapping to Dify. For example, when the user says "I'm a little thirsty", the knowledge base supplements it as "need to go to the tea room/kitchen to observe whether there is water or a drink", and then the decision layer decomposes this goal into steps such as navigation, observation, and return.

Three Dify configuration points need attention in the workflow:

• Task routing: classifies personal expressions such as "I'm hungry", "I'm thirsty", and "I'm a little sleepy" into the complex task branch.
• Decision-layer sample library/knowledge retrieval node: selects the uploaded intent mapping knowledge base so that the decision layer can retrieve the task description corresponding to personal intents.
• Conversation variable map_mapping: maps location names such as "tea room" and "office" to navigation target IDs such as A and B.

If any of these three is missing, the robot will not be able to reliably derive executable actions from "implicit intents".

5.2 Core Flow

xxxxxxxxxx
User inputs "I'm currently in the office area, and I feel a little thirsty"
  -> Dify task routing determines it as a complex task/intent inference
  -> Knowledge retrieval node recalls the "thirsty"-related task description from the intent mapping knowledge base
  -> Decision layer generates high-level steps combined with map_mapping
  -> Execution layer outputs only one action JSON per round
  -> llm_agent_node.py parses the action field and distributes it through action_registry
  -> NavigationController executes get_current_pose() / navigation(target)
  -> seewhat() uploads the current frame to Dify as robot feedback
  -> Dify continues to the next step based on navigation or image feedback
  -> finishtask() ends the current task cycle

The feedback loop of llm_agent_node.py reads the execution result and then uses "robot feedback: executing {actions} completed/failed" to continue requesting Dify. Complex tasks do not execute multiple actions at once, but instead follow "decide one round, execute one action, and after feedback, decide the next round".