8. Multimodal Visual Understanding + SLAM Navigation + Object Following

1. Course Content

1. Learn to use compound function cases such as the robot's visual understanding, SLAM navigation, and object following.
2. Learn the new key source code appearing in the tutorial.
3. Note: This chapter requires completing the map mapping file configuration according to the previous chapter, Multimodal Visual Understanding + SLAM Navigation.

2. Preparation

2.1 Content Description

The virtual machine used by default with this product starts the Dify Docker environment automatically at boot, so no additional startup is usually needed.

⚠️ The same test instructions will not produce exactly the same reply content from the large model, and will differ slightly from the screenshots in the tutorial. If you need to enhance or reduce the diversity of the large model's replies, refer to 【AI Large Model Development】-【RAG Knowledge Base Deployment】 to modify the knowledge base and sample library.

Open the virtual machine terminal, start the Dify environment:

sh ~/bringup_dify.sh

3. Running the Case

3.1 Starting the Program

Open the virtual machine terminal and enter the command to start the underlying proxy + camera topic + navigation:

xxxxxxxxxx
ros2 launch microros_v2_bringup nav2.launch.py

Then open another terminal and run the object tracking box program. This program will show a visual window following the object while the large model is following it:

xxxxxxxxxx
ros2 launch microros_v2_bringup super_tracker.launch.py

Open another virtual machine terminal and enter the command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then follow the process of starting the navigation function for initial localization. The RViz2 visualization interface will open. Click 2D Pose Estimate in the toolbar at the top to enter the selection state, roughly mark the robot's position and orientation on the map. After initial localization, the preparation is complete.

Then open another terminal and start:

xxxxxxxxxx
ros2 run multi_brains terminal_client

3.2 Test Cases

Reference test cases are given here. Users can create their own dialogue commands.

• Navigate to the office and observe whether there is a green block. If there is a green block, start following; if not, return to the starting point.

3.2.1 Case 1

Enter "Navigate to the office and observe whether there is a green block. If there is a green block, start following; if not, return to the starting point." in the terminal. The terminal prints information as follows:

Then the execution-layer large model will execute according to the decision-layer steps:

After the box is successfully selected, following begins on the frame. Slowly move the object, and the pan-tilt servo will follow. Move the object forward and backward, and the car will follow based on near-large-far-small (size-based) tracking. Because it is size-based tracking, it will not be particularly sensitive.

If there is no target to track in the image or the confidence is low, the program will count down for 10 seconds and the process will automatically end, treating the task as completed:

After the robot completes the task, it enters a waiting state. At this time, instructions will continue to be passed to the execution-layer large model, and the conversation history will be retained. You can continue the dialogue in the terminal and input the "/reset" command to let the robot end the current task cycle and start a new task cycle.

4. Source Code Analysis

This compound case chains together chassis motion, current position recording, SLAM navigation, visual understanding, box-selection layer bbox, and KCF object following. The current version no longer uses the old-version action server file or the depth camera following script; instead, the Dify box-selection layer outputs the target box, and KcfTracker calls the external tracking service.

Current relevant source code paths:

xxxxxxxxxx
~/microros_ws/src/multi_brains/multi_brains/llm_agent_node.py
~/microros_ws/src/multi_brains/multi_brains/controllers/navigation_controller.py
~/microros_ws/src/multi_brains/multi_brains/controllers/kcf_tracker.py

Possible action sequence:

xxxxxxxxxx
get_current_pose()
navigation(target)
seewhat()
track_kcf(x1,y1,x2,y2)
navigation(zero)
finishtask()

Key flow of the tracking part:

xxxxxxxxxx
seewhat() obtains the current frame
  -> Box-selection layer outputs bbox_OUT based on the image and target description
  -> Execution layer reads bbox_OUT and outputs track_kcf(x1,y1,x2,y2)
  -> KcfTracker calls /start_track
  -> External tracking node publishes /tracking_bbox
  -> Pan-tilt keeps the target centered, chassis follows based on the target box size and offset

If the box-selection layer returns found=false, or the bbox is missing, out of bounds, or not four integers, the execution layer should output finishtask() and explain that the target cannot be confirmed, and must not guess coordinates.