3. Multimodal Visual Understanding + Object Tracking and Following

1. Course Content

1. Learn to use the robot's visual understanding combined with the car's pan-tilt object tracking and following function.
2. Analyze newly appearing key source code.

2. Preparation

2.1 Content Description

The virtual machine used by default with this product starts the Dify docker environment automatically at boot, so no additional startup is needed.

⚠️ The same test instructions will not produce exactly the same reply content from the large model, and will differ slightly from the screenshots in the tutorial. If you need to enhance or reduce the diversity of the large model's replies, refer to 【AI Large Model Development】-【RAG Knowledge Base Deployment】 to modify the knowledge base and sample library.

Virtual machine terminal, start the Dify environment:

sh ~/bringup_dify.sh

3. Running the Case

3.1 Starting the Program

Open the virtual machine terminal and enter the command to start the underlying proxy + camera topic:

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

Open another virtual machine terminal and enter the command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then open another terminal and run the object tracking box program. This program will show a visual window following the object while the large model is following it:

xxxxxxxxxx
ros2 launch microros_v2_bringup super_tracker.launch.py

Then open another terminal and start the text dialogue:

xxxxxxxxxx
ros2 run multi_brains terminal_client

3.2 Test Cases

Reference test cases are given here. Users can create their own dialogue commands.

• Start xx tracking, object tracking.

3.2.1 Case 1: "Start tracking the green block"

Enter "Start tracking the green block" in the text dialogue terminal. The terminal prints information as follows:

A window titled frame will open for 3 seconds showing the robot's current view image, then close automatically:

Slowly move the object, and the pan-tilt servo will follow. Move the object forward and backward, and the car will follow based on near-large-far-small (size-based) tracking. Because it is size-based tracking, it will not be particularly sensitive.

If there is no target to track in the image or the confidence is low, the program will count down for 10 seconds and the process will automatically end, treating the task as completed:

To end the task manually, press the ENTER key in the terminal to continue the dialogue input, and enter 【Stop Task】 or 【End Tracking】:

🔧 In the current version, stopping tracking will use stop() to cancel object tracking and stop the chassis. If you see logs related to the external tracking node exiting in the terminal, it usually means the tracking task has been ended normally.

After the robot completes the task, it enters a waiting state. At this time, instructions are directly passed to the execution-layer large model, and all conversation history is always retained. We can input the "/reset" command again to let the robot end the current task cycle and start a new task cycle.

3.2.2 Case 2: "Please track the object in my hand"

⚠️ The coordinates of the tracked object obtained in this case are entirely inferred by the AI large model, so it is recommended to use a newer model for better results!

Enter "Please track the object in my hand" in the terminal. The terminal prints information as follows:

Please hold any object in your hand and put it in view, and keep it there until a tracking box appears. You can see the image captured by the car for 3 seconds in the process:

After the tracking box appears, slowly move the object, and the pan-tilt servo will follow. Move the object forward and backward, and the car will follow based on size-based tracking. Because it is size-based tracking, it will not be particularly sensitive.

If there is no target to track in the image, the program will count down for 10 seconds, print a 5-second countdown in the terminal, and the process will automatically end, treating the task as completed:

To end the task manually, you can input "stop" in the terminal, press the ENTER key to continue the dialogue input, and enter 【Stop】 or 【End Tracking】:

After the robot completes the task, it enters a waiting state. At this time, instructions are directly passed to the execution-layer large model, and all conversation history is always retained. We can input the "end current task" command again to let the robot end the current task cycle and start a new task cycle.

4. Source Code Analysis

In the current version, the "track target in frame" pipeline is a Dify three-layer workflow: the decision layer plans the tracking steps, the execution layer first outputs seewhat() to obtain the image, the box-selection layer outputs bbox based on the image, and the execution layer then outputs track_kcf(x1,y1,x2,y2).

Current relevant source code paths:

xxxxxxxxxx
~/microros_ws/src/multi_brains/multi_brains/llm_agent_node.py
~/microros_ws/src/multi_brains/multi_brains/controllers/kcf_tracker.py

Core flow:

xxxxxxxxxx
Start tracking the target
  -> seewhat() obtains the current frame
  -> Box-selection layer outputs bbox_OUT
  -> Execution layer outputs track_kcf(x1,y1,x2,y2)
  -> KcfTracker calls /start_track
  -> External tracking node publishes /tracking_bbox
  -> llm_agent_node controls the pan-tilt and chassis to follow

To stop tracking, input "stop tracking" or "park" in the terminal, and the execution layer will output stop(), which cancels KCF tracking and stops the chassis.