5. Multimodal Visual Understanding + Visual Line Following

1. Course Content

1. Learn to use the robot's visual understanding combined with the line-following autonomous driving function.
2. Analyze newly appearing key source code.

2. Preparation

2.1 Content Description

The virtual machine used by default with this product starts the Dify Docker environment automatically at boot, so no additional startup is usually needed.

⚠️ The same test instructions will not produce exactly the same reply content from the large model, and will differ slightly from the screenshots in the tutorial. If you need to enhance or reduce the diversity of the large model's replies, refer to 【AI Large Model Development】-【RAG Knowledge Base Deployment】 to modify the knowledge base and sample library.

Visual line following depends on color HSV calibration. Before starting this case, it is recommended to complete the calibration of the four colors red, green, blue, and yellow according to the color calibration process in 【AI Large Model Preparation】, and confirm that the color_hsv.txt file has been updated.

Virtual machine terminal, start the Dify environment:

sh ~/bringup_dify.sh

3. Running the Case

3.1 Starting the Program

Open the virtual machine terminal and enter the command to start the underlying proxy + camera topic:

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

Open another virtual machine terminal and enter the command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then open another terminal and start the text dialogue:

xxxxxxxxxx
ros2 run multi_brains terminal_client

3.2 Test Cases

Reference test cases are given here. Users can create their own dialogue commands:

• Start following the red line
• Start following the green line
• Start following the blue line
• Start following the yellow line

The colors include red, green, blue, and yellow. If the user does not specify a color, the execution layer will output follow_line(red) by default.

3.2.1 Case: "Start following the red line"

After the program starts, it will first adjust the camera to the line-following angle to look at the line on the ground.

Enter "Start following the red line" in the text dialogue terminal and press Enter:

The execution layer will convert the text instruction into follow_line(green). The robot will read the green HSV calibration values, recognize the green line in the lower half of the frame, and continuously publish chassis velocity to control the car to walk along the line.

During line following, if an obstacle is encountered, the car stops moving and the buzzer starts beeping. After the obstacle is removed, the program will continue trying to recognize the line and resume line following.

If the corresponding color line is not recognized in the frame for a long time, the program will count down 10 seconds and automatically stop, treating the current line-following task as ended.

To end the task manually, you can press the ENTER key in the terminal to continue the dialogue input:

• Stop line following
• Stop task
• Park

After the robot completes the task, it enters a waiting state. At this time, instructions will continue to be passed to the execution-layer large model, and the conversation history will be retained. You can input the "/reset" command again to let the robot end the current task cycle and start a new task cycle.

4. Source Code Analysis

In the current version, visual line following is implemented directly by FollowLineController inside the multi_brains package, and no longer starts the old-version line-following subprocess.

Current relevant source code paths:

xxxxxxxxxx
~/microros_ws/src/multi_brains/multi_brains/llm_agent_node.py
~/microros_ws/src/multi_brains/multi_brains/controllers/follow_line_controller.py
~/microros_ws/src/multi_brains/config/llm_agent_config.yaml
~/microros_ws/src/multi_brains/config/color_hsv.txt

Execution layer actions:

xxxxxxxxxx
follow_line(red)
follow_line(green)
follow_line(blue)
follow_line(yellow)

Core flow:

xxxxxxxxxx
User inputs "start following the green line"
  -> Execution layer outputs follow_line(green)
  -> llm_agent_node adjusts the pan-tilt to the line-following angle
  -> FollowLineController reads the HSV calibration values from color_hsv.txt
  -> Detects the corresponding color line in the lower half of the frame based on /camera/image_raw
  -> Publishes /cmd_vel to control the chassis to walk along the line
  -> Detects obstacles ahead via /scan, stops and turns on the buzzer if necessary
  -> Stops when the line is lost, a timeout occurs, an obstacle is detected, or a new instruction is received

The first four non-empty lines of color_hsv.txt correspond to red, green, blue, and yellow in order. Parameters such as line-following speed, PID parameters, line-loss timeout, and obstacle distance threshold are located in the FOLLOW_LINE configuration section of llm_agent_config.yaml.

If the user does not specify a color, the execution layer outputs follow_line(red) by default. If you need to adjust the color recognition effect, first update config/color_hsv.txt according to the color calibration process in 【AI Large Model Preparation】.