Unmanned driving integrated AI large model

1. Course Content

*Note: This program runs on a sandbox map; you will need to adjust the program for your personal map.*

This tutorial uses the factory-installed road sign and lane recognition models, primarily implementing autonomous driving road sign detection + map navigation + multimodal large model for intelligent control.

2. Course Prerequisites

• After completing the "Road Network Planning" chapter, save the road network navigation map and the road network map showing the car's trajectory.
• After completing the "AI Large Model Fundamentals" chapter, configure the large model's api_key and app_id.
• Obtain the pose data of the points the car needs to reach and store it in the large model's map_mapping.yaml file.
• Modify to road network navigation communication mode.

3. Configuration File Path

• Road Network Navigation Communication Mode

Open ~/.bashrc, uncomment this line, and change it as shown in the image. (After learning about the road network planning function, please restore this comment to avoid affecting the previous examples.)

vim ~/.bashrc
:wq #Save and exit
source ~/.bashrc #Update environment

• Road network navigation map and road network map

xxxxxxxxxx
~/map

The image shows the saved map and road network diagram. The map.geojson file contains the road network diagram of the car's trajectory.

• Configure AI large model parameters

xxxxxxxxxx
~/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/config/multi_brains_setting.yaml

multi_brains_setting.yaml：

The main configuration for this yaml file is the application body ID. Based on the camera model, select the corresponding application body key and disable the other one. After configuration, you need to recompile and run it. For the large AI model, you need to open an account and bind a key on the API platform beforehand. For specific configuration details, please refer to section 5, "Configuring the Large AI Model," in the Basic Knowledge chapter.

• Configure the pose data of the target points

xxxxxxxxxx
~/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/config/map_mapping.yaml

Add new target points under the route navigation section, as this course uses route navigation.

If you don't know how to obtain the pose data of a point, you can first refer to the "04-Route Planning Combined with Sandbox Map Case" in the route planning navigation section.

After modifying the yaml files, you need to recompile and then run the program.

xxxxxxxxxx
cd ~/yahboomcar_ros2_ws/yahboomcar_ws
colcon build --packages-select multi_brains

4. Program Startup

4.1 Startup Command

• Open the host terminal and enable the Dify environment.

xxxxxxxxxx
sh ~/bringup_dify.sh

• Launch large model + chassis + camera

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py enable_route_nav:=True

The following content will be displayed after initialization is complete

• Enter the road network navigation docker

xxxxxxxxxx
bringup_ros2kilted

xxxxxxxxxx
exec_ros2kilted

Enter this command in the terminal.

xxxxxxxxxx
ros2 launch road_net_route roadnet_nav2.launch.py 

• (On-board) Launch cartograph for quick location.

xxxxxxxxxx
ros2 launch yahboomcar_nav localization_imu_odom.launch.py use_rviz:=false load_state_filename:=/home/jetson/map/yahboomcar.pbstream

Upon entering, you'll notice that the point cloud and the map don't overlap; you'll need to use 2D Pose Estimate to reposition the car.

• Start YOLO Road Sign Recognition

xxxxxxxxxx
ros2 launch auto_drive auto_drive.launch.py enable_route_nav:=True

After successful launch, a visual interface will appear.

4.2 Test Cases

Here are some reference test cases. Users can create their own dialogue commands.

• Navigate to the construction site while following traffic rules

4.2.1 Case 1

First, wake up the robot with "Hello yahboom". The robot will respond with "I'm here, please give instructions". After the robot responds, the buzzer will sound briefly (beep—), then you can speak. The robot will perform voice activity detection (VAD); if there is voice activity, it will print 1, otherwise it will print -. When speaking ends, it will perform tail silence detection; if silent for more than 450ms, it will stop recording.

Dynamic voice activity detection (VAD) is shown in the figure below:

The robot first recognizes the user's speech, then makes decisions and takes actions according to the instructions. Simultaneously, the terminal prints the following information:

The decision-making layer's large model outputs the planned task steps:

Then, the execution layer's large model executes according to these task steps:

Simultaneously, as you can see, the road network navigation interface displays a grid route for the car to navigate to the construction site.

If we turn on a red light midway through the navigation, the car will stop and wait for the green light before resuming navigation.

When the green light comes on, the car will continue navigating. Note that the traffic lights should not be placed too close to the car's route; if they are too close, they may be perceived as obstacles and affect navigation.

5. Source Code Analysis

5.1 Path to the Large Model Source Code

xxxxxxxxxx
~/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py

action_service_nuwa.py: This file contains the main logic for the large model integration.

Key functionalities include:

• Providing action servers to handle navigation and robot control commands
• Integrating speech recognition and speech synthesis
• Enables switching between road network navigation and regular navigation
• Handles large language model interactions

It supports two navigation modes: Regular Navigation: a standard navigation mode based on a single point target; and Road Network Navigation: a navigation mode based on a predefined road network path. To integrate with autonomous driving, enable_route_nav: True needs to be enabled for road network navigation.

Navigation Process:

• Receive navigation request
• Select navigation strategy based on current mode (Road Network/Regular)
• Execute navigation actions
• Provide feedback and results

xxxxxxxxxx
# In the init_param_config method
self.declare_parameter("enable_route_nav", False)
self.enable_route_nav = self.get_parameter("enable_route_nav").get_parameter_value().bool_value

Navigation mode switching:

xxxxxxxxxx
def handle_navigation_request(self, destination):
    """Handling navigation requests"""
    if self.enable_route_nav:
        # Road network navigation mode
        if destination in self.route_network:
            self.follow_route(self.route_network[destination])
        else:
            self.get_logger().warn(f"No route found for destination: {destination}")
    else:
        # Normal navigation mode
        self.navigate_to_waypoint(destination)

5.2 Road Network Navigation Source Code Path

xxxxxxxxxx
#jetson orin nano
~/ros2_kilted/volumes/code_ws/src/road_net_route/launch/roadnet_nav2.launch.py
#RDK X5
/home/sunrise/ros2_kilted/volumes/code_ws/src/road_net_route/launch/roadnet_nav2.launch.py

roadnet_nav2.launch.py：

The launch file will directly call the file saved in the /root/map path. If it has a different name, you can change the name here and then recompile and run it.

xxxxxxxxxx
declare_map_yaml_cmd = DeclareLaunchArgument(
        'map',
        default_value='/root/map/map.yaml',
    )
    declare_pose_map_cmd = DeclareLaunchArgument(
        'pose_map',
        default_value='/root/map/map',
    )
    declare_roadnet_file_cmd = DeclareLaunchArgument(
        'roadnet_file',
        default_value='/root/map/map.geojson',
            description=".geojson格式路网文件路径",
    )

The parameter file used for navigation.

xxxxxxxxxx
declare_params_file_cmd = DeclareLaunchArgument(
        'params_file',
        default_value=os.path.join(get_package_share_directory('road_net_route'), 'config', 'nav2_kilted.yaml'),
        description='Full path to the ROS2 parameters file to use for all launched nodes',
    )

You can open the nav2_kilted.yaml file in this path and modify some navigation parameters. Normally, you don't need to modify the parameters; most of them are pre-tuned and don't require modification.

xxxxxxxxxx
~/ros2_kilted/volumes/code_ws/src/road_net_route/config/nav2_kilted.yaml

5.3 YOLO recognition source code

xxxxxxxxxx
~/yahboomcar_ros2_ws/yahboomcar_ws/src/auto_drive/launch/auto_drive.launch.py

auto_drive.launch.py：

The YOLO system is used to recognize road signs and instruct the car to perform corresponding actions. However, because this is based on a road network navigation system, the right turn and stop road signs will be disabled, as these two signs will interfere with the road network navigation.