AI Model with Track Map and SLAM Navigation

1. Course Content

1. Combine prerequisite AI large model functionality to use robot's AI visual understanding and SLAM navigation for comprehensive case studies in the track map. This course requires the use of the accompanying track map

Note: For a better experience, please first learn and master the following content in the product tutorial:

• AI Large Model: [04. AI Model Development], [05.AI Model-Voice Version]—[3.Multimodal visual understanding+SLAM navigation]
• Road Network Planning Navigation: [07.Road Network Planning and Navigation]

2. Preparation Work

• Use the following command to start the microROS agent and connect to the chassis

[!WARNING]

If the agent is already started, there is no need to start it again

start_agent

We can name any point on the map. Here is an example of naming as shown in the figure

• For methods to obtain vehicle target points on the map, please refer to [05.AI Model-Voice Version]—[3.Multimodal visual understanding+SLAM navigation—2.3 Configure Map Mapping File]. The method is completely the same. This course is a reference scenario with the track map
• Edit the map mapping file (any editing method can be used)

xxxxxxxxxx
~/yahboomM3_ws/user_ws/src/m3_bringup/config/multi_brains/map_mapping.yaml

• Regular navigation points need to be filled into the points under the common_map_areas field. Add or delete according to your needs

xxxxxxxxxx
common_map_areas: #Common navigation
  A: 
    name: 'School'
    position:
      x: 2.315
      y: 1.444
      z: 0.0
    orientation:
      x: 0.0
      y: 0.0
      z: 0.673
      w: 0.740
  B: 
    name: 'Library'
    position:
      x: 2.797
      y: 0.165
      z: 0.0
    orientation:
      x: 0.0
      y: 0.0
      z: -0.627
      w: 0.779

• Additionally, you need to update the new map mapping information in Dify and publish save settings. For configuration methods, refer to [05.AI Model-Voice Version]—[3.Multimodal visual understanding+SLAM navigation—2.4 Configure Map Mapping Variables in Dify]

[!TIP]

⚠️ For the same test command, the large model's response content will not be completely identical and will differ slightly from the screenshots in the tutorial. If you need to enhance or reduce the diversity of large model responses, refer to the [4.AI Model Development] - [05.AI Model Development—Chatbot] course section on configuring decision layer large model parameters.

3. Run Case Study

3.1 Start Program

• Start AI large model functionality on the vehicle terminal, or you can use the shortcut command multi_brains

For Raspberry Pi 5 users, if the ROS container is not started, you need to start the container first,

xxxxxxxxxx
bringup_m3

Open a container terminal,

xxxxxxxxxx
exec_m3

xxxxxxxxxx
ros2 launch m3_bringup llm_agent_control.launch.py

After initialization is complete, the following content will be displayed

• Start navigation node (uses AMCL localization algorithm by default)

xxxxxxxxxx
ros2 launch m3_bringup nav2.launch.py

Then follow the navigation function startup process to initialize positioning. The rviz2 visualization interface will open. Click 2D Pose Estimate in the top toolbar to enter selection state, roughly mark the robot's position and orientation on the map. After initializing positioning, the preparation work is complete

3.2 Test Case

Here are reference test cases. Users can create their own dialogue commands

xxxxxxxxxx
Please first remember your current position, then go to the fruit store and convenience store in sequence, return to your original position, and tell me what you saw.

First use "Hello yahboom" to wake up the robot. After the robot responds and enters recording state, users can speak. The robot will perform dynamic sound detection. If there is sound activity, it will print 1-1-1-1, if no sound activity it will print -------. When speaking ends, tail sound detection will be performed. If silence exceeds 1 second, recording will stop. Dynamic sound detection (VAD) is shown in the figure below:

The robot will first reply to the user, then execute actions according to the command, while printing information in the terminal as follows:

Decision layer large model outputs planned task steps:

After that, the execution layer large model will execute according to these task steps:

When the task is completed, use "Hello,yahboom" to wake up the robot, the robot responds: "I'm here, please give instructions", at this time you can give the next command

After the robot completes the task, it will enter a waiting state. At this time, the robot enters free dialogue state again, but all dialogue history will be retained. You can wake up yahboom again, "end current task", to let the robot end the current task cycle, clear the dialogue history, and start a new task cycle.