Intention Estimation + Multimodal Visual Understand + SLAM Navigation + Visual Functions (Voice Version)

1. Course Content

Note: This section requires you to first complete the map file configuration as described in [7. Multimodal Visual Understand + SLAM Navigation].

1. Learning to Use the RAG Knowledge Base to Train Personal Intention Understanding

Course Review:

If you need to increase or decrease the diversity of the big model's responses, refer to the section on configuring the decision-making big model parameters in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base].

Using a RAG knowledge base can expand the knowledge base of the large language model. This section explains how to expand the RAG knowledge base to enable the large language model to understand individual intentions.

Important Note! ! !:

1. The expanded personal intention understanding function may vary from user to user, and the large model's responses may vary. Actual debugging results may vary.

2. Expanding the personal intention understanding function must comply with social norms and laws and regulations. Technical Support assumes no responsibility for the final debugging results or any impact of this course.

2. Preparation

2.1 Content Description

This course uses the Jetson Orin NANO as an example. For users of the gimbal servo USB camera version, for example. For Raspberry Pi and Jetson-Nano boards, you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this lesson in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For users of the Orin board, simply open a terminal and enter the commands mentioned in this lesson.

📝 This example uses model:"qwen/qwen2.5-vl-72b-instruct:free","qwen-vl-latest"

⚠️ The responses from the large model may not be exactly the same for the same test command and may differ slightly from the screenshots in the tutorial. To increase or decrease the diversity of the large model's responses, refer to the section on configuring the decision-making large model parameters in [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base].

⚡️ I recommend trying the previous visual example first. This example adds voice functionality to the singleton example. The functionality is largely the same, so I won't go into detail about the implementation, code debugging, and results!

3. Configuring the Document

3.1 Creating a xlsx Document

Open the [Intent Mapping] folder in this lesson's folder and select the corresponding [Intent Mapping_xx.xlsx] file based on your robot configuration.

(You can add multiple files and more customized information as needed.)

[Intent Mapping]: Stores personal intentions and expectations for the large language model to control the robot's actions.

3.2 Uploading to the RAG Knowledge Base

Please modify according to the instructions in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base].

1. Access the IP address of the in-car system to enter the Dify backend.

2. Click on "Knowledge Base" and open the knowledge base for the corresponding camera.

3. Click on "Add File".

4. Add the prepared "Intent Mapping_xx.xlsx" file.

5. Click "Next", then "Save and Process".

6. Refresh the knowledge base; you can see that the intent mapping file has been added.

7. Click "Retrieval Test"; you can see that the imported intent mapping file has been successfully recalled.

8. Finally, go to the [Studio] interface, select the application corresponding to your camera model (here, we use [yahboom-nuwa_en] as an example).

Select [Task Routing] and add your custom intent mapping here, for example, "[I'm a little hungry]".

Click [Preview] to test it.

Finally, click "Publish" in the upper right corner to republish the corresponding application.

4. Run the Example

4.1 Starting the Program

4.1.1 Jetson Nano Board Startup Steps:

Due to Nano performance issues, navigation-related nodes must be run on a virtual machine. Therefore, navigation performance is highly dependent on the network. We recommend running in an indoor environment with a good network connection. You must first configure the following:

• Board Side (Requires Docker)

Open a terminal and enter

sudo vi /etc/chrony/chrony.conf

Add the following two lines to the end of the file: (Fill in 192.168.2.0/24 based on the actual network segment; this example uses the current board IP address of 192.168.2.41.)

xxxxxxxxxx
allow 192.168.x.0/24 #x indicates the corresponding network segment
local stratum 10

After saving and exiting, enter the following command to take effect:

xxxxxxxxxx
sudo service chrony restart

• Virtual Machine

Open a terminal and enter

xxxxxxxxxx
echo "server 192.168.2.41 iburst" | sudo tee /etc/chrony/sources.d/jetson.sources

The 192.168.2.41 entry above is the board's IP address.

Enter the following command to take effect.

xxxxxxxxxx
sudo chronyc reload sources

Enter the following command again to check the latency. If the IP address appears, it's normal.

xxxxxxxxxx
chronyc sources -v

• Start the program

Open the host machine terminal and start the Dify environment:

xxxxxxxxxx
sh ~/bringup_dify.sh

Open the terminal in Docker and enter the following command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization, the following content will be displayed:

On the virtual machine, create a new terminal and start.

xxxxxxxxxx
ros2 launch yahboomcar_nav display_nav_launch.py

Wait for the navigation algorithm to start before displaying the map. Then open a VM terminal and enter:

xxxxxxxxxx
ros2 launch yahboomcar_nav navigation_teb_launch.py

Note: When running the car's mapping function, you must enter the save map command on the VM to save the navigation map.

After that, follow the navigation function startup process to initialize positioning. The rviz2 visualization interface will open. Click 2D Pose Estimate in the upper toolbar to select it. Roughly mark the robot's position and orientation on the map. After initializing positioning, preparations are complete.

4.1.1 RDKX5, Raspberry Pi PI5 and ORIN board startup steps:

Open the host machine terminal and start the Dify environment:

xxxxxxxxxx
sh ~/bringup_dify.sh

For the Raspberry Pi PI5, you need to first enter the Docker container; RDKX5 and ORIN board does not require this.

Open the terminal of car system, and then enter the command in the terminal:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed.

Create a new terminal on the virtual machine and start the program.(For Raspberry Pi and RDKX5, it is recommended to run the visualization in the virtual machine.)

xxxxxxxxxx
ros2 launch yahboomcar_nav display_nav_launch.py

Wait for the navigation algorithm to start before the map is displayed. Then open the Docker terminal and enter

​x
#Choose one of the two navigation algorithms
#Normal navigation
ros2 launch yahboomcar_nav navigation_teb_launch.py

#Fast relocalization navigation (not supported on Raspberry Pi 5 or RDKX5 controllers)
ros2 launch yahboomcar_nav localization_imu_odom.launch.py use_rviz:=false
load_state_filename:=/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_nav/maps/yahboomcar.pbstream

ros2 launch yahboomcar_nav navigation_cartodwb_launch.py maps:=/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_nav/maps/yahboomcar.yaml
params_file:=/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_nav/params/cartoteb_nav_params.yaml

Note: yahboomcar.yaml and yahboomcar.pbstream must be mapped simultaneously, meaning they are the same map. Refer to the cartograph mapping algorithm to save the map.

After that, follow the navigation process to initialize positioning. The rviz2 visualization interface will open. Click 2D Pose Estimate in the upper toolbar to select it. Roughly mark the robot's position and orientation on the map. After initial positioning, preparations are complete.

4.2 Test Case

Here are some reference test cases; users can create their own dialogue commands.

• I'm in the office right now, and I'm feeling a bit thirsty.

4.2.1 Example

First, use "Hi, yahboom" to wake the robot. The robot responds, "I'm here, please." After the robot responds, the buzzer beeps briefly (beep—). The user can then speak. The robot then performs a sound detection, logging 1 if there's sound activity and - if there's no sound activity. When the speech ends, it detects the end of the voice, and stops recording if there's silence for more than 450ms.

The following figure shows the Voice Active Detection (VAD):

The robot will first communicate with the user, then respond to the user's instructions. The terminal will print the following message:

Network Abnormality: Decision-making AI Planning: The model service is abnormal. Check the large model account or configuration options. This does not affect the AI ​​model's decision-making function, but it is recommended to try again under smooth network conditions!

Normal: The decision-making AI planning will proceed through steps 1, 2, 3, 4, etc.

[model_service]: "action": ['seewhat()'], ​​"response": I've arrived at the tea room and am checking for water.

After arriving at the navigation destination, this step calls the seewhat method. A window will open on the VNC screen and display for 5 seconds. An image is retrieved and uploaded to the large model for inference.

The large model receives the result, noting that there is a bottle of water in the tea room. It then provides feedback to the user and executes the next command: Return to starting point.

After completing all tasks, the robot returns to a free conversation state, but all conversation history is retained. At this point, you can wake yahboom up again and click "End Current Task" to end the robot's current task cycle, clear the conversation history, and start a new task cycle.

5. Source Code Analysis

Source code located at:

Jetson Orin Nano:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

Jetson Nano, Raspberry Pi:

You need to first enter Docker.

xxxxxxxxxx
#NUWA Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

RDKX5：

xxxxxxxxxx
#NUWA Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

5.1 Example

action_service.py Program:

This example uses the seewhat, navigation, load_target_points, and get_dist methods from the CustomActionServer class. seewhat, navigation, load_target_points, and get_dist are described in the previous sections [2. Multimodal Visual Understanding], [6. Multimodal Visual Understanding + Depth Camera Distance Question Answering], and [7. Multimodal Visual Understanding + SLAM] navigation] have been explained, and there are no new procedures in this chapter.