Semantic Understanding and Command Compliance

1. Course Content

1. After running the large model program, users can interact with the robot through voice dialogue. User voice commands are first converted into text by the speech recognition large model, then the text is used to generate a large model and a visual multimodal system to accurately understand the user's commands and voice. Finally, the robot performs the specified action according to the user's command and responds to the user.

2. Start the Agent

Note: The agent must be started before testing any case. If it is already started, it does not need to be started again.

Enter the following command on the vehicle terminal:

start_agent

The terminal will print the following information, indicating a successful connection:

3. Run the Case

3.1 Start the Program

For Raspberry Pi 5 users, if the ROS container is not running, you need to start it first.

xxxxxxxxxx
bringup_m3

Open a terminal inside the container.

xxxxxxxxxx
exec_m3

Open the terminal on the vehicle terminal and enter the following command to start:

xxxxxxxxxx
shell ros2 launch m3_bringup llm_agent_control.launch.py

Alternatively, you can use the following shortcut:

xxxxxxxxxx
multi_brains

After initialization, the following will be displayed:

The most crucial module logs are:

• action_service: Indicates that the action server started successfully

xxxxxxxxxx
[action_service-3] [INFO] [1767699384.764757965] [action_service]: ROS_Action_Service Initialization completed

• model_service: Indicates the corresponding model used by the current speech recognition engine ASR_Engine and speech synthesis engine TTS_Engine, and also indicates that model_service started successfully

xxxxxxxxxx
[model_service-2] [INFO] [1767699385.642591925] [ASR_Detect]: paraformer-realtime-v2 initialized successfully
[model_service-2] [INFO] [1767699385.655452485] [model_service]: Tongyi TTS_Engine initialized successfully
[model_service-2] [INFO] [1767699385.656321749] [model_service]: ROS_LLM_Service Initialization completed

3.2 Test Cases

Two reference test cases are provided here. Users can create their own test commands.

• Please move forward quickly 1 meter, then slowly retreat 0.5 meters like a tortoise, then turn 30 degrees to the left, then 90 degrees to the right, then move horizontally 0.5 meters to the left, then horizontally 10 centimeters to the right.
• Please tell a joke.

3.2.1 Case 1

• First, wake up the robot using "Hello yahboom". The robot will respond. After the recording prompt tone, the user can speak. The robot will perform dynamic sound detection. If there is sound activity, it will print "1-1-1-1"; if there is no sound activity, it will print "---------". After speaking, it will perform tail sound detection. If there is silence for more than 1.5 seconds, recording will stop.
• The robot will first respond to the user, and then follow the instructions. The terminal will print the following information:

Large Model Response Information Interpretation:

• Decision Layer Large Model Output: 1. Quickly move forward 1 meter, slowly move backward 0.5 meters, turn left 30 degrees, turn right 90 degrees, move left 0.5 meters, move right 0.1 meters. Decision-Maintenance Model Output: action": ['set_cmdvel(0.5, 0, 0, 2)', 'set_cmdvel(-0.1, 0, 0, 5)', 'move_left(30, 1.5)', 'move_right(90, 1.5)', 'set_cmdvel(0, 0.5, 0, 1)', 'set_cmdvel(0, -0.1, 0, 1)'], "response": Okay, I'll start executing your instructions now. First, I'll move forward quickly, then slowly backward, and then turn around, just like performing a complex dance.action": ['finishtask()'], ​​"response": I've completed all the actions, and it feels like I've finished a wonderful dance performance! If you have any other tasks that you need my help with, feel free to let me know! The action list includes finish()

3.2.2 Case 2

Similar to Case 1, after waking up and speaking, the robot performs operations according to the user's instructions.

4. Code Analysis

This lesson applies the basic program framework of AI embodied intelligence. Code analysis refers to the 4. Embodied intelligent functions core source code chapter.

5. Common Problem Solutions

5.1 Microphone Recording Too Sensitive

If you find that the VAD Voice Activity Detection continuously displays "1-1-1-1" after the speaking phase ends, it indicates that the VAD detection is too sensitive. You can prioritize adjusting the microphone sensitivity; it's generally rare to adjust sensitivity at the software level.

First, connect to the robot's onboard screen via VNC. Click the option bar in the upper right corner and find the Settings option.

[!NOTE]

• The operation method is similar for Raspberry Pi and Jetson Nano models; the microphone sensitivity is adjusted in the settings in the upper right corner, only the UI style differs.

Pull down the Settings list on the left, find the Sound option, and on the Sound page, find the Input audio input Input Device. Drag the Volume below to adjust the sensitivity. Try adjusting it to a suitable value while recording.

5.2 Microphone Recording Insensitive

If the speaker is far from the robot, the VAD Voice Activity Detection may not detect sound activity, causing the recording to end prematurely before the speaker finishes speaking. In this case, refer to the steps in 5.1 Microphone Recording Too Sensitive to appropriately increase the microphone sensitivity.

Note:

• If the microphone sensitivity is set too high, it may increase the false positives for ambient noise, mistaking it for speech activity.

5.3 Software Adjustment of Speech Recognition Sensitivity

Configuration file path for the multi_brains package:

xxxxxxxxxx
~/yahboomM3_ws/user_ws/src/m3_bringup/config/multi_brains/multi_brains_setting.yaml

• Edit the file using nano or another method:

xxxxxxxxxx
nano ~/yahboomM3_ws/user_ws/src/m3_bringup/config/multi_brains/multi_brains_setting.yaml

• Locate the VAD_MODE parameter. The parameter range is 1-3; a higher number indicates higher sensitivity of the VAD in detecting speech activity.

xxxxxxxxxx
####################
#ASR function setting
#语音识别功能设置
####################
USE_OLINE_ASR : True                                  # Whether to use online ASR
ASR_SUPPLIER  : 'aliyun'                              #ASR Supplier (only effective when using online ASR): Chinese mainland: aliyun International: xunfei
OLINE_ASR_MODEL : 'paraformer-realtime-v2'
ASR_THREASHOLD : 3                                    #ASR识别结果阈值,单位：字符ASR recognition result threshold. Unit: characters
WAKEUP_THREASHOLD : 5.0                               #唤醒时间阈值，防止WAKEUP_THREASHOLD时间内被多次唤醒，单位：秒 Wake-up time threshold, preventing multiple awakenments within the WAKEUP_THREASHOLD time, unit: seconds
VAD_MODE: 1                                           #vad灵敏度  VAD sensitivity
MAX_SILENCE_FRAMES: 90                                #尾音时长检测，单位：帧数      Tail sound duration detection, unit: frames    

5.4 End-of-Phonetic Detection Duration

If you need to adjust the end-of-phony detection duration, you can modify the MAX_SILENCE_FRAMES parameter value in the multi_brains_setting.yaml configuration file mentioned above.