Embodied Intelligence

Function Introduction

This case is about starting a large program. You need to complete the configuration of the API-KEY related to the large model before you can use it normally. This function is based on the more complex action semantics expressed by the user. The robot dog performs an action. This function requires an Internet connection to work properly

Function Experience

1. Turn on the robot dog first, press the button on the upper right of the "dog head" to enter the sample mode, and then select the embodied intelligence function.

2. After entering the embodied intelligence function, first wake it up by voice, "lulu".

3. When you hear the sound of a ding, you can say the meaning of the command.

4. The robot dog will respond and perform actions based on the user's semantics.

Program source code

1. First, log in to the robot dog system through VNC
2. Then enter the terminal

​x
#cm4 run
cd /home/pi/RaspberryPi-CM4-main/demos/dog_agent/
​
#cm5 run
cd /home/pi/RaspberryPi-CM5/demos/dog_agent/
​
tree -L 1

3. Directory structure description (only the files related to this project are described) ├── AIagent_go.py #Main program entry of embodied intelligence - Chinese version ├── AIMain_en.py #Main program entry of embodied intelligence - English version ├── auto_platform.py #Environment dependency file ├── DAgent_en.py #Intelligent agent - English ├── Dogagent.mp3 #Synthesized audio ├── dog_agent.py #Intelligent agent - Chinese ├── dog_API_en.py #Large language model-English ├── dog_base_control.py #Basic control of mechanical dog ├── dog_caw_api.py #Grab function ├── dog_football_api.py #Kick function ├── dog_record.py #Record ├── dog_speakiat_en.py #Speech recognition-English version ├── dog_speak_iat.py #Speech recognition-Chinese version ├── dog_tongyiAPI.py #Large language model interface 1-Chinese ├── dog_tts.py #Synthetic audio interface ├── dog_UltraAPI.py #Large language model interface 1-English └── libnyumaya.py #Voice wake-up file

How to add more commands and actions

• First open the DAgent_en.py file to add a sample command

• Then you need to encapsulate some action execution functions and save them in the dog_base_control.py file

For example, if you encapsulate a forward function, it will be as shown in the figure below

• Finally, add the interface and usage method of the encapsulated function to the DAgent_en.py file.

How to replace the large model interface

The large model used in this function: the English version is OpenRouter

• You can first start from the python version program corresponding to the platform to use the interface, and fill in the necessary information according to the platform interface and instructions.
• Then encapsulate the executable file into a function. You can refer to the method of "dog_API_en.py" and put it in the path /home/pi/RaspberryPi-CM4-main/demos/dog_agent. For example, the added file name is "mychatgpt.py"

• Open DAgent_en.py and replace from dog_UltraAPI import * with the newly added from mychatgpt import * in the header
• Then find this place and replace it with the API function interface you encapsulated.

• Then restart the car and enter this function again, you can run the model platform you replaced. If it cannot run, it means there is an error, and you need to check the syntax and logic of the newly added file yourself.

How to run this case in the terminal

1. End the big program first to prevent the screen from being distorted. For how to end it, please go to the tutorial of ending the big program in Chapter 1. If the screen is distorted during the operation, it means that the big program has not been completely ended. Press the button in the lower left corner of the screen to check again if there is any distorted screen.
2. Enter the following command in the terminal

x
#cm4 run
cd ~/RaspberryPi-CM4-main
python3 demos/dog_agent/AIMain_en.py 
​
#cm5 run
cd ~/RaspberryPi-CM5
python3 demos/dog_agent/AIMain_en.py

3. When the waiting for keyword appears, wake up the robot dog with "lu lu". After a few seconds, a scrolling prompt "current volume, boot threshold, end threshold" will appear. You can then input voice to the robot dog.

4. The large model analyzes the recognized voice, prints the reply content on the terminal and broadcasts it. (The picture shows the English version of the reply)

Notes:

1. If the terminal reports an error timeout, the reason may be that the large model interface is blocked or the network is affected. Just restart the program.

How to change the length of the recording time

1. Input in the terminal

x
#cm4 run
nano ~/RaspberryPi-CM4-main/demos/dog_agent/dog_record.py
​
#cm5 run
nano ~/RaspberryPi-CM5/demos/dog_agent/dog_record.py

2. Find here and change the place indicated in the figure below, and you can adjust the recording time according to your environment.

Note: start_threshold>end_threshold, the adjustment of these two values ​​needs to be adjusted according to your own environment.

How to end a running task

If the reply word is long or the action execution time is long and you want to end the unfinished task in advance, if you are using a large program screen, exit and re-enter the function. If you are running the program in the terminal, exit and re-enter.

Functional principle

The specific flowchart is as follows: