Smart Dog

Function Introduction

This case is the gameplay of starting the big program. You need to complete the configuration of the API-KEY related to the configuration big model before you can use it normally. This function also belongs to the embodied intelligence gameplay, adding visual recognition processing gameplay to the multimodal interaction gameplay. This function needs to be connected to the Internet to work properly

The visual big model and the text big model use different big models

Function Experience

1. Turn on the robot dog first, enter the sample mode by pressing the button on the upper right of the "dog head", and then select the smart dog function.

2. After entering the smart dog function, wake it up by voice first, "lulu".

3. When you hear a ding, you can say the meaning of the command.

4. The robot dog will answer and perform actions based on the user's meaning and take a live picture.

Notes

• Once you enter the cases of face detection, bone detection, license plate detection, line patrol, and object tracking, you need to press the button on the upper right corner of the screen to exit the current mode. For example: face detection is followed by skeleton recognition and then dancing. If you want to enter skeleton recognition after entering face detection, first press the button in the upper right corner of the screen to exit face detection before entering skeleton recognition. The same is true if you want to reach the dance action.

Program source code

1. First, log in to the robot dog system through VNC
2. Then enter the terminal

​x
#cm4 run
cd /home/pi/RaspberryPi-CM4-main/demos/dog_agent/
​
#cm5 run
cd /home/pi/RaspberryPi-CM5/demos/dog_agent/
​
tree -L 1

3. Directory structure description (only the files related to this project are described) ├── AIagent_go_image.py #Main function of the intelligent agent - Chinese version ├── AIMImage_en.py #Main function of the intelligent agent - English version ├── auto_platform.py #Environment dependency ├── DAgent_Image_en.py #Intelligent dog's intelligent agent - English version ├── dog_agent_Image.py #Intelligent dog's intelligent agent - Chinese version ├── dog_API_en.py #Intelligent dog's large model interface - English version ├── dog_base_control.py #Basic action control interface ├── dog_caw_api.py #Clamping wood block interface ├── dog_football_api.py #Football API ├── dog_ImageAPI.py #Image API ├── dog_record.py #Recording API ├── dog_speakiat_en.py #Speech recognition API - Chinese version ├── dog_speak_iat.py #Speech recognition API - English version ├── dog_tongyiAPI.py #Smart dog large model API - Chinese version ├── dog_tts.py #Synthetic audio interface └── libnyumaya.py #Voice wake-up file

How to add more commands and actions

• First open the DAgent_Image_en.py file to add a sample command

• Then you need to encapsulate some action execution functions and save them in the dog_base_control.py file. For example, if you encapsulate a forward function, it will be as shown in the figure below.

• Finally, add the interface and usage method of the encapsulated function to the DAgent_Image_en.py file.

How to replace the large model interface

The large model used in this function is the Chinese version of iFlytek Spark + Tongyi Qianwen (visual processing), and the English version is OpenRouter**

• You can first start from the python version program of the platform corresponding to the interface, and fill in the necessary information according to the platform interface and instructions.
• Then encapsulate the executable file into a function. You can refer to the method of "dog_API_en.py" and put it in the path /home/pi/RaspberryPi-CM4-main/demos/dog_agent. For example, the added file name is "mychatgpt.py"
• Open DAgent_Image_en.py and add from mychatgpt import * in the head
• Then find this place and replace it with the API function interface you encapsulated.
• Then restart the car and enter this function again. You can run the model platform you replaced. If it cannot run, it means that there is an error. You need to check the syntax and logic of the newly added file yourself.

How to run this case in the terminal

1. End the big program first to prevent the screen from being distorted. For how to end it, please go to the tutorial of ending the big program in Chapter 1. It will not be described here.
2. Enter the following command in the terminal

x
#cm4 run
cd ~/RaspberryPi-CM4-main
python3 demos/dog_agent/AIMImage_en.py 
​
#cm5 run
cd ~/RaspberryPi-CM5
python3 demos/dog_agent/AIMImage_en.py 

3. When the waiting for keyword appears, wake up the robot dog with "lu lu". After a few seconds, a scrolling prompt "current volume, boot threshold, end threshold" will appear. You can then input voice to the robot dog.

4. The large model analyzes the recognized voice, prints the reply content on the terminal and broadcasts it. (The picture shows the English version of the reply)

Notes:

1. If the terminal reports an error timeout, the reason may be that the large model interface is blocked or the network is affected. Just restart the program.

How to change the length of the recording time

1. Enter in the terminal

x
#cm4 run
nano ~/RaspberryPi-CM4-main/demos/dog_agent/dog_record.py
​
#cm5 run
nano ~/RaspberryPi-CM5/demos/dog_agent/dog_record.py

2. Find here and change the place indicated in the figure below to adjust the recording time according to your environment.

Note: start_threshold>end_threshold, the adjustment of these two values ​​needs to be adjusted according to your own environment.

How to end a running task

If the reply word is long or the action execution time is long and you want to end the unfinished task in advance, if you are using a large program screen, exit and re-enter the function. If you are running the program in the terminal, exit and re-enter.

Functional principle

The specific flow chart is as follows: