1. Semantic Understanding and Instruction Following (Text Version)

1. Course Content

1. Learn the basic gameplay of the large model understanding user instructions.
2. After running the large model program, the user can interact with the robot through text. The text generation large model and vision multimodal model accurately understand the user's instructions, and finally the robot performs the specified actions according to the user's instructions and replies to the user.

2. Preparation

2.1 Content Description

The virtual machine used by default with this product starts the Dify docker environment automatically at boot, so no additional startup is needed.

2.2 Starting the Dify Environment

Open the virtual machine terminal:

sh ~/bringup_dify.sh

3. Running the Case

3.1 Starting the Program

Open the virtual machine terminal and enter the command to start the underlying proxy + camera topic:

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

In another terminal, enter the command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then open another terminal:

xxxxxxxxxx
ros2 run multi_brains terminal_client

Subsequently, enter dialogue commands in this terminal.

3.2 Test Cases

Reference test cases are given here. Users can create their own dialogue commands.

• Please tell me a joke about a kitten and a puppy.
• Please first move forward 1 meter quickly, then slowly move backward 0.5 meters like a turtle, then turn left 30 degrees and turn right 90 degrees.

3.2.1 Case 1

Enter the test case in the text dialogue terminal. After waiting for the model to think, the model replies to the user and executes actions according to the user's instructions.

3.2.2 Case 2

The method is the same as Case 1. Enter Case 2 in the terminal, and the model will reply and execute actions according to the instructions.