7. Multimodal Visual Understanding + SLAM Navigation + Visual Line Following

Introduction

In this tutorial, we will learn how to issue commands to Muto via chat within the Dify interface.

Operation Steps

Preparatory Steps

This lesson requires that the dify terminal be running beforehand and all configuration items have been configured as described in the previous section.

This lesson requires MutoRS to be powered on, as described in the previous chapters. Content: Starting the large model terminal in Docker

Starting the large model backend using the command

xxxxxxxxxx
./rebuild_and_launch.sh

Getting Started

In this section, we'll combine navigation and line-following functionality. Therefore, in addition to the terminal program for the large model and Dify, we also need to open the terminal programs for navigation and line-following simultaneously.

Start the radar node:

xxxxxxxxxx
ros2 launch yahboomcar_nav laser_bringup_launch.py

Start the radar odometry node:

xxxxxxxxxx
ros2 launch rf2o_laser_odometry rf2o_laser_odometry.launch.py

Start the navigation node:

xxxxxxxxxx
ros2 launch hexapod_nav hexapod_navigation_mppi.launch.py autostart:=true map:=/root/yahboomcar_ros2_ws/maps/test_map.yaml

Please note that a pre-built map file (*.yaml and *.pgm; here we are using the map /root/yahboomcar_ros2_ws/maps/test_map) is required.

Finally, we can start the Rviz2 node to easily observe the navigation process.

This step is recommended to be performed in a VMware virtual machine! **

xxxxxxxxxx
ros2 launch yahboomcar_nav display_nav_launch.py

It is crucial to change the selected area in the image to Transient Local here; otherwise, the map may not load.

After selecting it, we use the 2D Pose Estimate tool to set the initial position.

The scene where Muto is located is shown below.

The yellow area represents the route to be followed. The yellow line, and the red circles "end" and "start" represent two pre-saved navigation points.

Our task is: "Let Muto navigate from any position to 'end,' then begin following the yellow line. After traversing the line, observe the surrounding environment, and finally navigate to the 'start' point."

Therefore, before starting, we need to record the HSV value of the yellow line (refer to previous chapters), and then record the two navigation points "start" and "end."

We'll first place Muto at the 'start' position.

Important! Before saving the navigation point, you need to set the initial position in Rviz using the 2D Pose Estimate tool.

Then open the Divite terminal and type:

xxxxxxxxxx
Remember that the current position is the navigation point "start"

Next, manually move Muto to the starting point of the line, "end".

After moving, it's best to recalibrate the position using the RViz tool to minimize coordinate errors.

After moving, enter the following in the Dify terminal:

xxxxxxxxxx
Remember that the current position is the navigation point "end"

Next, we move Muto to any position, then reconfirm the initial position using RViz, and then enter the following command in the Dify page:

xxxxxxxxxx
Navigate to end, and then start following the yellow line. After completing the line, observe the surrounding environment, and then navigate back to the start point.

Finally, you can see that Muto has returned to the starting point.