7. Multimodal Visual Understanding + SLAM Navigation + Visual Line Following

Introduction

In this tutorial, we will learn how to issue commands via chat with Muto in the Dify interface.

Operation Steps

Preparatory Steps

This lesson requires that the Dify terminal be launched beforehand, and all configuration items have been configured as described in the previous section.

This lesson requires MutoRS to be powered on. Follow the steps outlined in the previous chapters to start the large model terminal within Docker.

In this tutorial, we'll use a plain text version of the large model, starting it with the --no-voice parameter. Using --no-voice eliminates the risk of being interrupted by external voice prompts during use!

cd muto-llm-2.0/ && ./rebuild_and_launch.sh --no-voice

Getting Started

For the large text model, you need to enter Muto's IP address here, which is the address used to access the Dify page.

In this section, we'll combine the navigation and line-following functions. Therefore, in addition to the large model's terminal program and Dify, we also need to open the navigation and line-following terminal programs.

Start the radar node:

xxxxxxxxxx
ros2 launch yahboomcar_nav laser_bringup_launch.py

Activate radar odometer node:

xxxxxxxxxx
ros2 launch rf2o_laser_odometry rf2o_laser_odometry.launch.py

Start the navigation node:

xxxxxxxxxx
ros2 launch hexapod_nav hexapod_navigation_mppi.launch.py autostart:=true map:=/root/yahboomcar_ros2_ws/maps/test_map.yaml

Please note that a pre-built map file (*.yaml and *.pgm; here we are using the map /root/yahboomcar_ros2_ws/maps/test_map) is required.

Finally, we can start the Rviz2 node to facilitate observation of the navigation process.

This step is recommended to be performed in a VMware virtual machine!!

xxxxxxxxxx
ros2 launch yahboomcar_nav display_nav_launch.py

Here, you must change the selected area in the image to Transient Local; otherwise, the map may not load.

After selecting it, we use the 2D Pose Estimate tool to set the initial position.

The scene where Muto is located is shown below:

The yellow area is the line to be followed. The yellow line, and the red circles "end" and "start" represent two saved navigation points.

Our task is: "Let Muto navigate from any position to 'end,' then begin following the yellow line. After traversing the line, observe the surrounding environment, and finally navigate to the 'start' point."

Therefore, before starting, we need to record the HSV value of the yellow line (refer to previous chapters), and then record the two navigation points "start" and "end."

We first place Muto at the 'start' position.

Important! Before saving the navigation points, you need to set the initial position in Rviz using the 2D Pose Estimate tool.

Then open the Divish terminal and enter:

xxxxxxxxxx
Remember that the current position is the navigation point "start."

Next, we manually move Muto to the starting point of the traversal, "end."

After moving, it's best to recalibrate the position using the RViz tool to minimize coordinate errors.

After moving, enter the following in the Dify terminal:

xxxxxxxxxx
Remember that the current position is the navigation point "end"

Next, we move Muto to any position, then reconfirm the initial position using RViz, and then enter the following command in the Dify page:

x
Navigate to end, and then start following the yellow line. After completing the line, observe the surrounding environment, and then navigate back to the start point.

Finally, you can see that Muto has returned to the starting point.