4. Multimodal Visual Understanding + Visual Line Following

Introduction

In this tutorial, we will learn how to issue commands via chat with Muto in the Dify interface.

Operation Steps

Preparatory Steps

This lesson requires that the Dify terminal be launched beforehand, and all configuration items be configured as in the previous section.

This lesson requires MutoRS to be powered on. Following the steps in the previous chapters, start the large model terminal in Docker.

In this tutorial, we will use a plain text version of the large model, starting it with the --no-voice parameter. Using the --no-voice parameter eliminates the risk of being interrupted by external voice prompts during use!

cd muto-llm-2.0/ && ./rebuild_and_launch.sh --no-voice

Getting Started

For the large text model, you need to enter Muto's IP address here, which is the address used to access the Dify page.

This section will explain the visual line-following function.

The visual line-following function requires opening a new terminal in the background, entering Docker, and running the visual line-following processing hub:

xxxxxxxxxx
enter_muto_docker

In the new terminal, execute:

xxxxxxxxxx
ros2 launch yahboomcar_linefollow follow_line_launch.py

As you can see, we've opened a window like this:

We use the left mouse button to select the yellow line and get its HSV range:

The HSV range in the top left corner is [35, 94, 96, 89, 253, 255]

We copy this HSV range, open Dify, find the env icon in the upper right corner, and edit our ability_function_name variable.

Edit the HSV value here, changing the yellow HSV value to the one we just obtained.

After making the changes, click save.

The command we use in this section is:

"Start autopilot along the line of color xx in front of you."

As you can see, Muto will announce the voice prompt and then begin to move automatically along the line.