4. Multimodal Visual Understanding + Visual Line Following

Introduction

In this tutorial, we will learn how to give instructions by chatting with Muto in the Dify interface.

Steps

Prerequisites

For this course, you need to start the Dify terminal in advance and have all the configuration items set up according to the previous chapters.

This course requires MutoRS to be powered on. Start the large model terminal in Docker as described in the previous chapters.

Use the command to start the large model backend.

./rebuild_and_launch.sh

Getting Started

In this section, we will explain the visual line following feature.

The visual line following feature requires opening a new terminal in the background, entering Docker, and running the visual line following processing center.

xxxxxxxxxx
enter_muto_docker

In the new terminal, execute:

xxxxxxxxxx
ros2 launch yahboomcar_linefollow follow_line_launch.py

You can see that a window like this has opened.

Use the left mouse button to select the yellow line to get its HSV range.

The HSV in the upper left corner is [35, 94, 96, 89, 253, 255].

Copy this HSV range, open Dify, find the env icon in the upper right corner, and edit our ability_function_name variable.

Edit the HSV value here, changing the yellow HSV value to the one we just obtained.

Click save after modification.

The command we use in this section is:

"Drive along the xx line in front of you."

We can interact with Muto by voice (the illustration shows text interaction, the voice effect is the same).

You can see that Muto will announce the voice command and then start moving forward along the line automatically.