2. Multimodal Visual Understanding

Introduction

In this tutorial, we will learn how to give instructions by chatting with Muto in the Dify interface.

Steps

Prerequisites

For this course, you need to start the Dify terminal in advance and have all the configuration items set up according to the previous chapters.

This course requires MutoRS to be powered on. Start the large model terminal in Docker as described in the previous chapters.

In this tutorial, we use the voice version of the large model. Do not include any parameters when starting it.

./rebuild_and_launch.sh

Getting Started

In this tutorial, let's take a look at Muto's large model capabilities combined with multimodal vision.

We open the Dify control page.

The scene in front of Muto is as follows:

You can enter the Docker container in another terminal using enter_muto_docker and then use rqt to view the camera data.

Type and send the following: "Tell me what's in front of you," and then ask it to guess what kind of drink it is.

When we see the content in the red box in the terminal as shown above, it means our large model terminal has started successfully.

• At this point, we can use the wake-up word "Hi yahboom" or "Hello yahboom" to activate it.

• After successful activation, we can hear Muto's response, "I am here," and see the following output in the terminal:

  

• Now, we can start speaking: “Go forward and see What is in front of you”

  

• If you are using a terminal that supports XServer (like MobaXTerm) or a terminal opened via VNC, you will see a window automatically pop up, displaying the current camera feed.

  

Randomness and Hallucinations in Large Models

In the text version of the vision section, the large model identified the drink as Coca-Cola.

(Excerpt from the text version tutorial)

Obviously, the picture shows a bottle of Pepsi, but the large model identified it as Coca-Cola. From this, we can see that the "intelligence" of the large model is unstable. So, if the model's feedback appears to be "nonsense" in subsequent processes, we can try switching to a better model for testing.

Taking the decision-making layer node as an example:

After clicking on it, you can see the available models.

If you want speed, you can choose the Flash series models.

If you want better accuracy, you can choose the Max series models.

In addition, the prompt is also a crucial key. By improving the quality of the prompt, we can also make the large model's "thinking conclusions" more in line with our ideas.

This time, we correctly identified it as Pepsi. ✅️

This once again confirms the uncertainty and existence of hallucinations in the model's output.