5. Multimodal Visual Understanding + Visual Following

Introduction

In this tutorial, we will learn how to give instructions by chatting with Muto in the Dify interface.

Steps

Prerequisites

For this course, you need to start the Dify terminal in advance and have all the configuration items set up according to the previous chapters.

This course requires MutoRS to be powered on. Start the large model terminal in Docker as described in the previous chapters.

Use the command to start the large model backend.

cd muto-llm-2.0/ && ./rebuild_and_launch.sh

Getting Started

In this section, we will demonstrate how to use the multimodal large model combined with a tracking algorithm to make Muto track a specific object in front of it.

The visual line following feature requires opening a new terminal in the background, entering Docker, and running the visual line following processing center.

xxxxxxxxxx
enter_muto_docker

In the new terminal, execute:

xxxxxxxxxx
ros2 launch yahboomcar_kcf_tracker kcf_tracker_launch.py

You can see that a window like this has opened.

We input the following command via voice:

Check if there's a Coca-Cola can in front of you, and if so, track it down.

Muto will first use the multimodal large model to find the position of the can, and then enable the tracking and following function.

Note that the KCF Tracker node we started here uses the CSRT algorithm for tracking by default. For the Raspberry Pi 5 8GB version, the tracking performance of the CSRT algorithm is significantly better than the default KCF algorithm.

Core Process Explanation

We can analyze the execution flow. Here, we will introduce the core process in text mode because the text version allows us to clearly see the execution flow of the large model in Dify, which is convenient for our learning and analysis.

First, the user's input question is passed to the decision-making layer.

The task planning steps output by the decision-making layer are as follows:

1. Start intelligent vision analysis and judge if there is a Coca-Cola can in the frame. If so, return the Coca-Cola can's position of the Pepsi can in the frame (x1, y1, x2, y2).
2. If a Coca-Cola can was detected in the previous step, broadcast that tracking is about to start, and execute KCF target tracking action; if not, broadcast 'There doesn't seem to be a Coca-Cola can here'

The execution result of the execution layer is as follows:

The finished_steps variable is used to store the execution result of each step.

The core code for enabling KCF tracking is as follows:

xxxxxxxxxx
def set_kcf_target(self, x1: int, y1: int, x2: int, y2: int, timeout: int = 300) -> tuple:
    """
        Set and start KCF tracking target area
​
        Args:
            x1 (int): Target area top-left x coordinate
            y1 (int): Target area top-left y coordinate  
            x2 (int): Target area bottom-right x coordinate
            y2 (int): Target area bottom-right y coordinate
            timeout (int): Command execution timeout (seconds), default 300
​
        Returns:
            tuple: (success, message, data)
        """
    try:
        # Ensure coordinates are integers
        x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
​
        # Calculate width and height for return information (optional)
        width = x2 - x1
        height = y2 - y1
​
        # Construct ROS2 command to publish initialization rectangle
        # Topic: /kcf_tracker/init_rect
        # Type: std_msgs/msg/Int32MultiArray
        # Data: [x1, y1, x2, y2]
        cmd = f'ros2 topic pub --once /kcf_tracker/init_rect std_msgs/msg/Int32MultiArray "{{data: [{x1}, {y1}, {x2}, {y2}]}}"'
​
        if self.debug:
            print(f"[DEBUG] Executing KCF init command: {cmd}")
​
            result = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
​
            if result.returncode == 0:
                return True, f"KCF追踪初始化成功: 区域[{x1}, {y1}, {x2}, {y2}]", {
                    "target_bbox": {"x1": x1, "y1": y1, "x2": x2, "y2": y2, "width": width, "height": height},
                    "init_rect": [x1, y1, x2, y2]
                }
            else:
                return False, f"初始化KCF追踪失败: {result.stderr}", {}
​
            except subprocess.TimeoutExpired:
                return False, "设置KCF追踪目标超时", {}
            except Exception as e:
                return False, f"设置KCF追踪目标异常: {str(e)}", {}