Welcome to DOFBOT-Ultra repository

Text Large Model + YOLO26 + Intelligent Garbage Classification

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

After the program starts, give the large model a natural language instruction related to garbage sorting. The large model will first understand whether the user wants to "sort a certain type of garbage", "keep a certain type of garbage", or "clean up all garbage on the desktop". Then it will plan the corresponding action steps, and finally control the robotic arm combined with YOLOv26 recognition results to sort the garbage-labeled wooden blocks on the desktop.

The "garbage" in this section is not real garbage, but wooden blocks with garbage labels. The YOLO model is responsible for recognizing the garbage category labels on the wooden blocks, and the robotic arm completes the grabbing and placing according to the large model's planning results.

This section supports the following three common types of tasks:

• Sort a single type of garbage: for example, "sort toxic garbage"

• Keep a certain type of garbage: for example, "sort all garbage except recyclable garbage"

• Sort all garbage on the desktop: for example, "clean up the garbage on the desktop"

2. Startup

After entering the Docker container, enter the following command:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and enter the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then enter garbage sorting related instructions in the text_chat terminal. You can refer to the following examples:

xxxxxxxxxx
Sort recyclable garbage

The garbage sorting has the following instructions:

Sort a single type of garbage: for example, sort toxic garbage Keep a certain type of garbage: for example, sort all garbage except recyclable garbage Sort all garbage on the desktop: for example, clean up the garbage on the desktop

Here are several recommended test statements:

• Sort recyclable garbage

• Sort toxic garbage

• Sort wet garbage

• Sort dry garbage

• Sort all garbage except recyclable garbage

• Keep wet garbage, clean up all other garbage

• Clean up the garbage on the desktop

3. Task Planning

The key point of this section is not the robotic arm movement itself, but how the large model converts natural language intent into garbage sorting actions. The overall execution flow can be summarized as:

1. User enters a natural language instruction;

2. The decision layer first understands the user's intent and determines which type of garbage task it belongs to;

3. The execution layer outputs specific action functions according to the decision layer's planning;

4. action_service calls garbage sorting related capabilities;

5. YOLO26 detects desktop garbage-labeled wooden blocks, and the robotic arm completes grabbing and placing.

3.1 Sorting a Single Type of Garbage

When the user explicitly specifies to handle only one garbage type, the decision layer maps the user's intent to a garbage category, and the execution layer directly outputs the corresponding garbage sorting action.

For example:

xxxxxxxxxx
Sort toxic garbage

The large model usually plans as follows:

• Recognize that "toxic garbage" corresponds to the internal category tox

• Output action: garbage_sort('tox')

The mapping relationships for other types are as follows:

• Recyclable garbage -> rec

• Toxic garbage / Hazardous garbage -> tox

• Wet garbage / Kitchen waste -> wet

• Dry garbage -> dry

3.2 Keeping a Certain Type of Garbage

The essence of this type of task is "exclusive sorting". That is, the user does not say "which type do I want to sort", but rather "which type do I want to keep, clear away the rest".

For example:

xxxxxxxxxx
Sort all garbage except recyclable garbage

The decision layer needs to first understand:

• The user wants to keep "recyclable garbage"

• Therefore, the actual types that need to be processed in order are: toxic garbage, wet garbage, dry garbage

Therefore, the execution layer will not output very complex actions at once, but executes one by one in sequence:

• garbage_sort('tox')

• garbage_sort('wet')

• garbage_sort('dry')

The benefits of this approach are:

• Only one type is processed at a time, with clear logic;

• Even if a certain type of garbage is not currently on the desktop, it will not affect the subsequent types from continuing to execute;

• The robotic arm movements are more stable, making it easier to locate problems after errors.

3.3 Sorting All Garbage on the Desktop

When the user asks to clean up all garbage on the desktop, the decision layer identifies the task as "full garbage sorting".

For example:

xxxxxxxxxx
Clean up the garbage on the desktop

This type of task is usually planned as:

• Direct call: garbage_sort_all()

In code implementation, garbage_sort_all() represents a complete cleaning process for all garbage targets on the desktop. It is more suitable for natural expressions like "desktop overall cleaning" and does not require the user to manually specify specific garbage types.

4. Core Code Analysis

4.1 Garbage Sorting Action Entry Point

The core entry point for garbage sorting actions is at:

xxxxxxxxxx
src/llm/largemodel/largemodel/action_service.py

The action function corresponding to single-category garbage sorting is garbage_sort(type_).

xxxxxxxxxx
def garbage_sort(self, type_):
    self.pub_arm(list(OBS_JOINTS), runtime=1000)
    time.sleep(1.0)
    type_ = type_.strip("'\"")
    if type_ == "rec":
        target_type_ = float(1)
    elif type_ == "tox":
        target_type_ = float(2)
    elif type_ == "wet":
        target_type_ = float(3)
    elif type_ == "dry":
        target_type_ = float(4)
    else:
        self.action_status_pub("garbage_sort_failed", type_=type_)
        return

    cmd1 = "ros2 run largemodel_arm grasp"
    cmd2 = f"ros2 run largemodel_arm yolov11_garbage --ros-args  -p target_type:={target_type_:.1f}"

    subprocess.Popen(cmd1, shell=True)
    subprocess.Popen(cmd2, shell=True)

This code accomplishes two things:

1. Maps the large model's output rec/tox/wet/dry to internal numeric types;

2. Starts grasp and yolov11_garbage two nodes to enter the visual detection and grabbing process.

4.2 YOLO26 Garbage Recognition Node

Note: For compatibility with older products, the file name is yolov11, but it actually calls the model trained based on yolo26.

The YOLO26 garbage recognition node source code is at:

xxxxxxxxxx
src/llm/largemodel_arm/largemodel_arm/yolov11_garbage.py

This node internally pre-defines four categories of garbage label sets:

xxxxxxxxxx
self.recyclable_waste = {"newspaper", "zip_top_can", "book", "old_school_bag"}
self.toxic_waste = {"syringe", "expired_cosmetics", "used_batteries", "expired_tablets"}
self.wet_waste = {"fish_bone", "egg_shell", "apple_core", "watermelon_rind"}
self.dry_waste = {"toilet_paper", "peach_pit", "cigarette_butts", "disposable_chopsticks"}

When the action layer passes target_type, the YOLO node only focuses on this one garbage category and filters out the corresponding targets in the image.

For example:

• target_type = 1 represents recyclable garbage

• target_type = 2 represents toxic garbage

• target_type = 3 represents wet garbage

• target_type = 4 represents dry garbage

After detecting the target, the node publishes the recognized center point and depth information, allowing the grabbing node to execute the robotic arm gripper action.

4.3 Why the Large Model Can Understand "Keeping a Certain Type of Garbage"

"Keeping a certain type of garbage" is not directly supported by the underlying vision node, but completed by the large model through semantic conversion.

For example, when the user says:

xxxxxxxxxx
Keep wet garbage, clean up all other garbage

The large model performs semantic equivalence conversion in the decision phase:

• Keep wet

• Clean rec + tox + dry

Then the execution layer proceeds with multiple garbage_sort() actions in sequence, rather than letting the underlying YOLO node handle all complex logic at once.

So this reflects:

• The large model is responsible for understanding natural language intent

• YOLO is responsible for recognizing garbage-labeled targets

• The robotic arm is responsible for executing grabbing and sorting

After combining the three, the system can support more natural expressions.

5. Operation Phenomenon Description

After entering the garbage sorting instruction in the terminal, the program usually runs according to the following process:

1. The large model first returns a natural language reply, for example, "Okay, I will now sort the recyclable garbage on the desktop."

2. The execution layer outputs an action function, for example garbage_sort('rec')

3. The robotic arm first returns to the observation position

4. The YOLOv11 node starts recognizing garbage-labeled wooden blocks on the desktop

5. If a target is recognized, the robotic arm grabs and places it in the corresponding area

6. If no target of the current type is detected, the system will feedback "No garbage of this type found"

If the user gives a "keep a certain type of garbage" instruction, the system will repeat the above process multiple times until all the garbage types that need to be cleaned up have been processed.