Single Garbage Sorting

If using a depth camera (DABAI_DCW2), make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The single garbage sorting function uses the YOLOv11 model to identify garbage names, then based on the garbage name category, grabs the block from the center cross and places it in the corresponding garbage category area.

For all functions involving grasping of square blocks, please use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo No. 6 stall and damage.

Note: Before starting the program, please follow the [Assembly Tutorial] -> [Install the Map] tutorial to correctly install the map before operation.

1. Startup and Operation

Experiment Setup

Place the block with the garbage category sticker on the cross, with the text direction aligned with the camera direction.

Start ROS Node Service

Open the system terminal and enter the following command. If already running, there is no need to start again.

xxxxxxxxxx
ros2 run dofbot_ultra_info kinematics_dofbotultra

Start the Program

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Raspberry Pi code path:

xxxxxxxxxx
/home/yahboom/dofbot_ultra_ws/src/dofbot_garbage_yolov11/Garbage_Sorting_Single.ipynb

RDK X5 code path:

xxxxxxxxxx
/home/sunrise/dofbot_ultra_ws/src/2d/dofbot_gesture/scripts/Gesture_Stacking.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

Experiment Effect

Place the block with the garbage category sticker on the cross, with the text direction aligned with the camera direction. The camera identifies the garbage, prints the garbage name, and the arm automatically goes to the center cross to grab the block, places it in the corresponding garbage category area, and returns to the initial pose.

The example below uses walnut:

Effect after grabbing and placing

Before performing the next recognition operation, please remove the block to avoid conflicts during placement.

To exit the program, click the [Exit] button.

Since garbage recognition requires loading model files which takes up a large amount of memory, [Exit] only ends the function but does not close the model file program. You need to close the current kernels to shut down the model file program. Click [Kernel] -> [Shut Down All Kernels] in the menu bar.

2. Technical Overview

The core workflow of this function:

1. Model Loading: Load the YOLOv11 object detection model at startup, which has been trained on a garbage classification dataset

2. Garbage Recognition: The camera captures real-time images, the YOLOv11 model performs inference, and identifies the garbage name (e.g., walnut, battery, etc.)

3. Category Mapping: Based on the recognized garbage name, query its garbage classification (recyclable, hazardous, wet, dry), corresponding to the 4 areas on the map

4. Grab and Place: The arm moves to the cross position to grab the block and places it in the corresponding garbage category area

3. Core Code Analysis

• Import header files

xxxxxxxxxx
#!/usr/bin/env python
# coding: utf-8
import Arm_Lib
import os
import cv2 as cv
import threading
from time import sleep
import ipywidgets as widgets
from IPython.display import display
from single_garbage_identify import single_garbage_identify
from dofbot_utils.fps import FPS

• Create instances and initialize parameters

xxxxxxxxxx
single_garbage = single_garbage_identify()
single_garbage.init_robot_joint()
fps = FPS()
model = "General"

• Create widgets

xxxxxxxxxx
button_layout      = widgets.Layout(width='320px', height='60px', align_self='center')
output = widgets.Output()
exit_button = widgets.Button(description='Exit', button_style='danger', layout=button_layout)
imgbox = widgets.Image(format='jpg', height=480, width=640, layout=widgets.Layout(align_self='center'))
controls_box = widgets.VBox([imgbox, exit_button], layout=widgets.Layout(align_self='center'))

• Main program

xxxxxxxxxx
def camera():
    from dofbot_utils.camera import create_camera
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            fps.update_fps()
            img = single_garbage.single_garbage_run(img)
            if model == 'Exit':
                cv.destroyAllWindows()
                capture.release()
                break
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
        except:
            capture.release()

• Start display

display(controls_box,output)
threading.Thread(target=camera, ).start()