Garbage Sorting

If using a depth camera (DABAI_DCW2), make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The garbage sorting function is built upon the single garbage sorting function, adding the ability to recognize and sort multiple garbage blocks simultaneously. It needs to be used with the calibration function.

For all functions involving grasping of square blocks, please use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo No. 6 stall and damage.

Note: Before starting the program, please follow the [Assembly Tutorial] -> [Install the Map] tutorial to correctly install the map before operation.

1. Startup and Operation

Start ROS Node Service

Open the system terminal and enter the following command. If already running, there is no need to start again.

xxxxxxxxxx
ros2 run dofbot_ultra_info kinematics_dofbotultra

Start the Program

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Raspberry Pi code path:

xxxxxxxxxx
/home/yahboom/dofbot_ultra_ws/src/dofbot_garbage_yolov11/Garbage_Sorting.ipynb

RDK X5 code path:

xxxxxxxxxx
/home/sunrise/dofbot_ultra_ws/src/2d/dofbot_garbage_yolov11/Garbage_Sorting.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

Operation Steps

After the program runs, the JupyterLab page will display controls, with the camera view on the left and related button functions on the right.

Place the blocks in the recognition area with the garbage sticker side facing up, text direction aligned with the camera direction, and the bottom edge as parallel as possible to the bottom edge of the recognition area.

Note: Due to the height of the blocks, try to place them towards the center area, otherwise the camera may not see the full garbage sticker and cannot correctly identify it.

Then click [target_detection] to start garbage recognition.

Then click the [grap] button to start sorting. The system will sort the recognized garbage into the corresponding garbage category areas in order.

To exit the program, click the [Exit] button.

Since garbage recognition requires loading model files which takes up a large amount of memory, [Exit] only ends the function but does not close the model file program. You need to close the current kernels to shut down the model file program. Click [Kernel] -> [Shut Down All Kernels] in the menu bar.

2. Technical Overview

This function adds multi-target batch processing capability on top of single garbage sorting:

1. Camera Calibration and Coordinate Mapping: Use calibrated camera intrinsics for automatic distortion correction, and map pixel coordinates to real-world coordinates via the six-point calibration homography matrix

2. Multi-target Recognition: The YOLOv11 model simultaneously detects multiple garbage blocks in the recognition area, obtaining each garbage's name and position

3. Classification Planning: Based on each garbage's name, query its category (recyclable, hazardous, wet, dry) and plan the grab order

4. Batch Sorting: The arm grabs each block sequentially and places it in the corresponding garbage category area

3. Core Code Analysis

• Import header files

xxxxxxxxxx
import cv2 as cv
import threading
from time import sleep
import ipywidgets as widgets
from IPython.display import display
from garbage_identify import garbage_identify
from dofbot_utils.fps import FPS

• Create instances and initialize parameters

xxxxxxxxxx
target = garbage_identify()
msg = {}
xy = [90, 120]
debug_pos = False
model = "General"

• Main program

xxxxxxxxxx
def camera():
    global model, msg, debug_pos
    from dofbot_utils.camera import create_camera
    capture = create_camera()
    fps = FPS()
    while capture.isOpened():
        try:
            _, img = capture.read()
            if model == 'Detection' or model == 'Grap':
                img, msg = target.garbage_run(img)
                if debug_pos:
                    debug_pos = False
                    print("detect msg", msg)
            if len(msg) != 0 and model == 'Grap':
                print("grap msg", msg)
                threading.Thread(target=target.garbage_grap, args=(msg, xy,)).start()
                msg = {}
                model = 'Detection'
            if model == 'Exit':
                capture.release()
                break
            fps.update_fps()
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
        except KeyboardInterrupt:
            capture.release()
        except Exception as e:
            print("program end:", e)
            capture.release()

• Startup

display(controls_box,output)
threading.Thread(target=camera, ).start()