Welcome to DOFBOT-Ultra repository

Garbage Recognition With Voice Broadcast

If using a depth camera (DABAI_DCW2), before running this example, be sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration, otherwise the robot arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

1. Operation Instructions

1.1. Startup

Before running the function, you need to shut down the web control program backend process. For the shutdown method, please refer to the [4. Preparation - 1. Manage Web control services].

Raspberry Pi version (within Docker), after starting the ROS2 core node, run the following commands:

Step 1 - Start the ROS2 core node:

ros2 run dofbot_ultra_info kinematics_dofbotultra

Step 2 - Start the garbage recognition broadcast program:

xxxxxxxxxx
python3 /home/yahboom/dofbot_ultra/2d/dofbot_voice/scripts/garbage_broadcast.py

RDK X5 version (native environment), enter the following command:

xxxxxxxxxx
python3 /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_voice/scripts/garbage_broadcast.py

1.2. Operation Description

Place the garbage label code block in the center of the image, then say "Hello, yahboom" to the voice module. The voice module will reply "here" to indicate successful wake-up. Then say "What garbage is this?" to the voice module. After the program recognizes it, the voice module will reply and broadcast what this garbage is and its type. Taking the figure below as an example, according to the recognition result, it will reply "This is a peach pit, which belongs to dry garbage".

2. Code Logic

This case uses YOLOv11 garbage recognition combined with voice broadcast, implementing the "voice trigger -> image recognition -> voice broadcast result" workflow.

Workflow:

1. The camera captures images, and the YOLOv11 model performs garbage object detection to identify the name of the waste item.
2. Look up the table according to the identified garbage name to confirm its waste category (recyclable waste / residual waste / kitchen waste / hazardous waste).
3. After the user asks "What garbage is this?" via voice input, the voice module broadcasts the name and category of the garbage.

YOLOv11 recognizes 16 types of garbage, divided into 4 categories:

• Recyclable (01): Zip-top can, old school bag, newspaper, book
• Other Waste (02): Toilet paper, peach pit, cigarette butts, disposable chopsticks
• Kitchen Waste (03): Egg shell, apple core, watermelon rind, fish bone
• Hazardous Waste (04): Expired tablets, expired cosmetics, used batteries, syringe

3. Source Code

Raspberry Pi: /home/yahboom/dofbot_ultra/2d/dofbot_voice/scripts/garbage_broadcast.py

RDK X5: /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_voice/scripts/garbage_broadcast.py

Recognition module:

Raspberry Pi: /home/yahboom/dofbot_ultra_ws/src/dofbot_garbage_yolov11/speech_garbage.py

RDK X5: /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_garbage_yolov11/speech_garbage.py

​x
from speech_garbage import speech_garbage
​
self.frame, msg = self.garbage_identify.garbage_run(self.frame)
​
for key, pos in msg.items():
    name = key
    if name == "Zip_top_can":           (self.garbage_num, self.garbage_class) = ('00', '01')
    if name == "Old_school_bag":        (self.garbage_num, self.garbage_class) = ('01', '01')
    if name == "Newspaper":             (self.garbage_num, self.garbage_class) = ('02', '01')
    if name == "Book":                  (self.garbage_num, self.garbage_class) = ('03', '01')
    if name == "Toilet_paper":          (self.garbage_num, self.garbage_class) = ('04', '02')
    if name == "Peach_pit":             (self.garbage_num, self.garbage_class) = ('05', '02')
    if name == "Cigarette_butts":       (self.garbage_num, self.garbage_class) = ('06', '02')
    if name == "Disposable_chopsticks": (self.garbage_num, self.garbage_class) = ('07', '02')
    if name == "Egg_shell":             (self.garbage_num, self.garbage_class) = ('08', '03')
    if name == "Apple_core":            (self.garbage_num, self.garbage_class) = ('09', '03')
    if name == "Watermelon_rind":       (self.garbage_num, self.garbage_class) = ('10', '03')
    if name == "Fish_bone":             (self.garbage_num, self.garbage_class) = ('11', '03')
    if name == "Expired_tablets":       (self.garbage_num, self.garbage_class) = ('12', '04')
    if name == "Expired_cosmetics":     (self.garbage_num, self.garbage_class) = ('13', '04')
    if name == "Used_batteries":        (self.garbage_num, self.garbage_class) = ('14', '04')
    if name == "Syringe":               (self.garbage_num, self.garbage_class) = ('15', '04')
​
result = mySpeech.speech_read()
if result == 94:
    if self.garbage_num == '00':
        mySpeech.void_write(94)
    elif self.garbage_num == '01':
        mySpeech.void_write(95)
    # ... continuously check and broadcast all garbage items

• garbage_identify.garbage_run() - Call YOLOv11 for garbage recognition, return identified garbage name and position
• Garbage names and classifications are mapped to codes, 16 garbage types correspond to 16 broadcast codes
• ID 94 corresponds to "Start Recognition" command, void_write broadcasts the specific garbage name