Welcome to DOFBOT-Ultra repository

Color Recognition And Sorting With Voice Broadcast

If using a depth camera (DABAI_DCW2), before running this example, be sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration, otherwise the robot arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly/ 3. Enter Terminal] section in [Close the web control program backend process started at boot]. After closing, please restart DofbotUltra!!!

This case requires placing building blocks within the camera's recognizable range, using 30x30x30mm building blocks, otherwise the arm cannot accurately grasp them.

1. Operation Instructions

1.1. Startup

Before running the function, you need to shut down the web control program backend process. For the shutdown method, please refer to the [4. Preparation - 1. Manage Web control services].

Raspberry Pi version (in Docker environment), enter the following command:

xxxxxxxxxx
python3 /home/yahboom/dofbot_ultra/2d/dofbot_voice/scripts/put_grab_broadcast.py

RDK X5 version (native environment), enter the following command:

xxxxxxxxxx
python3 /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_voice/scripts/put_grab_broadcast.py

1.2. Operation Description

After the program starts, the robotic arm automatically reaches the recognition position. Place the block on the cross stand. After the robotic arm correctly recognizes it 10 times continuously, the voice module will broadcast the recognized color, then automatically grab and place it in the corresponding color grid. Finally, it will broadcast "Placement complete". Taking the figure below as an example, the voice module will broadcast "I see Yellow", then grab and place it in the corresponding color grid. Note: If colors cannot be recognized or the target color cannot be tracked correctly, you may need to calibrate the HSV values in the program. Please refer to [Visual Basic Course] -> [Color Calibration].

HSV calibration:

xxxxxxxxxx
Raspberry Pi: python3 /home/yahboom/dofbot_ultra/2d/dofbot_color_identify/scripts/hsv_calibration_gui.py
RDK X5: python3 /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_color_identify/scripts/hsv_calibration_gui.py

2. Code Logic

This program integrates visual color recognition, voice broadcast, and robotic arm sorting to achieve an automated color-based sorting workflow.

The program continuously detects colored blocks in the camera's field of view using HSV color space for color recognition. To improve reliability, a continuous confirmation mechanism is adopted — the grasping action is triggered only when the same color is continuously recognized 10 times. After recognition is confirmed, the voice broadcasts the color name, and the robotic arm sorting action is executed in a separate thread, placing the colored block into the corresponding color area.

The sorting action uses the speech_color_sorting module (under dofbot_color_grab), which contains pre-defined position points and grasping logic for each color.

3. Source Code

Raspberry Pi: /home/yahboom/dofbot_ultra/2d/dofbot_voice/scripts/put_grab_broadcast.py

RDK X5: /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_voice/scripts/put_grab_broadcast.py

Sorting module:

Raspberry Pi: /home/yahboom/dofbot_ultra/2d/dofbot_voice/scripts/speech_color_sorting.py

RDK X5: /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_voice/scripts/speech_color_sorting.py

xxxxxxxxxx
from speech_color_sorting import speech_color_sorting

sorting = speech_color_sorting()
img = sorting.Sorting_grap(img, color_hsv)

def Sorting_grap(self, img, color_hsv):
    self.image = cv.resize(img, (640, 480))
    msg = {}
    if self.status == 'waiting':
        for key, value in color_hsv.items():
            point = self.get_Sqaure(key, value)
            if point != None:
                msg["name"] = key
        if len(msg) == 1:
            self.num += 1
            if self.num % 10 == 0 and self.status == 'waiting':
                self.status = "Runing"
                self.arm.Arm_Buzzer_On(1)
                sleep(0.5)
                threading.Thread(target=self.sorting_run, args=(msg['name'],)).start()
                self.num = 0
    return self.image

def sorting_run(self, name):
    if name == "red":
        mySpeech.void_write(88)
        time.sleep(1.0)
        mySpeech.void_write(46)
        self.sorting_move(self.robot.P_RED)
        mySpeech.void_write(81)
        self.status = 'waiting'

• Sorting_grap() - Frame-by-frame color detection, triggers sorting after 10 consecutive stable detections

• sorting_run() - Sorting action: broadcast color name, call sorting_move() to place block, broadcast "completed"

• self.status - State control, waiting = waiting for detection, Runing = executing sorting, preventing repeated triggering