Welcome to DOFBOT-Ultra repository

Voice Control Color Block Sorting

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

After the program starts, issue a color block sorting command via voice control. Upon receiving the command, the program controls the robotic arm to pick up the corresponding color block and place it at the designated position.

This tutorial uses the generic startup command camera_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes via depth camera pipeline;

• Other values: Executes via standard USB camera pipeline.

All functions involving picking up cubic blocks must use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo 6 stall and damage.

2. Startup and Operation

2.1. Startup

Simplified startup

Starting modules separately requires opening many terminals, which is cumbersome. A simplified launch file is provided here.

ros2 launch dofbot_ultra_voice_ctrl voice_color_sort.launch.py

Modular startup

Requires opening 8 terminals, recommended for debugging only.

​x
# Start camera and inverse kinematics
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py

# Start arm control
ros2 run dofbot_ultra_driver arm_driver

# Start speech recognition and broadcast
ros2 launch dofbot_yahboom_speech speech.launch.py

# Start voice router
ros2 run dofbot_ultra_voice_ctrl voice_router

# Start color recognition program (enters calibration flow directly after startup)
ros2 run dofbot_ultra_driver color_detect_VC

# Start robotic arm grasping program
ros2 run dofbot_ultra_driver grasp

# Start color sorting voice control
ros2 run dofbot_ultra_driver color_sort_VC

# Start voice broadcast
ros2 run dofbot_yahboom_speech voice_player

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

For RDK modular startup

xxxxxxxxxx
# Start camera and inverse kinematics
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py

# Start arm control
ros2 run dofbot_ultra_driver arm_driver

# Start speech recognition and broadcast
ros2 launch dofbot_yahboom_speech speech.launch.py

# Start color recognition program (enters calibration flow directly after startup)
ros2 run dofbot_ultra_driver color_detect_VC

# Start robotic arm grasping program
ros2 run dofbot_ultra_driver grasp

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

Notes:

1. camera_arm_kin.launch.py is the generic entry point, suitable for both camera modes.

2. The startup method for nodes like color_detect_VC, grasp, voice_router remains unchanged.

2.2. Operation Steps

2.2.1. Calibrating Color Block Colors

After all programs are running, you need to calibrate the color block colors. The upper left corner of the image will display the currently recognized color block color, initially red. The program starts the color calibration process. Select the top area of the red block with the mouse and release to complete calibration; then press the f key to start calibrating the green block. Similarly, select the top area of the green block with the mouse and release to complete calibration; press the f key to start calibrating the blue block, select the top area of the blue block with the mouse and release to complete calibration; press the f key to start calibrating the yellow block, select the top area of the yellow block with the mouse and release to complete calibration.

2.2.2. Voice Control

After completing the color block color calibration, say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Sort the red/green/blue/yellow blocks" to the speech module. The robotic arm will lower its gripper to grasp the target color block and the speech module will broadcast "OK". After grasping the target color block, it will place it at the designated position. Finally, the robotic arm returns to its initial posture, and the speech broadcast module will announce "Placement complete".

3. Core Source Code Analysis

3.1. Color Detection Node color_detect_VC.py

This tutorial actually starts with:

xxxxxxxxxx
ros2 run dofbot_ultra_driver color_detect_VC

Therefore, the actual color detection source code in use is at dofbot_ultra_driver/color_detect_VC.py, not the old version at dofbot_ultra_voice_ctrl/Color/color_detect_VC.py. This node is responsible for completing color calibration, identifying the target color block, and publishing the detected position.

xxxxxxxxxx
self.preset_sub = self.create_subscription(Int8, 'color_preset', self._preset_callback, 10)
self.grasp_done_sub = self.create_subscription(Bool, 'grasp_done', self._grasp_done_callback, 10)
self.pub_ColorInfo = self.create_publisher(AprilTagInfo, "PosInfo", 1)
self.pub_joint5 = self.create_publisher(Int8, "set_joint5", 10)

The workflow is:

1. Receives color_preset and switches to the target color;

2. Extracts the corresponding color block from the image, calculates the center point and angle;

3. Publishes PosInfo to the grasping node;

4. Publishes set_joint5 for gripper pose control;

5. Waits for grasp_done and resets, preparing for the next sorting.

3.2. Voice Sorting Control Node color_sort_VC.py

This tutorial also starts:

xxxxxxxxxx
ros2 run dofbot_ultra_driver color_sort_VC

This node is the "voice translation layer" for the color block sorting feature. It subscribes to voice_result and converts "Grasp red/green/blue/yellow color block" into standard control topics, rather than embedding voice logic into the color detection node.

xxxxxxxxxx
self.voice_sub = self.create_subscription(Int8, 'voice_result', self._voice_callback, 10)
self.player_id_pub = self.create_publisher(Int8, 'player_id', 10)
self.color_preset_pub = self.create_publisher(Int8, 'color_preset', 10)
self.grasp_enable_pub = self.create_publisher(Bool, 'grasp_enable', 10)

When receiving voice commands 100~103, color_sort_VC.py will:

1. First publish player_id=45, the voice replies "OK";

2. Convert the voice number into a color preset color_preset;

3. Publish grasp_enable=True, allowing the grasping node to start working.

3.3. Robotic Arm Grasping Node grasp.py

The source code that actually executes the descending, grasping, and placing actions is started in this tutorial:

xxxxxxxxxx
ros2 run dofbot_ultra_driver grasp

The corresponding file is dofbot_ultra_driver/grasp.py. This node receives PosInfo and set_joint5 from the detection node, and when grasp_enable=True, it performs inverse kinematics and action sequences.

xxxxxxxxxx
self.sub = self.create_subscription(AprilTagInfo, 'PosInfo', self.pos_callback, 1)
self.sub_joint5 = self.create_subscription(Int16, "set_joint5", self.get_joint5_callback, 1)
self.enable_sub = self.create_subscription(Bool, 'grasp_enable', self._enable_callback, 10)
self.pubGraspStatus = self.create_publisher(Bool, 'grasp_done', 1)

After grasping is complete, it publishes grasp_done=True to reset the detection node and control node, and the voice broadcast module announces "Placement complete".