Welcome to DOFBOT Pro repository

Voice Control Color Sorting

Before running the function, you need to close the App and large programs. For the closing method, refer to [4.Preparation] - [1. Manage APP control services].

1. Function Description

Through voice commands for color block sorting order, the robotic arm will sort color blocks according to the given order.

2. Startup and Operation

2.1. Startup

2.1.1. Orin Board

First start the ROS node service. Open the terminal on the main board and enter the following:

ros2 run dofbot_pro_info kinemarics_dofbot

Then enter the following command in the terminal to start:

xxxxxxxxxx
python ~/dofbot_voice/scripts/voice_color_sorting.py

2.1.2. Jetson-Nano Board

First start roscore. Open the terminal on the main board and enter the following:

xxxxxxxxxx
roscore

Then start the ROS node service. Open the terminal on the main board and enter the following:

xxxxxxxxxx
rosrun dofbot_pro_info kinemarics_dofbot_pro

Finally, open the terminal and enter the following command to start:

xxxxxxxxxx
python ~/dofbot_voice/scripts/voice_color_sorting.py

2.2. Operation Steps

After the program runs, the voice broadcast module will broadcast the colors of all blocks in the box. For example, in the figure below, it will broadcast "I see red, blue, yellow, green".

Then it will broadcast "The first color to sort is:". At this time, wake up the voice module with the keyword "Hello, yahboom" and say the first color to sort, for example "red". The program will record the first color to sort, then broadcast "The second color to sort is:", then continue to wake up with the keyword and say the color. If there is a color that doesn't need sorting, you can say "No". After sorting is completed, it will broadcast "Sorting completed, whether to start sorting". At this time, say "Start sorting", it will broadcast "ok", then execute the sorting action.

Note: If colors cannot be recognized, you may need to modify the HSV values in the program or adjust the ambient lighting.

3. Core Code Analysis

Source code path: ~/dofbot_voice/scripts/voice_color_sorting.py

xxxxxxxxxx
#Mainly look at the imported library file, speech_identify_target library, this library is located at ~/dofbot_pro/dofbot_color_sorting/scripts/identify_target.py
from speech_identify_target import identify_GetTarget
## Create target acquisition instance
target      = identify_GetTarget()
#Call select_color function to select, determine color and color block position
img, msg, coo1, coo2, coo3, coo4 = target.select_color(img, color_hsv, color_list)
#Start thread, start grabbing program
threading.Thread(target=target.target_run, args=(msg,xy)).start()

In the speech_identify_target library:

​x
#Select recognition color, the returned msg includes the selected color and the position of that color block
def select_color(self, image, color_hsv, color_list):
    '''
    Choose a recognition color
    :param image: input image
    :param color_hsv: Range threshold for HSV
    :param color_list: Color sequence: ['0': None '1': Red '2': Green '3': Blue '4': Yellow]
    :return: Output the processed image, (color, position)
    '''
    # canonical input image size
    self.image = image.copy()
    msg = {}
    # Get information about the first color sequence
    if len(color_list)==0:return self.image,msg
    if '1' in color_list:
        self.color_name=color_list['1']
        pos = self.get_Sqaure(color_hsv[self.color_name])
        if pos != None: msg[self.color_name] = pos
    if '2' in color_list:
        self.color_name=
 Voice Control Stacking
Garbage Recognition with Voice Broadcast（Jetson Nano）