K230 Rock-Paper-Scissors

Example Experimental Effect

In this section, we will learn static gesture recognition based on K230 and play a game of Rock-Paper-Scissors with the car

The game execution process is as follows:

After the game starts, we make the "rock", "scissors" or "paper" state with our hands and appear in the camera. At this time, the screen will show the program's punches and the winning and losing results. Each time the winner is determined, it means that the round is over. After each round, please move your hands out of the camera range.

The game has a total of three rounds. After three rounds, the program will get the final result according to the rule of two wins in three games. After the winner is determined, K230 will send a data frame to STM32. We will parse the msg according to the protocol and make corresponding actions. When the player wins, the car will light up green and shake, and when the machine wins, the red light will light up [There is a small BUG here. If the program does not give the final result immediately after the third round, you can let your hand enter the camera again, and the program will give the final result! Note: When connected to our AI voice interaction integrated module, there will be voice broadcast when the game is won or lost. Users who have not purchased the voice module do not need to worry. Without the voice module, K230 can still implement this case, but there is no voice broadcast function

Physical connection diagram

Motor wiring

Voice module wiring

K230 wiring (Note: The wiring in the figure below is for reference only. We are equipped with K230 double-headed PH2.0 4Pin all-black wire with fool-proof design, so you don’t have to worry about wiring problems)

Partial code analysis

Code structure

1. Initialization Phase:

   • Load detection and recognition models

   • Set parameters

   • Initialize detector and gesture recognition

   • Initialize game-related variables

2. Hand Detection Flow:

   • Run hand detection

   • Filter detection boxes

   • Pass to gesture recognition

3. Gesture Recognition Flow:

   • Configure preprocessing

   • Run model inference

   • Process output

4. Drawing Flow:

   • Clear display

   • Draw detection

   • Draw gesture recognition

   • Update display

5. Game Logic:

   • Process game logic

   • Check game state

   • Initialize new round

   • Update player and AI scores

   • Display final result

Communication protocol

msg: you lose, you win, tie

Voice module command word

​x
/**
 * 协议解析函数 - 解析特定格式的数据帧
 * 格式: [头部标识][字段1],[字段2],...,[字段n][尾部标识]
 * 
 * Protocol parsing function - parses data frames in a specific format
 * Format: [Header Flag][Field1],[Field2],...,[Fieldn][Trailer Flag]
 * 
 * @param data_buf 指向接收到的数据缓冲区的指针 Pointer to the received data buffer
 * @param num 数据缓冲区中的字节数 Number of bytes in the data buffer
 */
void Pto_Data_Parse(uint8_t *data_buf, uint8_t num)
{
    uint8_t pto_head = data_buf[0];           // 协议头部标识 Protocol header flag
    uint8_t pto_tail = data_buf[num-1];       // 协议尾部标识 Protocol trailer flag
    
    // 验证协议头部和尾部 Check protocol header and trailer
    if (!(pto_head == PTO_HEAD && pto_tail == PTO_TAIL)) {
        return;                             // 无效协议，退出 Invalid protocol, exit
    }

    uint8_t data_index = 1;                 // 字段索引计数器 Field index counter
    uint8_t field_index[PTO_BUF_LEN_MAX] = {0}; // 存储每个字段的起始位置 Stores the start position of each field
    int i = 0;

    // 解析逗号分隔符，将其替换为字符串结束符'\0'
    // Parse comma separators and replace them with string terminator '\0'
    for (i = 1; i < num-1; i++) {
        if (data_buf[i] == ',') {
            data_buf[i] = 0;                // 将逗号替换为字符串结束符 Replace comma with string terminator
            field_index[data_index] = i;    // 记录字段分隔位置 Record field separator position
            data_index++;                   // 增加字段计数器 Increment field counter
        }
    }

    // 处理无逗号的纯字符串情况 - 通常用于路标识别
    // Handle pure string case without commas - typically for road sign recognition
    if (data_index == 1) {
        uint8_t len = num - 2;              // 计算字符串长度 Calculate string length
        memcpy(lubiao, (char*)data_buf+1, len); // 复制数据到路标缓冲区 Copy data to road sign buffer
        lubiao[len] = '\0';                 // 确保字符串以'\0'结尾 Ensure string is null-terminated
    }
    else { // 处理多字段数据 Process multi-field data
        for (i = 0; i < data_index; i++) {
            uint8_t start = field_index[i] + 1; // 字段起始位置 Field start position
            uint8_t len;

            if (i == data_index - 1) {
                // 最后一个字段，长度计算到尾部标记前
                // Last field, calculate length up to the trailer flag
                len = (num - 1) - start;
            } else {
                // 中间字段，长度为下一个分隔符位置减去当前起始位置
                // Middle field, length is the next separator position minus current start position
                len = field_index[i + 1] - start;
            }

            // 特殊处理特定字段 Special handling for specific fields
            if (i == 6 && data_index == 7) {
                memcpy(msg, (char*)data_buf + start, len); // 复制消息字段 Copy message field
                msg[len] = '\0';        // 添加字符串结束符 Add string terminator
            }
            else if (i == 2 && data_index == 3) {
                memcpy(game, (char*)data_buf + start, len); // 复制游戏字段 Copy game field
                game[len] = '\0';       // 添加字符串结束符 Add string terminator
            }
            else {
                // 将字符转换为整数 Convert characters to integer
                values[i] = Pto_Char_To_Int((char*)data_buf + start);
            }
        }

        // 协议长度验证 Protocol length verification
        uint8_t pto_len = values[0];      // 从协议中获取的长度 Length obtained from protocol
        if (pto_len != num) {
            printf("pto_len error:%d , data_len:%d\n", pto_len, num);
            return;                     // 长度不匹配，退出 Length mismatch, exit
        }
        pto_id = values[1];             // 存储协议ID Store protocol ID
    }
}

xxxxxxxxxx
# 猜拳游戏任务类 / Rock-paper-scissors game task class
class FingerGuess:
    def __init__(self, hand_det_kmodel, hand_kp_kmodel, det_input_size, kp_input_size, labels, anchors, confidence_threshold=0.25, nms_threshold=0.3, nms_option=False, strides=[8, 16, 32], guess_mode=3, rgb888p_size=[1280, 720], display_size=[1920, 1080], debug_mode=0):
        # 手掌检测模型路径 / Hand detection model path
        self.hand_det_kmodel = hand_det_kmodel
        # 手掌关键点模型路径 / Hand keypoint model path
        self.hand_kp_kmodel = hand_kp_kmodel
        # 手掌检测模型输入分辨率 / Hand detection model input resolution
        self.det_input_size = det_input_size
        # 手掌关键点模型输入分辨率 / Hand keypoint model input resolution
        self.kp_input_size = kp_input_size
        self.labels = labels
        # anchors / Anchor boxes
        self.anchors = anchors
        # 置信度阈值 / Confidence threshold
        self.confidence_threshold = confidence_threshold
        # nms阈值 / Non-Maximum Suppression threshold
        self.nms_threshold = nms_threshold
        # nms选项 / NMS option
        self.nms_option = nms_option
        # 特征图针对输入的下采样倍数 / Feature map downsampling multipliers relative to input
        self.strides = strides
        # sensor给到AI的图像分辨率，宽16字节对齐 / Image resolution from sensor to AI, width aligned to 16 bytes
        self.rgb888p_size = [ALIGN_UP(rgb888p_size[0], 16), rgb888p_size[1]]
        # 视频输出VO分辨率，宽16字节对齐 / Video output VO resolution, width aligned to 16 bytes
        self.display_size = [ALIGN_UP(display_size[0], 16), display_size[1]]
        # debug_mode模式 / Debug mode
        self.debug_mode = debug_mode
        self.guess_mode = guess_mode
        # 石头剪刀布的贴图array / Rock-paper-scissors image arrays
        self.five_image = self.read_file("/sdcard/utils/five.bin")
        self.fist_image = self.read_file("/sdcard/utils/fist.bin")
        self.shear_image = self.read_file("/sdcard/utils/shear.bin")
        self.counts_guess = -1                                                               # 猜拳次数 计数 / Count of rock-paper-scissors rounds
        self.player_win = 0                                                                  # 玩家 赢次计数 / Player win count
        self.k230_win = 0                                                                    # k230 赢次计数 / K230 win count
        self.sleep_end = False                                                               # 是否 停顿 / Whether to pause
        self.set_stop_id = True                                                              # 是否 暂停猜拳 / Whether to suspend the game
        self.LIBRARY = ["fist", "yeah", "five"]                                              # 猜拳 石头剪刀布 三种方案的dict / Dictionary of three rock-paper-scissors options
        self.hand_det = HandDetApp(self.hand_det_kmodel, self.labels, model_input_size=self.det_input_size, anchors=self.anchors, confidence_threshold=self.confidence_threshold, nms_threshold=self.nms_threshold, nms_option=self.nms_option, strides=self.strides, rgb888p_size=self.rgb888p_size, display_size=self.display_size, debug_mode=0)
        self.hand_kp = HandKPClassApp(self.hand_kp_kmodel, model_input_size=self.kp_input_size, rgb888p_size=self.rgb888p_size, display_size=self.display_size)
        self.hand_det.config_preprocess()

    # run函数 / Run function
    def run(self, input_np):
        # 先进行手掌检测 / First perform hand detection
        det_boxes = self.hand_det.run(input_np)
        boxes = []
        gesture_res = []
        for det_box in det_boxes:
            # 对检测的手做手势识别 / Perform gesture recognition on detected hands
            x1, y1, x2, y2 = det_box[2], det_box[3], det_box[4], det_box[5]
            w, h = int(x2 - x1), int(y2 - y1)
            if (h < (0.1 * self.rgb888p_size[1])):
                continue
            if (w < (0.25 * self.rgb888p_size[0]) and ((x1 < (0.03 * self.rgb888p_size[0])) or (x2 > (0.97 * self.rgb888p_size[0])))):
                continue
            if (w < (0.15 * self.rgb888p_size[0]) and ((x1 < (0.01 * self.rgb888p_size[0])) or (x2 > (0.99 * self.rgb888p_size[0])))):
                continue
            self.hand_kp.config_preprocess(det_box)
            results_show, gesture = self.hand_kp.run(input_np)
            boxes.append(det_box)
            gesture_res.append(gesture)
        return boxes, gesture_res

    # 绘制效果 / Draw results
    def draw_result(self, pl, dets, gesture_res):
        pl.osd_img.clear()
        # 手掌的手势分类得到用户的出拳，根据不同模式给出开发板的出拳，并将对应的贴图放到屏幕上显示
        # Get user's gesture from hand gesture classification, determine the board's gesture based on different modes, and display corresponding images on screen
        if (len(dets) >= 2):
            pl.osd_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "请保证只有一只手入镜 Make sure only one hand on the screen", color=(255, 255, 0, 0))
        elif (self.guess_mode == 0):
            draw_img_np = np.zeros((self.display_size[1], self.display_size[0], 4), dtype=np.uint8)
            draw_img = image.Image(self.display_size[0], self.display_size[1], image.ARGB8888, alloc=image.ALLOC_REF, data=draw_img_np)
            if (gesture_res[0] == "fist"):
                draw_img_np[:400, :400, :] = self.shear_image
            elif (gesture_res[0] == "five"):
                draw_img_np[:400, :400, :] = self.fist_image
            elif (gesture_res[0] == "yeah"):
                draw_img_np[:400, :400, :] = self.five_image
            pl.osd_img.copy_from(draw_img)
        elif (self.guess_mode == 1):
            draw_img_np = np.zeros((self.display_size[1], self.display_size[0], 4), dtype=np.uint8)
            draw_img = image.Image(self.display_size[0], self.display_size[1], image.ARGB8888, alloc=image.ALLOC_REF, data=draw_img_np)
            if (gesture_res[0] == "fist"):
                draw_img_np[:400, :400, :] = self.five_image
            elif (gesture_res[0] == "five"):
                draw_img_np[:400, :400, :] = self.shear_image
            elif (gesture_res[0] == "yeah"):
                draw_img_np[:400, :400, :] = self.fist_image
            pl.osd_img.copy_from(draw_img)
        else:
            draw_img_np = np.zeros((self.display_size[1], self.display_size[0], 4), dtype=np.uint8)
            draw_img = image.Image(self.display_size[0], self.display_size[1], image.ARGB8888, alloc=image.ALLOC_REF, data=draw_img_np)
            if (self.sleep_end):
                time.sleep_ms(2000)
                self.sleep_end = False
            if (len(dets) == 0):
                self.set_stop_id = True
                return
            if (self.counts_guess == -1 and gesture_res[0] != "fist" and gesture_res[0] != "yeah" and gesture_res[0] != "five"):
                draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "游戏开始 GAME START", color=(255, 255, 0, 0))
                time.sleep_ms(500)
            elif (self.counts_guess == self.guess_mode):
                draw_img.clear()
                if (self.k230_win > self.player_win):
                    draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "你输了 You lose", color=(255, 255, 0, 0))
                elif (self.k230_win < self.player_win):
                    draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "你赢了 You win", color=(255, 255, 0, 0))
                else:
                    draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "平局 | tie", color=(255, 255, 0, 0))
                self.counts_guess = -1
                self.player_win = 0
                self.k230_win = 0
                self.sleep_end = True
            else:
                if (self.set_stop_id):
                    if (self.counts_guess == -1 and (gesture_res[0] == "fist" or gesture_res[0] == "yeah" or gesture_res[0] == "five")):
                        self.counts_guess = 0
                    if (self.counts_guess != -1 and (gesture_res[0] == "fist" or gesture_res[0] == "yeah" or gesture_res[0] == "five")):
                        k230_guess = randint(1, 10000) % 3
                        if (gesture_res[0] == "fist" and self.LIBRARY[k230_guess] == "yeah"):
                            self.player_win += 1
                        elif (gesture_res[0] == "fist" and self.LIBRARY[k230_guess] == "five"):
                            self.k230_win += 1
                        if (gesture_res[0] == "yeah" and self.LIBRARY[k230_guess] == "fist"):
                            self.k230_win += 1
                        elif (gesture_res[0] == "yeah" and self.LIBRARY[k230_guess] == "five"):
                            self.player_win += 1
                        if (gesture_res[0] == "five" and self.LIBRARY[k230_guess] == "fist"):
                            self.player_win += 1
                        elif (gesture_res[0] == "five" and self.LIBRARY[k230_guess] == "yeah"):
                            self.k230_win += 1
                        if (self.LIBRARY[k230_guess] == "fist"):
                            draw_img_np[:400, :400, :] = self.fist_image
                        elif (self.LIBRARY[k230_guess] == "five"):
                            draw_img_np[:400, :400, :] = self.five_image
                        elif (self.LIBRARY[k230_guess] == "yeah"):
                            draw_img_np[:400, :400, :] = self.shear_image
                        self.counts_guess += 1
                        draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "第" + str(self.counts_guess) + "回合 | Round " +  str(self.counts_guess), color=(255, 255, 0, 0))
                        self.set_stop_id = False
                        self.sleep_end = True
                    else:
                        draw_img.draw_string_advanced(self.display_size[0] // 2 - 50, self.display_size[1] // 2 - 50, 30, "第" + str(self.counts_guess + 1) + "回合 | Round " +  str(self.counts_guess), color=(255, 255, 0, 0))
            pl.osd_img.copy_from(draw_img)

    # 读取石头剪刀布的bin文件方法 / Method to read rock-paper-scissors bin files
    def read_file(self, file_name):
        image_arr = np.fromfile(file_name, dtype=np.uint8)
        image_arr = image_arr.reshape((400, 400, 4))
        return image_arr

Voice module firmware burning

xxxxxxxxxx
Burning the CI1302_EN_SingleMic_V00916_UART1_115200_2M.bin file we provide can realize all cases related to the intelligent voice interaction module of this car. Only one burning is required. For the burning method, see [Voice Interaction Control Tutorial] [Section 6] Voice Module Firmware Burning

K230 Code Burning

Required tools: K230, SD card, CanMV IDE K230

We find the finger_guessing.py file in the project folder, then drag it to CanMV IDE K230, and then choose to save the opened script as main.py to CanMV

System program flow chart