Welcome to DOFBOT-Ultra repository

Palm Targeting

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Use MediaPipe Hands to detect the palm position, draw hand keypoints and bounding box on the image, and print the palm center coordinates to the terminal in real-time. This function is the foundation for subsequent palm tracking (14. Robotic arm tracks the palm).

1、Startup

1.1、Program Description

After the program starts, the camera captures the image, the program detects the palm, draws the hand skeleton and bounding box, and prints the center coordinates (X, Y) of the palm to the terminal.

1.2、Program Startup

• Enter the following command to start the program

xxxxxxxxxx
ros2 run dofbot_mediapipe 13_FindHand.py

Press the q key in the image window or Ctrl+C in the terminal to exit the program.

1.3、Source Code

Raspberry Pi code path:

xxxxxxxxxx
~/dofbot_ultra_ws/src/dofbot_mediapipe/scripts/13_FindHand.py

RDK X5 code path:

xxxxxxxxxx
~/dofbot_ultra_ws/src/2d/dofbot_mediapipe/scripts/13_FindHand.py

#!/usr/bin/env python3
# encoding: utf-8
import threading
import numpy as np
import cv2 as cv
from dofbot_utils.camera import create_camera
from dofbot_utils.fps import FPS
from media_library import HandDetector


class FindHand:
    def __init__(self):
        self.hand_detector = HandDetector()
        self.fps = FPS()

    def process(self, frame):
        frame, lmList, bbox = self.hand_detector.findHands(frame)
        if len(lmList) != 0:
            threading.Thread(
                target=self._print_info,
                args=(lmList, bbox), daemon=True).start()
        self.fps.update_fps()
        self.fps.show_fps(frame)
        return frame

    def _print_info(self, lmList, bbox):
        indexX = (bbox[0] + bbox[2]) / 2
        indexY = (bbox[1] + bbox[3]) / 2
        print("index X: %.1f, Y: %.1f" % (indexX, indexY))


def main():
    finder = FindHand()

    cap = create_camera()
    if not cap.isOpened():
        raise RuntimeError("Failed to open camera")

    try:
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                continue
            frame = finder.process(frame)
            cv.imshow('FindHand', frame)
            if cv.waitKey(1) & 0xFF == ord('q'):
                break
    finally:
        cap.release()
        cv.destroyAllWindows()


if __name__ == '__main__':
    main()