Welcome to ROS-WiFi Camera Module repository
6. Facial effects
Facial effects
1、synopsis
MediaPipe is a data stream processing machine learning application development framework developed and open-source by Google. It is a graph based data processing pipeline that enables the construction of various forms of data sources, such as video, frequency, sensor data, and any time series data. MediaPipe is cross platform and can run on embedded platforms (such as Raspberry Pi), mobile devices (iOS and Android), workstations, and servers, while supporting mobile GPU acceleration. MediaPipe provides cross platform, customizable ML solutions for real-time and streaming media. The core framework of MediaPipe is implemented in C++and provides support for languages such as Java and Objective C. The main concepts of MediaPipe include Packets, Streams, Calculators, Graphs, and Subgraphs.
The characteristics of MediaPipe：
- End to end acceleration: Built in fast ML inference and processing that can accelerate even on regular hardware。
- Build once, deploy anytime, anywhere: Unified solution suitable for Android, iOS, desktop/cloud, web, and IoT。
- Instant solution: A cutting-edge ML solution that showcases all features of the framework。
- Free and open source: Framework and solution under Apache 2.0, fully scalable and customizable.
2、Facial effects
2.1、activate
- First, set the proxy IP for the ROS-wifi image transmission module. For specific steps, please refer to the tutorial 1. Basic Usage of ROS-Wifi Image Transmission Module on Micros Car. This tutorial will not cover this process in detail.
- The Linux system is connected to the ROS-wifi image transfer module, and the Docker is started. Enter the following command to establish a connection with the ROS-wifi image transfer module.
#Use the provided system for direct inputsh start_Camera_computer.shxxxxxxxxxx#Systems that are not data:docker run -it --rm -v /dev:/dev -v /dev/shm:/dev/shm --privileged --net=host microros/micro-ros-agent:humble udp4 --port 9999 -v4
If the preceding information is displayed, the proxy connection is successful
- Open a new terminal and execute the following command
xxxxxxxxxxros2 run yahboom_esp32_mediapipe 06_FaceLandmarks
- If the camera picture is upside down, see 3. Camera picture correction (must-read) tutorial, this tutorial is no longer explained
2.2、Code parsing
The location of the function source code is located,
xxxxxxxxxx~/yahboomcar_ws/src/yahboom_esp32_mediapipe/yahboom_esp32_mediapipe/06_FaceLandmarks.py
xclass FaceLandmarks: def __init__(self, dat_file): self.hog_face_detector = dlib.get_frontal_face_detector() self.dlib_facelandmark = dlib.shape_predictor(dat_file)
def get_face(self, frame, draw=True): gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) self.faces = self.hog_face_detector(gray) for face in self.faces: self.face_landmarks = self.dlib_facelandmark(gray, face) if draw: for n in range(68): x = self.face_landmarks.part(n).x y = self.face_landmarks.part(n).y cv.circle(frame, (x, y), 2, (0, 255, 255), 2) cv.putText(frame, str(n), (x, y), cv.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 255), 2) return frame
def get_lmList(self, frame, p1, p2, draw=True): lmList = [] if len(self.faces) != 0: for n in range(p1, p2): x = self.face_landmarks.part(n).x y = self.face_landmarks.part(n).y lmList.append([x, y]) if draw: next_point = n + 1 if n == p2 - 1: next_point = p1 x2 = self.face_landmarks.part(next_point).x y2 = self.face_landmarks.part(next_point).y cv.line(frame, (x, y), (x2, y2), (0, 255, 0), 1) return lmList
def get_lipList(self, frame, lipIndexlist, draw=True): lmList = [] if len(self.faces) != 0: for n in range(len(lipIndexlist)): x = self.face_landmarks.part(lipIndexlist[n]).x y = self.face_landmarks.part(lipIndexlist[n]).y lmList.append([x, y]) if draw: next_point = n + 1 if n == len(lipIndexlist) - 1: next_point = 0 x2 = self.face_landmarks.part(lipIndexlist[next_point]).x y2 = self.face_landmarks.part(lipIndexlist[next_point]).y cv.line(frame, (x, y), (x2, y2), (0, 255, 0), 1) return lmList
def prettify_face(self, frame, eye=True, lips=True, eyebrow=True, draw=True): if eye: leftEye = landmarks.get_lmList(frame, 36, 42) rightEye = landmarks.get_lmList(frame, 42, 48) if draw: if len(leftEye) != 0: frame = cv.fillConvexPoly(frame, np.mat(leftEye), (0, 0, 0)) if len(rightEye) != 0: frame = cv.fillConvexPoly(frame, np.mat(rightEye), (0, 0, 0)) if lips: lipIndexlistA = [51, 52, 53, 54, 64, 63, 62] lipIndexlistB = [48, 49, 50, 51, 62, 61, 60] lipsUpA = landmarks.get_lipList(frame, lipIndexlistA, draw=True) lipsUpB = landmarks.get_lipList(frame, lipIndexlistB, draw=True) lipIndexlistA = [57, 58, 59, 48, 67, 66] lipIndexlistB = [54, 55, 56, 57, 66, 65, 64] lipsDownA = landmarks.get_lipList(frame, lipIndexlistA, draw=True) lipsDownB = landmarks.get_lipList(frame, lipIndexlistB, draw=True) if draw: if len(lipsUpA) != 0: frame = cv.fillConvexPoly(frame, np.mat(lipsUpA), (249, 0, 226)) if len(lipsUpB) != 0: frame = cv.fillConvexPoly(frame, np.mat(lipsUpB), (249, 0, 226)) if len(lipsDownA) != 0: frame = cv.fillConvexPoly(frame, np.mat(lipsDownA), (249, 0, 226)) if len(lipsDownB) != 0: frame = cv.fillConvexPoly(frame, np.mat(lipsDownB), (249, 0, 226)) if eyebrow: lefteyebrow = landmarks.get_lmList(frame, 17, 22) righteyebrow = landmarks.get_lmList(frame, 22, 27) if draw: if len(lefteyebrow) != 0: frame = cv.fillConvexPoly(frame, np.mat(lefteyebrow), (255, 255, 255)) if len(righteyebrow) != 0: frame = cv.fillConvexPoly(frame, np.mat(righteyebrow), (255, 255, 255)) return frame
class MY_Picture(Node): def __init__(self, name,landmarkss): super().__init__(name) self.bridge = CvBridge() self.sub_img = self.create_subscription( CompressedImage, '/espRos/esp32camera', self.handleTopic, 1) self.landmarksros = landmarkss
def handleTopic(self, msg): start = time.time() frame = self.bridge.compressed_imgmsg_to_cv2(msg) frame = cv.resize(frame, (640, 480)) cv.waitKey(10) frame = self.landmarksros.get_face(frame, draw=False) frame = self.landmarksros.prettify_face(frame, eye=True, lips=True, eyebrow=True, draw=True)
end = time.time() fps = 1 / (end - start) text = "FPS : " + str(int(fps)) &