Emotion Recognition

This tutorial is specific to CM5 version; it cannot be used directly in CM4 version.

1. Experiment Objective

Rider-Pi learns to perform a detection based on a person's facial expression. Once a certain emotion is detected, the real-time screen will select and label the detected result.

2. Experiment Steps

Enter the RiderPI system, terminate the RiderPI main program, and enter "ip (where ip is the RiderPI's IP):8888" in your browser. After logging in, enter the password "yahboom".

Then log in and navigate to the path cd /home/pi/Rider-pi_class/5.AI Visual Recognition Course/18. Emotion_Recognition/ and run emotion_demo.ipynb.

Finally, run the program.

3. Experimental Phenomena

1. After the program runs, this function will perform a detection based on a person's facial expressions. If a certain emotion is detected, the real-time screen will select and mark the detected result.

2. You can exit the program by pressing the button in the lower left corner of the screen.

4. Experimental source code analysis

​x
while True:
    bgr_image = picam2.capture_array()
    gray_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2GRAY)
    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
    faces = detect_faces(face_detection, gray_image)
​
    for face_coordinates in faces:
        x1, x2, y1, y2 = apply_offsets(face_coordinates, emotion_offsets)
        gray_face = gray_image[y1:y2, x1:x2]
        try:
            gray_face = cv2.resize(gray_face, (emotion_target_size))
        except:
            continue
​
        gray_face = preprocess_input(gray_face, True)
        gray_face = np.expand_dims(gray_face, 0)
        gray_face = np.expand_dims(gray_face, -1)
        emotion_prediction = emotion_classifier.predict(gray_face)
        emotion_probability = np.max(emotion_prediction)
        emotion_label_arg = np.argmax(emotion_prediction)
        emotion_text = emotion_labels[emotion_label_arg]
        emotion_window.append(emotion_text)
        if emotion_text != last_emotion:
            Count_num = 0
        else:
            Count_num += 1
        last_emotion = emotion_text
​
        if len(emotion_window) > frame_window:
            emotion_window.pop(0)
        try:
            emotion_mode = mode(emotion_window)
        except:
            continue
​
        if emotion_text == 'angry' and Count_num >= Count :
            color = emotion_probability * np.asarray((255, 0, 0))
            Count_num = 0
        elif emotion_text == 'sad' and Count_num >= Count :
            color = emotion_probability * np.asarray((0, 0, 255))
            Count_num = 0
        elif emotion_text == 'happy' and Count_num >= Count :
            color = emotion_probability * np.asarray((255, 255, 0))
            Count_num = 0
        elif emotion_text == 'surprise' and Count_num >= Count :
            color = emotion_probability * np.asarray((0, 255, 255))
            Count_num = 0
        elif emotion_text == 'neutral' and Count_num >= Count :
            color = emotion_probability * np.asarray((0, 255, 255))
            Count_num = 0
        else:
            color = emotion_probability * np.asarray((0, 255, 0))
​
        color = color.astype(int)
        color = color.tolist()
        
        draw_bounding_box(face_coordinates, rgb_image, color)
        draw_text(face_coordinates, rgb_image, emotion_mode,
                  color, 0, -45, 1, 1)
​
    bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2BGR)
    
    image_widget.value = bgr8_to_jpeg(bgr_image)
      
    
    if button.press_b():
        break
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

detect_faces(): The main business logic of this function is to detect faces.

apply_offsets(): The business logic of this function is to detect facial expressions.

draw_bounding_box(): Draws the bounding box of the recognition results.

draw_text(): Draws the recognized emotional text.

bgr8_to_jpeg(): Displays the image.