Object Detection

1. Experiment Objective

Use the OpenCV DNN module to load the SSD MobileNet V2 pre-trained model and detect common objects in the camera image (COCO 90 classes).

2. Experiment Principle

1. Key Terms

2. Technical Architecture

Terminal 1: start agent + camera + joint models
Terminal 2: start object detection

Data Flow: ESP32 camera → /camera/image_raw → image conversion → blobFromImage converts to a 300×300 blob → SSD MobileNet V2 forward inference → filter by confidence > 0.4 → draw bounding boxes + labels → cv2.imshow real-time display.

3. Experiment Steps

1. Hardware Preparation

2. Start Agent + Camera + Joint Model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start Object Detection (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_opencv_visual_pkg detect_object.launch.py

4. Operation Instructions

Workflow: Subscribe to /camera/image_raw → image conversion → blobFromImage (300×300, mean subtraction) → SSD inference → filter by confidence > 0.4 → draw a bounding box (cv2.rectangle) + class name: confidence label (cv2.putText) for each detection → display with cv2.imshow.

Interaction: Press q to exit.

5. How to Stop

Press Ctrl+C to stop Terminal 2 → Terminal 1 in order.

4. Experiment Observations

• Normal detection: Objects are enclosed by colored rectangles, with class name: confidence displayed above the box (e.g. person: 0.95)
• Multiple objects: Each detected object is drawn independently; objects of the same class share a uniform color
• No-object scene: All detection box confidences are ≤ 0.4, showing the raw image without annotation

5. Program Analysis

Source code paths:

• Launch: ~/microros_ws/src/microros_v2_opencv_visual_pkg/launch/detect_object.launch.py
• Node: ~/microros_ws/src/microros_v2_opencv_visual_pkg/microros_v2_opencv_visual_pkg/detect_object.py

1. Core Node Analysis

ObjectDetectionNode loads the SSD MobileNet V2 model, performs object detection on every frame, and visualizes the results.

​x
# Source file: ~/microros_ws/src/microros_v2_opencv_visual_pkg/microros_v2_opencv_visual_pkg/detect_object.py
class ObjectDetectionNode(Node):
    def __init__(self):
        super().__init__('object_detection_node')
        self.bridge = CvBridge()
        topic_name = '/camera/image_raw'
​
        self.subscription = self.create_subscription(
            Image, topic_name, self.image_callback,
            QoSProfile(history=QoSHistoryPolicy.KEEP_LAST, depth=1,
                       reliability=QoSReliabilityPolicy.BEST_EFFORT,
                       durability=QoSDurabilityPolicy.VOLATILE))
​
        # Load the COCO class list file
        config_path = ('.../microros_v2_opencv_visual_pkg/config')
        with open(f'{config_path}/object_detection_coco.txt', 'r') as f:
            self.class_names = f.read().split('\n')
​
        self.COLORS = np.random.uniform(0, 255, size=(len(self.class_names), 3))
​
        # Load the SSD MobileNet V2 model
        self.model = cv.dnn.readNet(
            model=f'{config_path}/frozen_inference_graph.pb',
            config=f'{config_path}/ssd_mobilenet_v2_coco.txt',
            framework='TensorFlow')
​
    def target_detection(self, image):
        image_height, image_width, _ = image.shape
        blob = cv.dnn.blobFromImage(
            image=image, size=(300, 300), mean=(104, 117, 123), swapRB=True)
        self.model.setInput(blob)
        output = self.model.forward()
​
        for detection in output[0, 0, :, :]:
            confidence = detection[2]
            if confidence > 0.4:
                class_id = int(detection[1])
                # Bounding box coordinate denormalization
                box_x = int(detection[3] * image_width)
                box_y = int(detection[4] * image_height)
                box_w = int(detection[5] * image_width)
                box_h = int(detection[6] * image_height)
                # Draw bounding box + label
                cv.rectangle(image, (box_x, box_y), (box_x+box_w, box_y+box_h),
                             self.COLORS[class_id], 2)
                label = f'{self.class_names[class_id-1]}: {confidence:.2f}'
                cv.putText(image, label, (box_x, box_y-5),
                           cv.FONT_HERSHEY_SIMPLEX, 0.7, self.COLORS[class_id], 2)
        return image

Key logic notes:

• blobFromImage with swapRB=True: OpenCV uses BGR by default while the model expects RGB; this parameter performs the conversion automatically
• Mean subtraction (104, 117, 123): subtracts the ImageNet training set mean from each channel to normalize the input distribution
• output[0, 0, :, :]: the SSD output tensor has the shape [1, 1, N, 7], each row being [_, class_id, confidence, x1, y1, x2, y2]
• Coordinate denormalization: the model outputs relative coordinates in the 0~1 range; multiplying by the actual image width and height yields pixel coordinates

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_opencv_visual_pkg/launch/detect_object.launch.py
• Node: ~/microros_ws/src/microros_v2_opencv_visual_pkg/microros_v2_opencv_visual_pkg/detect_object.py

3. Node Description

6. Frequently Asked Questions