3D Object Recognition

1. Experiment Objective

Use MediaPipe Objectron to detect 3D objects (shoe, chair, cup, camera) in real time and draw 3D bounding boxes and coordinate axes.

2. Experiment Principle

1. Key Terms

2. Technical Architecture

Terminal 1: start agent + camera + joint models
Terminal 2: start 3D object recognition

3. Experiment Steps

1. Hardware Preparation

2. Start the agent + camera + joint models (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start 3D object recognition (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_mediapipe_pkg objectron.launch.py

4. Observed Results

• Press F to switch among the four object types to detect
• Place one of the four objects in the camera's field of view and the object is enclosed by a 3D bounding box
• The RGB three-color axes indicate the object's spatial orientation (red=X, green=Y, blue=Z)

5. Code Walkthrough

Source code paths:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/objectron.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/Objectron.py

1. Core Node Analysis

​x
# Source file: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/Objectron.py
class Objectron(Node):
    def __init__(self):
        super().__init__("objectron_node")
        self.declare_parameter("static_image_mode", False)
        self.declare_parameter("max_objects", 1)
        self.declare_parameter("detection_con", 0.3)
        self.declare_parameter("process_every_n", 3)
        self.declare_parameter("object_name", "Shoe")
​
        self.mp_objectron = mp.solutions.objectron
        object_model = getattr(self.mp_objectron.ObjectModel, object_name)
        self.objectron = self.mp_objectron.Objectron(
            static_image_mode=False, max_num_objects=1,
            min_detection_confidence=0.3, min_tracking_confidence=0.3,
            model_name=object_model)
​
    def image_callback(self, msg):
        frame = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        results = self.objectron.process(cv.cvtColor(frame, cv.COLOR_BGR2RGB))
        if results.detected_objects:
            for obj in results.detected_objects:
                self.mp_draw.draw_landmarks(frame, obj.landmarks_2d,
                    self.mp_objectron.BOX_CONNECTIONS)
                self.mp_draw.draw_axis(frame, self.camera_matrix,
                    self.dist_coeffs, obj.rotation, obj.translation)
        cv.imshow(self.window_name, frame)

Key logic:

• getattr(self.mp_objectron.ObjectModel, object_name): dynamically fetches the enum value from one of the four model names
• process_every_n=3: processes one frame out of every 3 to reduce computational load
• min_detection_confidence=0.3: the default threshold is lower than for Hands/Pose because 3D object detection is more difficult
• draw_axis: draws the RGB three-color axes; requires camera intrinsics (loaded from camera.yaml)
• Objectron tends to SIGSEGV under the Ubuntu VM's SVGA3D/EGL stack, so the startup script sets LIBGL_ALWAYS_SOFTWARE=1 to force software rendering

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/objectron.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/Objectron.py

3. Node Description

6. Frequently Asked Questions