Welcome to Muto S2 Robot repository

7、Body movement control robot

The Raspberry Pi motherboard series does not currently support this tutorial.

7.1、Introduction to gameplay

The body control robot gameplay is a function realized by combining the poseNet module of jetson-inference and the motion control of the Muto robot. poseNet is mainly responsible for detecting the joint points of the human body in the image, and drawing the joint points of the human body through points and lines, and then the system The program analyzes the coordinate parameters of each joint point to control the robot's movement.

7.2、Core content analysis

Initialize the Muto robot and set the initial angle of the camera gimbal. The default is S1=90 and S2=60. The angle of the camera gimbal can be modified according to the actual viewing angle.

g_bot = Muto()
g_bot.Gimbal_1_2(90, 60)

Import the poseNet module from jetson-inference, and import the image input and output modules from jetson_utils.

xxxxxxxxxx
from jetson_inference import poseNet
from jetson_utils import videoSource, videoOutput, Log

Configure input parameters：

The input source defaults to USB camera device /dev/video0. The USB camera device number can be modified according to the device number found in the system.

The output source defaults to the graphical desktop.

The network network model defaults to resnet18-pose.

The overlay layer defaults to keypoints and links.

The default detection threshold is 0.15, which can be modified according to the actual recognition effect. The recommended range is 0.1-1.0.

​x
parser = argparse.ArgumentParser(description="Run pose estimation DNN on a video/image stream.",
                                 formatter_class=argparse.RawTextHelpFormatter,
                                 epilog=poseNet.Usage() + videoSource.Usage() + videoOutput.Usage() + Log.Usage())

parser.add_argument("input", type=str, default="/dev/video0",
                    nargs='?', help="URI of the input stream")
parser.add_argument("output", type=str, default="",
                    nargs='?', help="URI of the output stream")
parser.add_argument("--network", type=str, default="resnet18-pose",
                    help="pre-trained model to load (see below for options)")
parser.add_argument("--overlay", type=str, default="links,keypoints",
                    help="pose overlay flags (e.g. --overlay=links,keypoints)\nvalid combinations are:  'links', 'keypoints', 'boxes', 'none'")
parser.add_argument("--threshold", type=float, default=0.15,
                    help="minimum detection threshold to use")
try:
    args = parser.parse_known_args()[0]
except:
    print("")
    parser.print_help()
    sys.exit(0)

# load the pose estimation model
net = poseNet(args.network, sys.argv, args.threshold)

# create video sources & outputs
input = videoSource(args.input, argv=sys.argv)
output = videoOutput(args.output, argv=sys.argv)

The IDs corresponding to the joint points detected by poseNet are as follows:

xxxxxxxxxx
-- ID:  0 (nose)
-- ID:  1 (left_eye)
-- ID:  2 (right_eye)
-- ID:  3 (left_ear)
-- ID:  4 (right_ear)
-- ID:  5 (left_shoulder)
-- ID:  6 (right_shoulder)
-- ID:  7 (left_elbow)
-- ID:  8 (right_elbow)
-- ID:  9 (left_wrist)
-- ID:  10 (right_wrist)
-- ID:  11 (left_hip)
-- ID:  12 (right_hip)
-- ID:  13 (left_knee)
-- ID:  14 (right_knee)
-- ID:  15 (left_ankle)
-- ID:  16 (right_ankle)
-- ID:  17 (neck)

Read the camera image and then transmit it to detectNet to calculate and output the detection result.

xxxxxxxxxx
img = input.Capture()
poses = net.Process(img, overlay=args.overlay)

Traverse the poses results, extract all the joint point information of the first detected human body in the result, and then transfer all the joint point information of this human body to the parse_action function for analysis.

xxxxxxxxxx
for pose in poses:
    # print(pose)
    # print(pose.Keypoints)
    # print('Links', pose.Links)
 Object detection and action
Gesture controlled robot