Design of visual module for ergonomics assessment in human robotic collaborative applications

Research Internship at Technical University of Munich (TUM)

pose_ergonomic

• A ROS 2 package to subscribe synchronized images as input, detect 2D body and hands pose via openpose, performing ergonomic assessment and publish results.
  

• Installing openpose firstly following its prerequisites and installation instructions.
  

• Build in ROS 2 workspace: colcon build --packages-select pose_ergonomic
  Remember to source after building: . install/setup.bash
  Run: ros2 run pose_ergonomic pose_ergonomic
  

• Input options:
  --save_result: True: save results; False: do not save.
  --result_path: a path to store result.
  --single_view: pose estimation from single view or two views to be further ergonomically evaluated.
  --rot90: clockwise rotate 90 degrees.
  --camera_topic: the camera topic to subscribe as input.
  --ergonomic: 1: perform RULA; 2: perform NERPA.
  

• Output:
  

1. /pose: publish images with keypoints estimation results.
   
2. /angles: publish joint angular values.
   
3. /risk: publish risk levels from ergonomic assessment.
   

• ergonomic evaluation module: performing ergonomic assessment according to Rapid Upper Limbs Assessment(RULA) criteria based on 2D body and hands keypoints obtained from openpose.
  ergonomic_nerpa module: performing ergonomic assessment according to Novel Ergonomic Postural Assessment Method (NERPA).
  

syn_img_pub

• Subscribe two topics synchronously using messages filters in ROS Melodic.
  
• Approximate time synchronously subscribe images(side view and front-top view) from two cameras, rotate and resize side view image to make the two synchronized images into one image for later processing by OpenPose.
  
• Output: /syn_img: synchronized and integrated image.
  
• Run: rosrun syn_img_pub syn_img_pub.py
  

mocap_ergonomic

• A ROS 2 package to subscribe synchronized images as input, detect 3D body and hands pose via FrankMocap, performing ergonomic assessment and publish results.
  

• Installing FrankMocap firstly following its installation instructions.
  

• Build in ROS 2 workspace: colcon build --packages-select mocap_ergonomic
  Remember to source after building: . install/setup.bash
  Run: ros2 run mocap_ergonomic mocap_ergonomic
  

• Input:
  Subscribing /side_img and /front_img processed by syn_img_frank.
  

• Output:
  

1. /pose: publish images with keypoints estimation results.
   
2. /angles: publish joint angular values.
   
3. /risk: publish risk levels from ergonomic assessment.
   

• eva3d module: performing ergonomic assessment according to Rapid Upper Limbs Assessment(RULA) criteria based on 3D body and hands keypoints obtained from FrankMocap.
  

syn_img_frank

• Subscribe two topics synchronously using messages filters in ROS Melodic.
  
• Approximate time synchronously subscribe images(side view and front-top view) from two cameras, rotate and resize side view image. Finally, publish the synchronized images from different views sequentially for later processing by FrankMocap.
  
• Output: /syn_img: synchronized and integrated image.
  
• Run: rosrun syn_img_frank syn_img_frank.py
  

ROS1 Bridge

• ROS1 Bridge is necessary to transmit synchronized images from ROS melodic to ROS eloquent.
• Following the "build the ROS1 Bridge" and "run the ROS1 Bridge" parts of instructions here to build and run your ROS1 Bridge.

Ergonomics Results Analysis

In order to compare OpenPose results and Inverse Kinematics (IK) results, and analyze accuracy of OpenPose detection:

1. Establish a ROS1 Bridge.
   
2. Run syn_img_frank following the instruction above.
   
3. Run pose_ergonomic: ros2 run pose_ergonomic pose_ergonomic save_result False
   
4. Play the dataset (a rosbag):
   rosbag play -r 0.11 name of the rosbag
   In order to synchronize IK frames and OpenPose frames, a proper publish rate needs to be chosen. The publish rate here used is 0.11.
   

Note

You may have problems when using modules from OpenPose and FrankMocap in these packages. Pay attention to the path.
The installation paths of OpenPose and FrankMocap in this research internship are /home/student/openpose and /home/studnet/frankmocap.
The installed version:
FrankMocap: commit 88d6dce8828e609d9f576e8e2a78800bd9184069 (HEAD -> master, origin/master, origin/HEAD)
OpenPose: commit de3bea5f715e877c3fc90ad975123a62ec6a42a3 (HEAD -> master, origin/master, origin/HEAD)