Jack Schofield, Null Atwood, Travis Hudson, Rohan B Ballapragada, Zachary Taylor, Jet Ittihrit, Aditya Raj
ROS2 packages for the patient-monitoring robot. For the patient-monitoring Android application, go to pmr-app-test.
- TurtleBot3 Waffle
- Intel® RealSense™ Depth Camera
Note: The patient-monitoring nodes were tested using an Intel RealSense™ D405, but should function with any camera compatible with the Intel® RealSense™ ROS2 wrapper. Please inform us of any issues that arise using different camera models.
- Ubuntu 20.04 (link)
- ROS2 Foxy (link)
- Intel® RealSense™ SDK 2.0 (link)
- Intel® RealSense™ ROS2 wrapper (link)
Before continuing, please install Ubuntu 20.04 on the Turtlebot3's onboard computer. If you plan on using a remote PC, it will require Ubuntu 20.04 (and a ROS2 foxy install) as well. This guide will not cover setting up a remote PC on ROS2.
The TurtleBot3 Quick Start Guide (Section 3) is a good resource for setting up our ROS2 environment. Please refer to it (and make sure you are following the steps for "Foxy").
In our usage, we replaced the onboard Raspberry Pi that ships with the Turtlebot3 with an Intel® NUC to allow for real-time image processing onboard the robot. If you are planning to do something similar, please see the annotated notes below:
- Section 3. 1. PC Setup can be applied to the onboard PC, as we will require a working ROS2 installation to run nodes directly on the robot.
- Section 3. 1. 6. Network Configuration can be skipped, although we recommend completing it. Even if you want your image processing nodes to run directly on the robot, it is still useful to have a remote PC to see system diagnostics.
- Section 3. 2. SBC Setup can be entirely skipped if you are not using a Raspberry Pi, except for:
- Section 3. 2. 6. Configure the Raspberry Pi for those using a remote PC setup (the Raspberry Pi correspondsROS_DISTRO=<YOUR_SYSTEM_ROS_DISTRO> # set your ROS_DISTRO: humble, galactic, foxy source /opt/ros/$ROS_DISTRO/setup.bash cd ~/ros2_ws . install/local_setup.bash to the onboard PC, in our case the NUC).
- Section 3. 2. 7. NEW LDS-02 Configuration if you are using an LDS-02 (2022+ TurtleBot3 models). ROS2 packages default to the older LDS-01.
- Section 3. 3. OpenCR Setup can be followed, but we recommend attempting to bring-up the robot first and seeing if the current firmware on the OpenCR board is functional.
- Section 3. 5. 1. Bringup TurtleBot3 demonstrates run basic TurtleBot3 programs and test basic operation.
- You do not need to follow Step 1 of 3. 5. 1. if you are running onboard the robot.
- Step 2 of 3. 5. 1. recommends running
export TURTLEBOT3_MODEL=burger
. You will have to run this command every terminal emulator instance. Instead, we recommendexport TURTLEBOT3_MODEL=burger >> ~/.bashrc
to save this environmental variable.
- Section 3. 6. 1. Teleoperation is a good way to test functionality of your robot past simply running the
turtlebot3_bringup
node. If your robot does not respond to teleoperation commands, we've documented some common issues we ran into below:- Ensure the 12V power and the USB to the OpenCR board is plugged in. If only the USB is connected, ROS2 can successfully communicate with the board, but the board cannot send movements to the wheels. ROS2 will not display any error messages if this occurs.
- Ensure the USB port the OpenCR is plugged into has appropriate privileges (/dev/ttyACM0 by default).
chmod 777 /dev/ttyACM0
. This is also a common issue with the LDS-01/02 (/dev/ttyUSB0 by default)
The realsense-ros repository's README contains installation instructions for the Intel® RealSense™ SDK 2.0 and ROS2 Wrapper.
- Create a ROS2 workspace (this should have been completed in Step 2).
mkdir -p ~/ros2_ws/
cd ~/ros2_ws/
- Clone the patient-monitoring package and copy the
/src
directory to your ROS2 workspace.
git clone https://github.com/schofija/patient-monitoring
cp -r patient-monitoring/src/ ~/ros2_ws/
- Install dependencies
sudo apt-get install python3-rosdep -y
sudo rosdep init # "sudo rosdep init --include-eol-distros" for Eloquent and earlier
rosdep update # "sudo rosdep update --include-eol-distros" for Eloquent and earlier
rosdep install -i --from-path src --rosdistro $ROS_DISTRO --skip-keys=librealsense2 -y
- Build patient_monitoring and my_robot_interfaces package
colcon build --packages-select patient_monitoring && colcon build --packages-select my_robot_interfaces
- Source environment (do this in a different terminal than the one you ran
colcon build
in to avoid weird errors)
cd ~/ros2_was
. install/setup.bash
The turtlebot3_bringup
and realsense2_camera_node
nodes must be running for the patient_monitoring
nodes to function.
Option 1: Run the robot_launch
launchfile
cd ~/ros2_ws/src/launch
ros2 launch robot_launch.py
Option 2: Manually launch nodes
ros2 launch turtlebot3_bringup robot.launch.py
ros2 launch realsense2_camera rs_launch.py depth_module.profile:=1280x720x30 align_depth.enable:=true
Please ensure your environment is sourced to avoid 'package not found' errors.
- patient_watcher:
ros2 run patient_monitoring patient_watcher
Reads camera feed and detects humans on screen. Publishes position and landmark data (as well as other extrinsics) to the Person
message, which can be used by other ROS2 nodes.
- patient_watcher_yolo:
ros2 run patient_monitoring patient_watcher_yolo
Alternative version of patient_watcher
that utilizes yolov7-tiny. Currently incompatible with fall_detection
- patient_follower:
ros2 run patient_monitoring patient_follower
Uses positioning from Person
message. Attempts to keep human at the center of screen aswell and an optimal distance away from the robot.
- fall_detection:
ros2 run patient_monitoring fall_detection
Uses landmark data from Person
message to detect falling movements. Detected falls are pushed to a realtime database to display push notifications on the patient-monitoring android application (pmr-app-test)
User Research
Abnormal Motion Research
- FortyFive-Robot_ws (2019-2020 Capstone)