This repo contains a series of components for Unity XR teleoperation with ROS integration. It is designed to be run on a Quest 3 VR headset, and support OpenXR, hand tracking, Unity input system, and is built with Unity 2022.3.12f1.
For information on setting up Unity and opening this project Unity Quickstart, and for Quest information see Quest Quickstart.
On the ROS side, the custom TCP Endpoint needs to be run somewhere on the ROS network. This node is pretty lightweight so it can be run directly on a robot. Once the node is running the device running this app needs to connect to the ROS network, and the IP of the device running the TCP Endpoint needs to be set in the menu (see Menu for more information). If everything works, the menu should turn green, and data should be streaming between the app and ROS.
| Component | Description | Location | Preview | Status |
|---|---|---|---|---|
| Camera Viewer | Renders a ROS image stream to a floating image window | Assets/Components/CameraView |  |
Functional |
| Hands | Hand tracking and pose publishing over ROS, compatible with Ability hand models | Assets/Components/Hands |  |
Functional |
| Haptic | Bhaptic glove support plus controller haptics | Assets/Components/Haptics |  |
Functional |
| Headset Publisher | Publishes headset and hand poses on TF and Pose | Assets/Components/HeadsetPublisher |  |
Functional |
| Lidar | GPU rendering for LiDAR and PointCloud2 point viz from ROS | Assets/Components/Lidar |  |
Functional |
| Menu | Palm menu for interaction and toggling | Assets/Components/Menu |  |
Functional |
| NeRFViewer | Handheld viewer for rendering NeRFs and scene interaction | Assets/Components/NeRFViewer |  |
Functional |
| PosePublisher | Publishes poses and Joystick commands for locomotion | Assets/Components/PosePublisher |  |
Functional |
| Robots | Robot model manager, holds the meshes, materials and the scripts to swap between Anymal, ALMA, Standalone Dynaarm and Franka Panda | Assets/Components/Robots |  |
Functional |
| Stereo | Stereo camera rendering, renders to each eye for human depth perception | Assets/Components/StereoImage |  |
Functional |
| TF | WiP new TF system for managing robots and reorientation | Assets/Components/TFSystem | |
Incomplete |
| Voxblox | Voxel mesh rendering | Assets/Components/VoxBlox |  |
Functional |
| VR Debug | Debugging tools for VR, namely a console | Assets/Components/VRDebug |  |
Functional |
| VR Streamer | Streams a the VR view to a ROS topic (w/o AR view) | Assets/Components/VRStreamer |  |
Functional |
In general the scenes should have a few objects by default:
- Light source (usually the default directional light)
- MR Interaction Setup (this enables AR/VR support and acts as a camera)
- Debug canvas (will autolink to the menu and shows debug messages)
- Palm Menu (menu to interact and toggle things with)
- Root (The roof of the TF/object tree and tagged at 'root')
- Added versioning with display in debug mode
- Added standalone dynaarm, and franka panda robots
- Updated dynaarm for newer model (may still need simplification)
- Added robot manager for switching between robots
If you use this project in your work please cite this paper:
@article{wildersmith2024rfteleoperation,
author = {Maximum Wilder-Smith, Vaishakh Patil, Marco Hutter},
title = {Radiance Fields for Robotic Teleoperation},
journal = {arXiv},
year = {2024},
}