SiT Dataset: Socially Interactive Pedestrian Trajectory Dataset for Social Navigation Robots

Example of SiT dataset

Updates

• [2024-10] The Google Form link for download has been changed. Google Form
• [2024-06] The final dataset is released! Download it via the link at the end of the Google Form. Google Form
• [2023-09] Our paper is accepted to NeurIPS 2023 Dataset and Benchmark Track! Paper Link
• [2023-06] Semantic map data of SiT Dataset released on Github.
• [2023-05] We opened SiT Dataset Github.
• [2023-05] We opened SiT Dataset Website.

Upcomings

• [2024-09] Trained weights and updated code for 3D object detection and Trajectory prediction release on public.

Overview

Our Social Interactive Trajectory (SiT) dataset is a unique collection of pedestrian trajectories for designing advanced social navigation robots. It includes a range of sensor data, annotations, and offers a unique perspective from a robot navigating crowded environments, capturing dynamic human-robot interactions. It's meticulously organized for training and evaluating models across tasks like 3D detection, 3D multi-object tracking, and trajectory prediction, providing an end-to-end modular approach. It includes a comprehensive benchmark and exhibits the performance of several baseline models. This dataset is a valuable resource for future pedestrian trajectory prediction research, supporting the development of safe and agile social navigation robots.

Robot Platform & Sensor Setup

• Ubuntu 18.04 LTS
• ROS Melodic
• Clearpath Husky UGV
• Velodyne VLP-16 * 2
• RGB Camera Basler a2A1920-51gc PRO GigE * 5
• MTi-680G IMU & GPS * 1
• VectorNAV VN-100 IMU * 1

Ground Truth

We provide GT boxes for 2D and 3D data as below.

• 2D: Class name, Track ID, Camera number, Top left X coordinate, Top left Y coordinate, Width (w), and Height (h)
• 3D: Class name, Track ID, Height (h), Length (l), Width (w), X, Y, and Z coordinates, and rotation (rot).

Benchmarks

We provide benchmarks and weights of trained models for 3D pedestrian detection, 3D Multi-Object Tracking, Pedestrian Trajectory Prediction and End-to-End Motion Forecasting.

3D Object Detection

Methods	Modality	mAP ↑	AP(0.25) ↑	AP(0.5) ↑	AP(1.0) ↑	AP(2.0) ↑	Trained
FCOS3D	Camera	0.170	0.000	0.037	0.219	0.423	Gdrive
BEVDepth	Camera	0.270	0.019	0.183	0.361	0.516	Gdrive
PointPillars	LiDAR	-	-	-	-	-	TBD
CenterPoint-P	LiDAR	-	-	-	-	-	TBD
CenterPoint-V	LiDAR	-	-	-	-	-	TBD
Transfusion-P	Fusion	-	-	-	-	-	TBD
Transfusion-V	Fusion	-	-	-	-	-	TBD

3D Multi-Object Trajectory Tracking

Method	sAMOTA↑	AMOTA↑	AMOTP(m)↓	MOTA↑	MOTP(m)↓	IDS↓
PointPillars + AB3DMOT	-	-	-	-	-	-
Centerpoint Detector + AB3DMOT	-	-	-	-	-	-
Centerpoint Tracker	-	-	-	-	-	-

Pedestrian Trajectory Prediction

Name	Map	ADE5 ↓	FDE5 ↓	ADE20 ↓	FDE20 ↓	Trained
Social-LSTM	X	1.781	2.816	1.683	2.722	TBD
Y-NET	X	-	-	-	-	TBD
Y-NET	O	-	-	-	-	TBD
NSP-SFM	X	1.451	2.329	0.812	1.331	TBD
NSP-SFM	O	0.883	1.809	0.624	1.237	TBD

End-to-End Pedestrian Motion Forecasting

Method	mAP ↑	mAPf ↑	ADE5 ↓	FDE5 ↓	Trained
Fast and Furious	-	-	-	-	TBD
FutureDet-P	-	-	-	-	TBD
FutureDet-V	-	-	-	-	TBD

Download Dataset

• Download Full dataset via the link at the end of the Google Form. Google Form

License

The SiT dataset is published under the CC BY-NC-ND License 4.0, and all codes are published under the Apache License 2.0.

Acknowledgement

The SiT dataset is contributed by Jongwook Bae, Jungho Kim, Junyong Yun, Changwon Kang, Junho Lee, Jeongseon Choi, Chanhyeok Kim, Jungwook Choi, advised by Jun-Won Choi.

We thank the maintainers of the following projects that enable us to develop SiT Dataset: MMDetection by MMLAB