SRIP19: Self-Driving and Multi-task Learning

Content

• BDD_action_gt: use IMU and GPS info from BDD dataset to generate single action ground truth.
• multiple_action_labels: use AWS Mturk to label multiple actions and reasons of selected 12k BDD videos.
• data_info: contains names of train, test and validation datasets.
• mask-rcnn: Mask-RCNN model, forker from Facebook AI group and modified with action prediction.
• I3D: inflated Conv3D model, adapted to Pytorch 1.0 and our new annotated BDD multi-action dataset.
• maskrcnn-video: Using our customized I3D backbone with 640x360 image sequences input to extract glob features and roi features with selectors, performing end-to-end training.

Papers for reference

Self-Driving Review

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges.

Existing Self-Driving Datasets

• BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling
• The ApolloScape Dataset for Autonomous Driving
• The Cityscapes Dataset for Semantic Urban Scene Understanding

Multi-task Learning methods

• An Overview of Multi-Task Learning in Deep Neural Networks
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving
• UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-,and High-Level Vision using Diverse Datasets and Limited Memory
• Cross-stitch Networks for Multi-task Learning

Video Prediction in self-driving

• Trajectory prediction summary
• DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
  • Code: https://github.com/yadrimz/DESIRE
• Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
• Predicting Deeper into the Future of Semantic Segmentation
  • Code: https://github.com/facebookresearch/SegmPred
• Predicting Future Instance Segmentation by Forecasting Convolutional Features
  • Code: https://github.com/facebookresearch/instpred

Some Prediction models

Summary: Video prediction papers with code

• Deep Multi-scale video prediction beyond mean square error
  • Code in lua: https://github.com/coupriec/VideoPredictionICLR2016
  • Code in tf: https://github.com/dyelax/Adversarial_Video_Generation
• Prediction Under Uncertainty with Error-Encoding Networks
  • Code: https://github.com/mbhenaff/EEN
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
  • Code: https://github.com/coxlab/prednet
• Peeking into the Future: Predicting Future Person Activities and Locations in Video
  • Code: https://github.com/google/next-prediction

Some semantic segmentation approaches

Summary: Semantic segmentation papers with code

• Fast-SCNN: Fast Semantic Segmentation Network
  • Code: https://github.com/DeepVoltaire/Fast-SCNN
• DeeplabV3: Rethinking Atrous Convolution for Semantic Image Segmentation
  • Code: https://github.com/fregu856/deeplabv3

---

Useful github repo

• Mask-rcnn benchmark
• Inflated_convnets_pytorch
• pytorch_i3d with training