A Deep Neural Network Plays Crew 2

• Adapted and inspired by sentdex’s Python Plays GTA V series.

• Demo: Second version

Objective

• Crew 2 is a open world racing game which has a relatively big map which takes approximately 1 hour to drive across. In this project, a deep neural network model is trained to drive as fast as possible by learning from front camera of the vehicle and pressed keys while playing.
• This is a self-playing car project rather than a self-driving car application. The project's main objective is building & improving a low-cost real-time multi-task learning model on edge.

Hardware Specs

• GPU: GeForce GTX 1060 Max-Q
• CPU: Intel i7-7700HQ
• RAM: 16GB

Dataset Description

• Dataset Size: 88 GB Training, (80%) & 22 GB Testing (20%) and ~10 GB Validation. Distribution of classes:
• Multilabel dataset:

• Multiclass dataset:

• Dataset Shape: 180.800 training, 45.200 testing & 20.000 validation samples

Demo Videos

First version Second version

Training Details:

• Dataset: 40 GB Training (80%) & Testing (20%) and 10 GB Validation.
• Model Architecture: DenseNet121 with fine-tuning on ImageNet weights.
• Duration: Approx. 10 Hours, 8 Epochs.

Why the model was fine-tuned on ImageNet weights?

• From the paper “How transferable are features in deep neural networks?” by J. Yosinski et al., even big datasets can benefit from transfer learning which improves generalization.
• Presentation of the review of the paper

TODO:

• New dataset with 1M samples
• Image compression for dataset storage
• Balance dataset
• There're 9 classes for each pressed key but these are not mutually exclusive. Convert the objective to multi-label multiclass classification problem.
• Fix storage problems
• Shuffle big dataset
• Create a Keras data generator
• Optimize training time by using the reference
• Use mixed precision for training: cannot be used efficiently with GTX 1060 Max Q due to 7.0+ compute capability requirement
• Preprocess all dataset before training:

%timeit preprocess_input(df_feature) >> 1.01 s ± 8.95 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

• Optimize inference time, current: 50ms, target: 10 ms
• Optimize screen capturing
• Adapt post-processing to main objective
• Add object detection for crash prevention

Disclaimer

• The project aims to review real-time deep learning models while respecting the copyright holders in terms of fair use. This repository is prepared for the review of recent the state-of-art methods that are available for end-to-end deep learning with nonprofit educational purposes only. The dataset was generated by screen capturing and no game files were used during the project. The generated dataset from screenshots and training weights are neither shared nor stored. The end result is not used for personal benefit in the online game.