LipReading

Visual Speech Recognition using deep learing methods

Fig. 1: LipNet architecture.

Quick Links

• Dependencies
• Getting Started
• Training
• DataSet
• References
• License

Dependencies

• Install Dependencies $ pip install -r requirements.txt
• Download Pretrained Weights Here

Getting Started

• Project Structure

.
├── src
│   ├── nn
│   │   └── model.py
│   └── utils
│       ├── dataProvider.py
│       ├── labelConverter.py
│       ├── losses.py
│       ├── misc.py
│       ├── trainUtils.py
│       └── transforms.py
├── config.py
└── train.py

Training

CTC Loss

CTC takes a sequence $\ H = h_1 , . . . , h_T$ , where $\ T$ is the sequence length, and outputs the probability of $\ \pi$, which is defined as

$$P(\pi|H) = \prod_{t = 1}^T y_{{\pi}_t}^t$$

where $\ y_{{\pi}_t}^t$ is the probability of generating character $\ \pi_t$ at each time step $\ t$.

Fig. 2: Model Training History.

DataSet

• You Can Find Dataset Here

References

• LIPNET: End-to-End Sentence-Level Lipreading

🛡️ License

Project is distributed under MIT License