Implementing a handwritten digits recognition application on an ESP32-CAM board using Tensorflow Lite for Microcontrollers.
The goal here is to perform inference on the 28000 test images, from this Kaggle competition, on an ESP32-CAM board and to submit the results in a csv file automatically to Kaggle using the Kaggle API.
The 28000 images will be served to an http Client on ESP32 by a Python http Server (over a wifi connection), and in the same way each inference result will be sent back to the server. When the processing of all the 28000 images is done, a POST request will be sent by the ESP32 to the server so that the inference results can be submitted to Kaggle.
First of all, we need to build a machine learning model to perform the digits' recognition task. For that, I used TensorFlow and Keras in a Kaggle notebook to build, train and evaluate a simple Convolutional Neural Network model using the MNIST dataset provided by the Digit Recognizer kaggle competition .
After making sure that the model is working, we need to compress it so that it can be used for inference on memory-constrained devices like the ESP32. For that, we have to generate a TensorFlow Lite model, using one of the 2 techniques described in the notebook :
- Post-training quantization : Generate a TFLite model from the baseline model with and without quantization.
- Quantization-aware training : Generate the TFLite model from a quantization-aware model that was trained with the quantization step in mind.
Now that we have our TFLite model, we can generate a C file containing the model's weights and characteristics, which will be used by TensorFlow Lite for Microcontrollers, using the Xxd tool as described in the last step in the notebook.
Our TFLite Micro model is now ready to be deployed on the edge, we just need the TensorFlow Lite For Microcontrollers' library compiled for ESP32. To get the library, the best option is to use the Docker image provided by the following Github repository (Thank you @atomic14) to compile the library with the TensorFlow & ESP-IDF versions of your choice (I recommend using the latest stable versions).
The resulting TFLM library will be used with the Visual Studio Code extension PlatformIO which really facilities the building process for ESP32 applications. Here is an official PlatformIO guide that explains the different configuration options for ESP32 projects. Also, this project is build with the official Espressif IoT Development Framework (ESP-IDF) not the ESP32 Arduino Framework. The library must be placed in both the /components & /lib directories.
The platformio.ini file already contains the necessary configuration options to build & deploy this project on the ESP32-CAM, and it will be detected automatically once the project in opened in VSCode with the PlatformIO extension installed.