Within three years deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software. The field took off last year when Tony Beltramelli introduced the pix2code paper and Airbnb launched sketch2code. Currently, the largest barrier to automating front-end development is computing power. However, we can use current deep learning algorithms, along with synthesized training data, to start exploring artificial front-end automation right now!
Click this button to open a Workspace on FloydHub that will train this model.
In this notebook, we’ll build a neural network to code a basic a HTML and CSS website based on a picture of a design mockup.
Image from the Blog
We’ll use a dataset of generated bootstrap websites from the pix2code paper. By using Twitter’s bootstrap, we can combine HTML and CSS and decrease the size of the vocabulary.
Instead of training it on the bootstrap markup, we’ll use 17 simplified tokens that we then translate into HTML and CSS. The dataset includes 1500 test screenshots and 250 validation images. For each screenshot there are on average 65 tokens, resulting in 96925 training examples.
By tweaking the model in the pix2code paper, the model can predict the web components with 97% accuracy (BLEU 4-ngram greedy search).
We will:
- Preprocess webpage images and the code related HTML for this mixed NLP-CV task
- Build and train the
pix2codemodel using Keras and Tensorflow - Evaluate our model on the test set