An application that recommends personalised skincare and makeup products based on the skin metrics inferred from user’s selfie, using Computer Vision algorithms. Image processing and CNN models were utilized in the extraction of Skin Tone, Skin Type and Acne concern level. With these data points, products with high cosine-similarity were recommended in the order of their relevance. The web application was written in React-Flask.
/ - ImageInput
This is the initial page, the user is prompted to take a selfie. Once the user grants permission to use their device's camera, a realtime video of their camera (profile : user) runs in 4:3 aspect ratio. To ensure that the image is taken with precautions such as only one face is there in the image, proper luminance and that the majority of the image is populated by the user’s face, Face recogniton was implemented using face-api.js. The user is given text prompts as directions. When the user takes a selfie, is then redirected to /form.
/form - Form
The results of the skin metrics inferred from the user's selfie are presented as prefilled form elements. User can alter these values, along with selecting their other skin concerns from the given list. Once the form is submitted, the page is redirected to /recs.
/recs - Recommendations
Here the recommended products are presented in the form of cards, with their details. The cards when clicked redirect to their appropriate product page.
[PUT]/upload
Accepts a base64 image, converts it into png and feeds that into the pipeline yielding predictions for skin tone, type and acne, and returns these attributes in JSON format.
[PUT]/recommend
Accepts request body containing details about the user's skin type, tone, and concerns, and returns top 5 recommended skincare products from each category in JSON format.
Obtaining skin tone consists of :
- Detecting and extracting skin pixels
- Classifying those color values into the appropriate skin tone class
This paper was followed in extracting the skin pixels. Skin detection has three major steps i.e.., initial segmentation, prediction of skin pixels and k-means clustering. Initial segmentation is applied with the threshold value → average of TOTSU and TMAX. These values are aquired from the image histogram of the grayscale image
The thresholded image is then converted to, HSV and YCrCb color spaces. These colorspaces are less sensitive to light conditions. Potential skin color pixels are selected with :
(Hue <= 170) and (140 <= Cr <= 170) and (90 <= Cb <= 120).
A binary image is formed with the selected pixels.
We defined special dataset made of input features in order to cluster pixels on an image. This dataset contains some components of two color spaces (Hue, Cr, Cb), positions of pixels on the image (Xp, Yp) and rough estimation of skin pixels (I). Since all information is contained in a dataframe, we converted all six aforementioned components (Cr, Cb, Hue, Xp, Yp and I) into appropriate vectors.
Image pixels are clustered into three clusters: background, foreground and skin pixels. We used square Euclidean measure as a distance. Aproximated skin pixels (I), determine which cluster represents skin.
The mean color values obtained from the cluster are then used for classifying the tone into Fitzpatrick scale using a KNN model. The model was trained using the color values dataset gotten from the image dataset of Von Luschan's chromatic scale.
Facial skin type is inferred by analysing the picture with the utilization of Convolutional Neural Network (CNN) which classifies the image into three classes : Dry, Oily and Normal. To increase the accuracy of the model, transfer learning (EfficientNet B0) is used with training accuracy 87.10 % and validation accuracy 80%. The main concern we faced here is the amount of quality face images.
One of the skin metrics, Acne concern level is classified into three classes : Low, moderate and Severe. The model structure possess similar architecture to Skin types CNN model with the use of transfer learning which provides us with an accuracy of approx 68% over both training and validation image sets. The dataset we used is obtained from Kaggle.
EfficientNet Architecture :
Link to Dataset : https://www.kaggle.com/rutviklathiyateksun/acne-grading-classificationdataset
Given the user's skin metrics and concerns, how do we fetch relevant skincare products that shall possibly address his/her skin concerns?
Since the dataset that has been used contains data straight from the Myntra Beauty Section itself, each product in the dataset is associated with skin tone and one/more skin concerns (acne, blemishes, redness, etc).
A good strategy would be to fetch those products whose product attributes (skin tone + concerns) is similar to the user's skin metrics and concerns. Mathematically, this similarity can be quantified in the form of cosine similarity between product feature vector and user skin attribute vector.
The key idea is: To find relevant skincare products from a particular category, given the user's skin features, we simply obtain the top n values of similarity(skin vector, product vector) for the products in dataset belonging to that said category, and return the products corresponding to those values.
Clone this repo, head to the root directory and create a virtual env.
$ pip install -r requirements.txt
Then,
$ cd backend
$ python app.py
After that,
$ cd frontend
$ npm install
$ npm start
The web app can be accessed at localhost:3000
Frontend : React
Backend : Flask, OpenCV, Tensorflow