MLAB framework

Overview

Mlab is a framework designed to manage machine learning models in a production environment. More specifically, Mlab enables developers to upload new machine learning models, either manually through the Mlab dashboard or using an api-rest, to a framework where developers can create full pipelines of training and release machine learning models. When a model is released to the Mlab architecture, models can be actived or deactived in a specific host; hence, Mlab allows developers to create A/B test for the models. Once a model is released in a host, this can be consumed by business applications through a web service. Full documentation about web service is deployed by swagger as static documentation.

Table of content

1. Documentation
2. Getting started
3. Architecture
4. Requirements
5. Dashboard Orchestrator
   1. Install
   2. Usage
6. Worker
   1. Install
   2. Usage
7. Running with Docker
8. Status
9. License

Documentation

TODO

Getting started

Getting started with Mlab is quite simple. It has two main components: workers and an orchestator/dashboard. One worker instance is a webservice, whose main goal is load machine learning models in memory and keep the sincronization with the orchestator. On the other hand, the orchestrator is in charge of controlling the workers where the machine learning algorithm is loaded (remember that models can be loaded through the dashboard or REST-API).

Architecture

Requirements

sudo apt-get update
sudo apt-get install -y python3-pip virtualenvwrapper python3 nodejs npm 
sudo ln -s /usr/bin/nodejs /usr/bin/node

Dashboard Orchestrator

Install

mkvirtualenv --python=/usr/bin/python mlab_dashboard_env
cd dashboard/src/dashboard 
npm install 
./node_modules/bower/bin/bower install 
./node_modules/gulp/bin/gulp.js 
cd ${PROJECT_ROOT_DIRECTORY}/dashboard
#As mlab_dashboard_env virtual environment activated in the terminal session.
pip install .

Usage

To run the server, please execute the following using mlab_dashboard_env.

gunicorn -w 2 -b 0.0.0.0:5000 dashboard.app:app

Now, we can open the dashboard, in the uri: http://localhost:5000/dashboard

Worker

Install

mkvirtualenv --python=/usr/bin/python mlab_worker_env
cd worker
#As mlab_worker_env virtual environment activated in the terminal session.
pip install .

Usage

To run the server, please execute the following using mlab_dashboard_env.

gunicorn -b 0.0.0.0:9090 -w 4 --config=python:worker.application.conf.gunicorn_conf worker.app:app

Now, we can open the dashboard, in the uri: http://localhost:5000/dashboard

Running with Docker

To run the server on a Docker container. Please, execute the following from the root directory:

# run docker-compose
docker-compose up --build

The dashboard service is running by default on http://localhost:5000/dashboard The worker swagger documentation is running on port 9090.

Status

This project is in the earliest phase of its development. Use it under your own responsibility.

License

Mlab framework
Copyright (C) 2018 Bluekiri BigData Team <bigdata@bluekiri.com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

Contributors

Thanks goes to these wonderful people:

Cristófol Torrens 🤔 🐛	David Martín 🤔 🐛	Daniel bestard 🌍 🤔	Joan Martín Miralles 💻