Getting started with VeloxMR

Installation

Firstly follow the steps of the README file. Additionally add those env variables to your ~/.bashrc

Change vicente for your user

export LDFLAGS+="-L/usr/local/lib -L/home/vicente/sandbox/lib"
export PATH="/home/vicente/sandbox/bin:$PATH"
export LIBRARY_PATH="/home/vicente/sandbox/lib:/usr/local/lib"
export CPLUS_INCLUDE_PATH="/home/vicente/sandbox/include/"
export LD_LIBRARY_PATH="/home/vicente/sandbox/lib"

Also, install eclipsed script helped to let you start/stop/debug your VeloxMR instance

Usage

At this moment we have to update our application files manually to each of the servers running VeloxMR:

                                                                     +
                        Client                                       |                        Remote Nodes
                                                                     |
                                                                     |
                        +---------------+                            |
                        |               |                            |
                        |               |                            |
                        | App.cpp       |                            |    +---------+    +---------+    +---------+    +---------+
                        |               |                            |    | VMR     |    | VMR     |    | VMR     |    | VMR     |
                        |               |                            |    | Node    |    | Node    |    | Node    |    | Node    |
                        +---------------+                            |    +---------+    +---------+    +---------+    +---------+
                             |      |                                |         |              |              |              |
                             |      |                                |         v              v              v              v
                             |      |                                |      Executes       Executes       Executes       Executes
       For the client        |      |          to upload in each     |         +              +              +              +
                             |      |          of the servers        |         |              |              |              |
                             |      |                                |         v              v              v              v
     +---------------+       |      |         +---------------+      |    +---------+    +---------+    +---------+    +---------+
     |               |       |      |         |               |      |    | App.so  |    | App.so  |    | App.so  |    | App.so  |
     |               |       |      |         |               |      |    +----^----+    +----^----+    +----^----+    +----^----+
     |  binary       | <-----+      +-------> | app.so        |      |         |              |              |              |
     |               |                        |               +-------------------------------------------------------------+
     |               |                        |               |      |                   Scp (copied to each node)
     +---------------+                        +---------------+      |
                                                                     |
       The client will just                                          |
       run ./binary                                                  |
                                                                     |
                                                                     +

Creating binary and dynamic library

Soon, we will have a new API. But as for the moment we have to create our dynamic library manually.

Assuming that we want to create the dynamic library for the example wc application, those would be the commands:

g++ -c -std=c++14 -Wall -Werror -fpic ../EclipseMR/src/targets/wc.cc
gcc -shared -fPIC -o wc.so wc.o

Then we have to scp to each of the remote servers (change vicente for your user name):

scp wc.so ferrari01:/scratch/vicente/app/
scp wc.so ferrari02:/scratch/vicente/app/
scp wc.so ferrari03:/scratch/vicente/app/

Configuration file reference:

{
 "name"    : "Raven cluster eclipseMR file",
 "max_job" : 1024,

 "log" : {
   "type" : "LOG_LOCAL6",
   "name" : "ECLIPSE"
 },

 "path" : {
   "scratch"    : "/scratch/vicente/storage",
   "idata"      : "/scratch/vicente/idata",
   "metadata"   : "/scratch/vicente/",
   "applications"   : "/scratch/vicente/app"
 },

 "cache" : {
   "numbin" : 100,
   "size" :   200000,
   "concurrency" : 1
 },

 "network" : {
   "ports" : {
     "internal" : 8008,
     "client"   : 8009
   },

   "serialization"  : "binary",
   "iface"          : "em1",
   "nodes"          : [
     "10.20.13.123",
     "10.20.13.124",
     "10.20.13.125"
   ]
 },

 "filesystem" : {
   "block"  : 137438953,
   "buffer" : 512,
   "replica" : 1
 },

 "mapreduce" : {
   "reduce_slot" : 1,
   "iblock_size" : 1024,
   "write_buf_size" : 512
 }
}