jupyter/terraform_local_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# README (Ignore if you are running on Mac/Linux)\n",
    "\n",
    "If you are running on Windows, make sure you have started the Jupyter Notebook in a Bash shell.\n",
    "Moreover, all the requirements below must be installed in this Bash (compatible) shell.\n",
    "\n",
    "This can be achieved as follows:\n",
    "\n",
    "1. Enable and install WSL(2) for Windows 10/11 [official documentation](https://docs.microsoft.com/en-us/windows/wsl/install)\n",
    "    * On newer builds of W10/11 you can install WSL by running the following command in an *administrator* PowerShell terminal. Which will install by default an Ubuntu instance of WSL.\n",
    "    ```bash\n",
    "   wsl --install\n",
    "    ```\n",
    "2. Start the Ubuntu Bash shell by searching for `Bash` under Start, or by running `bash` in a (normal) PowerShell terminal.\n",
    "\n",
    "Using a Bash terminal as started under step 2 above, you can install the Requirements as described below as if you are running it under Linux or Ubuntu/Debian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Requirements\n",
    "These requirements may also be installed on Windows, however, development has only been tested on Linux/macOS.\n",
    "\n",
    "Before we get started, first make sure to install all the required tools. We provide two lists below, one needed for setting up the testbed. And one for developing code to use with the testbed. Feel free to skip the installation of the second list, and return at a later point in time.\n",
    "\n",
    "\n",
    "### Deployment\n",
    "\n",
    " > ⚠️ All dependencies must be installed in a Bash-compatible shell. For Windows users also see [above](#read-me)\n",
    "Make sure to install a recent version of each of the dependencies.\n",
    "\n",
    "\n",
    " * (Windows only) Install every dependency in a Windows Subsystem for the Linux, Bash shell (see also README above).\n",
    " * MiniKube\n",
    " * Kubectl (>= 1.22.0)\n",
    " * Helm (>= 3.9.4)\n",
    " * Terraform (>= 1.2.8)\n",
    " * Python3.9/10\n",
    "   * jupyter, ipython, bash_kernel\n",
    "```bash\n",
    "pip3 install -r requirements-jupyter.txt\n",
    "python3 -m bash_kernel.install\n",
    "```\n",
    "\n",
    "### Development\n",
    "For development, the following tools are needed/recommended:\n",
    "\n",
    " * Docker (>= 18.09).\n",
    "    - If you don't have experience with using Docker, we recommend following [this](https://docs.docker.com/get-started/) tutorial.\n",
    "    - ⚠️Make sure that you have BuildX installed, follow the official GitHub [page](https://github.com/docker/buildx#installing).\n",
    " * Python3.9\n",
    " * pip3\n",
    " * (Recommended) JetBrains PyCharm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Preparation\n",
    "\n",
    "To make sure we can request resources on Google Cloud Platform (GCP), perform the following;\n",
    "\n",
    "1. Make sure to use the `Bash` kernel, not a Python or other kernel. For those on windows machines, make sure to launch the `jupyter notebook` server from a bash-compliant command line, we recommend Windows Subsystem for Linux.\n",
    "\n",
    "⚠️ Make sure to run this Notebook within a cloned repository, not standalone/downloaded from GitHub. This allows you to see changes you made more easily!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Deployment\n",
    "\n",
    "⚠️ This notebook assumes that commands are executed in order. Executing the provided commands multiple times should not result in issues. However, re-running cells with `cd` commands, or altering cells (other than variables as instructed) may result in unexpected behaviour.\n",
    "\n",
    "## Getting started\n",
    "\n",
    "First, we will set a few variables used **throughout** the project. We set them in this notebook for convenience, but they are also set to some example default values in configuration files for the project. If you change any of these, make sure to change the corresponding variables as well in;\n",
    "\n",
    "* [`../terraform/terraform-gke/variables.tf`](../terraform/terraform-gke/variables.tf)\n",
    "* [`../terraform/terraform-dependencies/variables.tf`](../terraform/terraform-dependencies/variables.tf)\n",
    "\n",
    "\n",
    "> ⚠️ As you have changed the `PROJECT_ID` parameter to a unique project name, also change the `project_id` variable in the following files. This allows you to run `terraform apply` without having to override the default value for the project.\n",
    "\n",
    "> ℹ️ Any variable changed here can also be provided to `terraform` using the `-var` flag, i.e.  `-var terraform_variable=$BASH_VARIABLE`. An example for setting the `project_id` variable is also provided later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# VARIABLES THAT NEEDS TO BE SET\n",
    "\n",
    "TERRAFORM_DEPENDENCIES_DIR=\"../terraform/terraform-dependencies-local\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Starting MiniKube\n",
    "First, let us make sure that the MiniKube cluster is started properly. Additionally, we will activate the local clusters' docker environment, allowing us to directly build containers to be used within the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "minikube start\n",
    "\n",
    "# This will activate the docker environment for the current shell.\n",
    "# When deploying locally, make sure to build the (tagged) docker container with this\n",
    "# environment active, as otherwise the container might contain old code/not available.\n",
    "eval $(minikube docker-env)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "\n",
    "When the previous command completes successfully, we can start the deployment. Depending on any changes you may have done, this might take a while."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Installing dependencies\n",
    "Lastly, we need to install the dependencies on our cluster. First change the directories, and then run the `init`, `plan` and `apply` commands as we did for creating the GKE cluster.\n",
    "\n",
    "Init the directory, to initialize the Terraform module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR init -reconfigure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Check to see if we can plan the deployment. This will setup the following:\n",
    "\n",
    "* Kubeflow training operator (used to deploy and manage PyTorchTrainJobs programmatically)\n",
    "* NFS-provisioner (used to enable logging on a persistent `ReadWriteMany` PVC in the cluster)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR plan -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "When the previous command completes successfully, we can start the deployment. This will install the NFS provisioner and Kubeflow Training Operator dependencies\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR apply -auto-approve -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Deploying extractor\n",
    "\n",
    "Lastly, we deploy the extractor pod, which also provides PVCs which can be used for artifact retrieval.\n",
    "\n",
    "Retrieval can be done by running\n",
    "\n",
    "```bash\n",
    "EXTRACTOR_POD_NAME=$(kubectl get pods -n test -l \"app.kubernetes.io/name=fltk.extractor\" -o jsonpath=\"{.items[0].metadata.name}\")\n",
    "kubectl cp -n test $EXTRACTOR_POD_NAME:/opt/federation-lab/logging ./logging\n",
    "```\n",
    "\n",
    "For copying from the extractor path `/opt/federation-lab/logging` to a directory locally named `logging`.\n",
    "\n",
    "First build the docker container, following the instructions of the [readme](https://github.com/JMGaljaard/fltk-testbed#creating-and-uploading-docker-container).\n",
    "\n",
    "\n",
    "N.B. Make sure to have setup a working authentication provider for docker, such that you can push to your repository."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Run this in a terminal in the content-root directory (so `fltk-testbed` if the project name was not altered).\n",
    "```bash\n",
    "python3 -m venv venv\n",
    "source venv\n",
    "pip3 install -r requirements-cpu.txt\n",
    "python3 -m fltk extractor configs/example_cloud_experiment.json\n",
    "```\n",
    "\n",
    "Make sure to have run `gcloud auth configure-docker` in an external terminal.\n",
    "\n",
    "Make sure to allow docker to build/push/run without `sudo` [link](https://cloud.google.com/artifact-registry/docs/docker/authentication).\n",
    "\n",
    "\n",
    "⚠️‼️Make sure to update `fltk-testbed` in the commands below to your GCP project name, incase you want to debug something locally, and have already updated chart names to account for this. Otherwise, deployment will likely fail.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Build the docker container with buildkit. Make sure you have Docker Desktop running on Windows/MacOS\n",
    "DOCKER_BUILDKIT=1 docker build --platform linux/amd64 ../ --tag gcr.io/fltk-testbed/fltk\n",
    "# This will automatically be available in your cluster."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Install the extractor, and set the projectName to $PROJECT_ID.\n",
    "# In case you get a warning regarding the namespace test, this means that the dependencies have not been properly installed.\n",
    "# Make sure to check whether you have enough resources available, and re-run the installation of dependencies. (see above).\n",
    "\n",
    "# Deploy extractor, in test namespace with updated image reference (--set overwrites values from `fltk-values.yaml`).\n",
    "helm install extractor ../charts/extractor -f ../charts/fltk-values.yaml --namespace test --set provider.projectName=fltk-testbed"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Testing the deployment\n",
    "\n",
    "To make sure that the deployment went OK, we can run the following command to test whether we can use Pytorch-Training operators.\n",
    "\n",
    "This will create a simple deployment using a Kubeflow pytorch example job.\n",
    "\n",
    "This will create a small (1 master, 1 client) training job on mnist on your cluster. You can follow the deployment by navigating to your cluster on [cloud.google.com](cloud.google.com)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# This cell is optional, but the next shell should show that a pytorch train job is created.\n",
    "kubectl create -f https://raw.githubusercontent.com/kubeflow/training-operator/master/examples/pytorch/simple.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Retrieve all CRD Pytorchjob from Kubeflow.\n",
    "kubectl get pytorchjobs.kubeflow.org --all-namespaces\n",
    "\n",
    "# Alternatively, we can remove all jobs, this will remove all information and logs as well.\n",
    "# kubectl delete pytorchjobs.kubeflow.org --all-namespaces --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Cleaning up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling down the cluster\n",
    "\n",
    "This is the preferred way to scale down.\n",
    "Scale node pools down to prevent idle resource utilization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "minikube stop"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "title": "Tutorial (Terraform)"
 },
 "nbformat": 4,
 "nbformat_minor": 1
}