29Apr2021_MLOps_II_Laboratory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W_RuDIS7w9SF"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/datarobot-community/DRU-MLOps/blob/master/29Apr2021_MLOps_II_Laboratory.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pcHrfkRkwUDC"
   },
   "source": [
    "# **MLOps II Laboratory**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cLTGmStO-gGe"
   },
   "source": [
    "Welcome to the MLOps II Hands-On Lab!\n",
    "\n",
    "**Pre-requisites:**\n",
    "1. You will need a DataRobot account and an API key.  \n",
    "2. Add your API Key to the first cell in the notebook. The API Key is found in the Developer Tools which is located on the profile icon in the DataRobot GUI App.\n",
    "3. Once you create a model package and deploy it, you will need the model ID and deployment ID\n",
    "\n",
    "\n",
    "**Documentation:**\n",
    "\n",
    "The MLOps Agent tarball includes documentation in the /docs folder.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TiQhSVSe8d7R"
   },
   "source": [
    "### ***You will complete certain lines of code in this notebook to provide the necessary functionality!***\n",
    "\n",
    "HINTS: \n",
    "* Shell commands that take no parameters are shown as ___\n",
    "* API calls that take no parameters are shown as \\_\\_\\_()\n",
    "* API calls that take 1 parameter are shown as \\_\\_\\_(\\_\\_\\_)\n",
    "* API calls that take 2 parameters are shown as \\_\\_\\_(\\___ , \\_\\__)\n",
    "* You get the idea.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KmXI8EwnKs2R"
   },
   "source": [
    "# 1.- Create and deploy a remote model package via the GUI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5p2RSJQekkJS"
   },
   "source": [
    "This is something that you have done already via the MLOps GUI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n73VMdJBI8-t"
   },
   "source": [
    "# 2.- Specify Model ID and Deployment ID\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ooOxNPA0pWhR"
   },
   "source": [
    "We need to supply the Deployment ID and Model ID found in the code sample provided in MLOps under \"Predictions\" -> \"Monitoring\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "La924wIpzE0q"
   },
   "outputs": [],
   "source": [
    "DEPLOYMENT_ID = \"\"\n",
    "MODEL_ID = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AP50QKUSlf3N"
   },
   "source": [
    "# 3.- Add your API_KEY and the location of the DataRobot instance you are using.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xO-YkhlNMJ0d"
   },
   "outputs": [],
   "source": [
    "import yaml\n",
    "import requests\n",
    "import re\n",
    "API_KEY = \"\"\n",
    "DR_URL = \"https://app.datarobot.com\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hH8kNamy0fQe"
   },
   "source": [
    "The following two shell commands will show you \\\n",
    "a) where we are within the Colab runtime and \\\n",
    "b) what is contained within it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SKt1hmm_CXlF"
   },
   "outputs": [],
   "source": [
    "% pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CQYHjeTOCZrj"
   },
   "outputs": [],
   "source": [
    "% ls -al"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qSB9WHitSpg8"
   },
   "source": [
    "# 4.- Download the MLOps Agent tarball to the local Colab directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "INs7K0_Fpjax"
   },
   "outputs": [],
   "source": [
    "# This cell downloads the MLOps Agent tarball\n",
    "url = DR_URL + \"/api/v2/mlopsInstaller\"\n",
    "\n",
    "headers = {'Authorization': 'Bearer {}'.format(API_KEY)}\n",
    "response = requests.request(\"GET\", url, headers=headers)\n",
    "if 'UNAUTHORIZED' in response.reason:\n",
    "    print('Put your real API key in')\n",
    "with open(\"mlops-agents.tar.gz\", \"wb\") as f:\n",
    "    f.write(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bCy3QeyFZ7rW"
   },
   "outputs": [],
   "source": [
    "# Lets grab the filename which has the latest version of the tarball\n",
    "d = response.headers['content-disposition']\n",
    "fname = re.findall(\"filename=(.+).tar.gz\", d)[0]\n",
    "n = fname.rfind(\"-\")\n",
    "filename = fname[:n]\n",
    "filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_DYJydZ-Cdey"
   },
   "outputs": [],
   "source": [
    "% ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7UFVSkdw1qpJ"
   },
   "source": [
    "As shown by the output of the previous shell command, we now have the MLOps Agent tarball within the runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gk7-p5MR8dTR"
   },
   "source": [
    "# 5.- Untar the MLOPs Agent tarball, and then create a tmp directory to spool the predictions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gVWLKh-CMQBv"
   },
   "outputs": [],
   "source": [
    "# Untar the tarball\n",
    "!tar -xvf /content/mlops-agents.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A4ZF00o-8dTW"
   },
   "outputs": [],
   "source": [
    "# Here we create the directory where the spool file will be located\n",
    "# This is where the MLOps Agent will look for prediction data\n",
    "%cd $filename\n",
    "!mkdir -p /tmp/ta\n",
    "%ls -al"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iFDxyVAx8dTZ"
   },
   "source": [
    "# 6.-  Install the MLOps library.\n",
    "\n",
    "### The tarball contains a Wheel file that wiil be used to install the MLOps Agent Libraries:  \n",
    "### **lib/datarobot_mlops-7.1.8-py2.py3-none-any.whl** \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vZmRJck88dTa"
   },
   "outputs": [],
   "source": [
    "# We now install the MLOps Agent Libraries\n",
    "!pip install lib/datarobot_mlops-7.1.8-py2.py3-none-any.whl   ##If you have a newer version of the agent, this could be different filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install lib/datarobot_mlops_connected_client-7.1.8-py2.py3-none-any.whl   ##If you have a newer version of the agent, this could be different filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vHnHZ-bh8dTd"
   },
   "source": [
    "# 7.- Edit mlops.agent.conf.yaml\n",
    "\n",
    "This file contains the properties used in the configuration of the MLOps service.  For this notebook, you will only need to set the DR host and your API token.\n",
    "\n",
    "For this purpose, we will edit the Configuration YAML file by reading it into a dictionary, modifying the corresponding fields in it, and then writing this dictionary back to the YAML file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QQr9bvXW8dTe"
   },
   "outputs": [],
   "source": [
    "with open(r'conf/mlops.agent.conf.yaml') as file:      # read the yaml file as a dictionary\n",
    "    documents = yaml.load(file)\n",
    "\n",
    "# Set your DR host:\n",
    "documents['mlopsUrl'] = DR_URL                         # set the required values in this dictionary\n",
    "\n",
    "# Set your API token\n",
    "documents['apiToken'] = API_KEY\n",
    "\n",
    "with open('conf/mlops.agent.conf.yaml', \"w\") as f:     # write back the dictionary to the yaml file\n",
    "    yaml.dump(documents, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4tnrBpIXoP0n"
   },
   "source": [
    "In this notebook we will use FS_SPOOL as the messaging channel. More sophisticated monitoring will likely use other channels.\n",
    "\n",
    "channelConfigs:\n",
    "   - type: “FS_SPOOL”\n",
    "     details: {name: “bench”, spoolDirectoryPath: “/tmp/ta”}\n",
    "   - type: “SQS_SPOOL”\n",
    "     details: {name: “sqsSpool”, queueUrl: “https://SQS_URL”}\n",
    "   - type: “PUBSUB_SPOOL”\n",
    "     details: {name: “pubsubSpool”, projectId: “yourprojectId”, topicName: “yourtopicName”}\n",
    "   - type: “RABBITMQ_SPOOL”\n",
    "     details: {name: “rabbit”, queueName: “rabbitmq”, queueUrl: “https://SQS_URL”}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7HJ4TMHkIl9a"
   },
   "source": [
    "Verify the changes in the mlops.agent.conf.yaml.  You should see the correct MLOps URL and API token.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oaII-yRZ8dTg"
   },
   "outputs": [],
   "source": [
    "print(open('conf/mlops.agent.conf.yaml').read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E-z9kqii8dTp"
   },
   "source": [
    "# 8.- Start the agent and get its status\n",
    "\n",
    "The following shell commands are required to \\\n",
    "a) start the MLOps Agent service. \\\n",
    "b) get the status of the MLOps Agent service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pFfZ0Wsg8dTq"
   },
   "outputs": [],
   "source": [
    "# Start the agent\n",
    "!___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3m8RqUBV8dTt",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Get agent status\n",
    "!___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J2hQkAYG8dT0"
   },
   "source": [
    "# 9.- Load sample data and split it into training and testing sets \n",
    "\n",
    "* The training data is the exact same one used to train the model pipeline that will be used in this laboratory \\\n",
    "* **The test data will play the role of the scoring data** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yqI3KAE38dT1"
   },
   "outputs": [],
   "source": [
    "# Some required modules\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "import csv\n",
    "import datetime\n",
    "import joblib\n",
    "\n",
    "# Data for surgical complications is loaded. \n",
    "# The target is \"complication\"\n",
    "HISTORICAL_DATA = './examples/data/surgical-dataset.csv'\n",
    "\n",
    "df = pd.read_csv(HISTORICAL_DATA)\n",
    "\n",
    "columns = list(df.columns)\n",
    "arr = df.to_numpy()\n",
    "\n",
    "np.random.shuffle(arr)\n",
    "\n",
    "split_ratio = 0.8\n",
    "prediction_threshold = 0.5\n",
    "\n",
    "train_data_len = int(arr.shape[0] * split_ratio)\n",
    "\n",
    "train_data = arr[:train_data_len, :-1]\n",
    "label = arr[:train_data_len, -1]\n",
    "test_data = arr[train_data_len:, :-1]\n",
    "test_df = df[train_data_len:]                 # test_df will play thre role of scoring dataset\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = columns[len(columns) - 1]\n",
    "orig_labels = test_df[target_column_name].tolist()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AmeZP5emw_SF"
   },
   "source": [
    "# 10.- Upload a pickle file with a pre-trained model pipeline to Google Colab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "phQ1SHC3x23o"
   },
   "source": [
    "We will load a pickle file named \"pipeline.pkl\" (found in the zip file that contains the class material); this file contains a pre-trained ML model pipeline. Navigate to the folder where the class material is and select the file named \"**pipeline.pkl**\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EMwqJ21pSivy"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "uploaded = files.upload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ICed3RBQ8dT_"
   },
   "source": [
    "# 11.- Run Model Predictions\n",
    "\n",
    "## 11.1.- We call the remote model's predict function and send prediction data to MLOps. Note that the model is supplied using the pickle file uploaded in the previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L-VqByyy_LCG"
   },
   "outputs": [],
   "source": [
    "# MLOps Agent Library imports\n",
    "from datarobot.mlops.mlops import MLOps\n",
    "from datarobot.mlops.common.enums import OutputType\n",
    "from datarobot.mlops.connected.client import MLOpsClient\n",
    "from datarobot.mlops.common.exception import DRConnectedException\n",
    "from datarobot.mlops.constants import Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n5RFjpepz5o7"
   },
   "outputs": [],
   "source": [
    "# Some necessary variables will be defined first\n",
    "\n",
    "CLASS_NAMES = ['0', \"1\"]\n",
    "\n",
    "# Here we define the parameters of the spool file that is used as messaging channel\n",
    "# Spool directory path must match the Monitoring Agent path configured by admin in the YAML configuration file.\n",
    "SPOOL_DIR = \"/tmp/ta\"\n",
    "MLOPS_FILESYSTEM_MAX_FILE_SIZE = 104857600\n",
    "MLOPS_FILESYSTEM_MAX_NUM_FILES = 5\n",
    "\n",
    "# name of the file that contains actuals\n",
    "ACTUALS_OUTPUT_FILE = \"actuals.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vPscl_CBdemM"
   },
   "outputs": [],
   "source": [
    "# Spool file parameters are defined as environment variables\n",
    "!export MLOPS_FILESYSTEM_MAX_FILE_SIZE\n",
    "!export MLOPS_FILESYSTEM_MAX_NUM_FILES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l-PeFHso0ew_"
   },
   "outputs": [],
   "source": [
    "# load pickle file with model pipeline\n",
    "model = joblib.load(filename=\"pipeline.pkl\")\n",
    "\n",
    "# Get predictions\n",
    "start_time = time.time()\n",
    "predictions = model.predict_proba(test_data).tolist()\n",
    "end_time = time.time()\n",
    "\n",
    "# number of predictions\n",
    "num_predictions = len(predictions)\n",
    "\n",
    "# time required to generate the predictions\n",
    "prediction_time = end_time - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xgD8o-MR0li3"
   },
   "outputs": [],
   "source": [
    "# Generate assocation ids for the predictions so we can match them with actuals\n",
    "# this is necessary for accuracy monitoring\n",
    "# The association ids are generated by taking the current time and appending a row counter to it\n",
    "def generate_unique_association_ids(num_samples):\n",
    "    ts = time.time()\n",
    "    return [\"x_{}_{}\".format(ts, i) for i in range(num_samples)]\n",
    "\n",
    "association_ids = generate_unique_association_ids(len(test_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3CNlodKzcXjP"
   },
   "outputs": [],
   "source": [
    "# Initialize a MLOPS instance\n",
    "mlops = MLOps().___(___) \\\n",
    "               .___(___) \\\n",
    "               .___(___) \\\n",
    "               .___()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eTmpSIOqncL-"
   },
   "outputs": [],
   "source": [
    "# MLOPS: report the number of predictions in the request and the execution time.\n",
    "print(\"Send MLOps deployment stats\")\n",
    "mlops.___(___, ___)\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1rkoiz8YnjaH"
   },
   "outputs": [],
   "source": [
    "# MLOPS: report the predictions data: features, predictions, class_names\n",
    "print(\"Send MLOps prediction data\")\n",
    "mlops.___(features_df=___,  predictions=___, class_names=___, association_ids=___)\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D6HObkyAJm7t"
   },
   "source": [
    "## 11.2.- In the next steps we are simulating a situation in which we receive a file with actual outcomes observed by the business. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LxGAVcyO8dT_"
   },
   "outputs": [],
   "source": [
    "# We are now going to deine a function to write a simulated actuals file to the Colab runtime\n",
    "import pytz\n",
    "print(\"Wrote actuals file: %s\" % ACTUALS_OUTPUT_FILE)\n",
    "def write_actuals_file(out_filename, test_data_labels, association_ids):\n",
    "    \"\"\"\n",
    "      Generate a CSV file with the association ids and labels, this example\n",
    "      uses a dataset that has labels already.\n",
    "      In a real use case actuals (labels) will show after prediction is done.\n",
    "\n",
    "    :param out_filename:      name of csv file\n",
    "    :param test_data_labels:  actual values (labels)\n",
    "    :param association_ids:   association id list used for predictions\n",
    "    \"\"\"\n",
    "    with open(out_filename, mode=\"w\") as actuals_csv_file:\n",
    "        writer = csv.writer(actuals_csv_file, delimiter=\",\")\n",
    "        writer.writerow(\n",
    "            [\n",
    "                Constants.ACTUALS_ASSOCIATION_ID_KEY,\n",
    "                Constants.ACTUALS_VALUE_KEY,\n",
    "                Constants.ACTUALS_TIMESTAMP_KEY\n",
    "            ]\n",
    "        )\n",
    "        tz = pytz.timezone(\"America/Los_Angeles\")\n",
    "        for (association_id, label) in zip(association_ids, test_data_labels):\n",
    "            actual_timestamp = datetime.datetime.now().replace(tzinfo=tz).isoformat()\n",
    "            writer.writerow([association_id, \"1\" if label else \"0\", actual_timestamp])           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JcvczFeF04hC"
   },
   "outputs": [],
   "source": [
    "# Write csv file with labels and association IDs\n",
    "write_actuals_file(ACTUALS_OUTPUT_FILE, orig_labels, association_ids)\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rDF_DvT3oXSb"
   },
   "outputs": [],
   "source": [
    "# MLOPS: release MLOps resources when finished.\n",
    "mlops.___()\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ak-Oh7EN8dUC"
   },
   "source": [
    "# 12.- Upload actuals back to MLOps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cs0p2stg09a0"
   },
   "outputs": [],
   "source": [
    "# A couple of utility functions\n",
    "\n",
    "# If we deal with regression we return a number, otherwise a string\n",
    "def _get_correct_actual_value(deployment_type, value):\n",
    "    if deployment_type == \"Regression\":\n",
    "        return float(value)\n",
    "    return str(value)\n",
    "\n",
    "# convert True/False strigns to boolean values\n",
    "def _get_correct_flag_value(value_str):\n",
    "    if value_str == \"True\":\n",
    "        return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WjnMjSnZ8dUC"
   },
   "outputs": [],
   "source": [
    "# We now define another function to \n",
    "# 1) Read data from the \"actuals.csv\" file\n",
    "# 2) Place the actual values in an array\n",
    "# 3) Place actuals in the messaging channel\n",
    "\n",
    "def upload_actuals():\n",
    "    print(\"Connect MLOps client\")           # create connected client object\n",
    "    mlops_connected_client = ___(___, ___)\n",
    "\n",
    "    # get deployment type\n",
    "    deployment_type = mlops_connected_client.___(___)\n",
    "\n",
    "    # read actuals file\n",
    "    actuals = []\n",
    "    with open(ACTUALS_OUTPUT_FILE, mode=\"r\") as actuals_csv_file:\n",
    "        reader = csv.DictReader(actuals_csv_file)\n",
    "        for row in reader:\n",
    "            actual = {}\n",
    "            for key, value in row.items():\n",
    "                if key == Constants.ACTUALS_WAS_ACTED_ON_KEY:\n",
    "                    value = _get_correct_flag_value(value)\n",
    "                if key == Constants.ACTUALS_VALUE_KEY:\n",
    "                    value = _get_correct_actual_value(deployment_type, value)\n",
    "                actual[key] = value\n",
    "            actuals.append(actual)\n",
    "         \n",
    "    # Upload actuals to MLOps\n",
    "    print(\"Submit actuals\")\n",
    "    mlops_connected_client.___(___, ___)\n",
    "    \n",
    "    print(\"Done!\")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XUXOwmPV1CLd"
   },
   "outputs": [],
   "source": [
    "upload_actuals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d3GLIykY8dUF"
   },
   "source": [
    "# 13.- Stop the mlops service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ye0gxF7xmwoX"
   },
   "outputs": [],
   "source": [
    "% ls bin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xRTyi3Tr8dUF"
   },
   "outputs": [],
   "source": [
    "!___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cAAScaS0FbtQ"
   },
   "source": [
    "# 14.- Inspect the MLOps agent logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EyA854mr8dUI"
   },
   "outputs": [],
   "source": [
    "cat /content/datarobot_mlops_package-7.1.8/logs/mlops.agent.log"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "29Apr2021 - MLOps II Laboratory",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}