2_External_DMM_Quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1c01e6a6-9c77-473d-a1f5-d7e2f427df76",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Sample External Model with Domino Model Monitoring\n",
    "\n",
    "Example notebook to set up external Domino Model Monitoring (DMM).\n",
    "\n",
    "## Background\n",
    "The key difference between monitoring external models and Domino's integrated model monitoring is that with external models, Domino does not capture your model's training data using TrainingSets or prediction data through the DataCaptureClient. You will need to provide the model's training data, prediction data and, optionally, ground truth labels in a Monitoring Data Source, and then point your model in DMM to that data.\n",
    "\n",
    "Other notes:\n",
    "\n",
    "(1) The model does not need to be trained in Domino. It can be an existing model trained elsewhere.\n",
    "\n",
    "(2) It does not matter where the external model is hosted. It could be on an edge device, on-prem, in your cloud hosting service, or run as a batch Job in Dominio.\n",
    "\n",
    "The steps below can be done via the DMM UI, or automated using DMM's API. To use the UI, follow the steps documented here:\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/679cc1/set-up-model-monitor/\n",
    "\n",
    "The examples below demonstrate the setup & monitoring of external models using DMM's API. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "086d1b6d-ecce-4c7b-8281-9a71a3495707",
   "metadata": {},
   "source": [
    "### Monitoring Data Source\n",
    "\n",
    "Domino requires an external data source to register an external model. For integrated models, you only need a Monitoring Data Source if you are ingesting ground truth labels. \n",
    "\n",
    "This datasource should already be connected and set up form the notebook '1_Initial_Setup.ipynb'\n",
    "\n",
    "For an External model external data source stores:\n",
    "\n",
    "(1) The training dataset\n",
    "\n",
    "(2) Prediction data & model predictions\n",
    "\n",
    "(3) Ground truth labels (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cef27f-7154-4f0c-95d8-fdce267aa768",
   "metadata": {},
   "source": [
    "### 1. Upload the Training Data to Monitoring Data Source\n",
    "\n",
    "Since we already connected the Monitoring data source to the Workbench in the setup Notebook, we'll upload the training data using a \n",
    "\n",
    "To upload the training dataset from the Workbench, add the same Monitoring Data Source registered above to this project. In a workspace, you can add it through the Data tab on the left.\n",
    "\n",
    "The Iris training dataset is already saved in the \"data\" folder in the mnt directory as \"iris_training_data.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "183a6e84-d4f3-4beb-8ce0-83dc336aae4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the config file\n",
    "import yaml\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "# Get your username so we don't overwrite the other users files!\n",
    "import os\n",
    "\n",
    "user_name = os.environ['DOMINO_USER_NAME']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c8819e79-9275-4052-9dd3-bfc4db9afc56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the Training Data from Domino Workbench to the DMM data source\n",
    "from domino.data_sources import DataSourceClient\n",
    "\n",
    "# instantiate a client and fetch the data source instance\n",
    "object_store = DataSourceClient().get_datasource(\"{}\".format(config['workbench_datasource_name']))\n",
    "\n",
    "# Upload the existing training data to your DMM data source from the Workbench\n",
    "object_store.upload_file('{}_iris_training_data.csv'.format(user_name), '/mnt/code/data/iris_training_data.csv')\n",
    "\n",
    "# Upload the first batch of scoring data to the DMM datasource.\n",
    "object_store.upload_file('{}_external_model_scoring_data.csv'.format(user_name), '/mnt/code/data/external_model_scoring_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f2db7f-89b7-4646-92d7-6aae3bf69315",
   "metadata": {},
   "source": [
    "### 2. Register an External Model in Domino Model Monitoring\n",
    "\n",
    "The first step is to register your new model in Domino Model Monitoring. To register a new external model:\n",
    "\n",
    "(1) Upload the training data used for your model to that Monitoring Data Source, and note the path to your training data file. DMM will need this to initiate the model.\n",
    "\n",
    "(2) Prepare your **Monitoring Config JSON** file. In the UI, the config json looks like the example below.\n",
    "\n",
    "It contains 3 components:\n",
    "\n",
    "(A) **variables**: A list of variable names, data types, and variable types for each column that you want to monitor. This can include the target variable if you'd like to monitor drift in your model's predictions.\n",
    "\n",
    "(B) **datasetDetails**: The name and location of your training dataset that you just uploaded into the DMM datasource\n",
    "\n",
    "(C) **modelMetadata**: The name and description of your model to render in Domino Model Monitoring\n",
    "\n",
    "Like with DMM Data Sources, Monitoring Config JSONs can be copied and pasted into the UI or automatically sent to Domino via APIs. Full documentation for Monitoring Config JSONs here:\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/bb88ca/monitoring-config-json/\n",
    "\n",
    "**Pro Tip:** Domino recommends saving your config json as a file in your Project files for future reference and modification. See \"Example_Model_Config.json\"\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "{\n",
    "    \"variables\": [\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"petal.length\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"sepal.length\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"petal.width\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"sepal.width\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"categorical\",\n",
    "            \"variableType\": \"prediction\",\n",
    "            \"name\": \"variety\"\n",
    "        }\n",
    "    ],\n",
    "    \"datasetDetails\": {\n",
    "        \"name\": \"iris.csv\",\n",
    "        \"datasetType\": \"file\",\n",
    "        \"datasetConfig\": {\n",
    "            \"path\": \"iris.csv\",\n",
    "            \"fileFormat\": \"csv\"\n",
    "        },\n",
    "        \"datasourceName\": \"dmm-shared-bucket\",\n",
    "        \"datasourceType\": \"s3\"\n",
    "    },\n",
    "    \"modelMetadata\": {\n",
    "        \"name\": \"iris_model\",\n",
    "        \"modelType\": \"classification\",\n",
    "        \"version\": \"1.01\",\n",
    "        \"description\": \"classification_iris_model\",\n",
    "        \"author\": \"John Doe\"\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "76e734ab-55b4-4136-94ac-b959465c3e6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New model id is: 66ebed56b7a424ef7c18d8ce\n",
      "DONE!\n"
     ]
    }
   ],
   "source": [
    "#### Example to register a model via the API\n",
    "# API Reference: https://docs.dominodatalab.com/en/latest/user_guide/a94c1c/model-monitoring-apis/#_model\n",
    "\n",
    "import os\n",
    "import json\n",
    "import requests\n",
    "\n",
    "# File names & format for the training data in your external datasource uploaded in Step 1.\n",
    "training_dataset_name = '{}_iris_training_data.csv'.format(user_name)\n",
    "training_dataset_path = '{}_iris_training_data.csv'.format(user_name)\n",
    "training_dataset_fileFormat = \"csv\"\n",
    "\n",
    "# Name for your model in DMM\n",
    "model_name = \"{} Example External Model\".format(user_name)\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "datasource_url = \"https://{}/model-monitor/v2/api/model\".format(config['url'])\n",
    "\n",
    "# Set up call headers\n",
    "headers = {\n",
    "           'X-Domino-Api-Key': os.environ['DOMINO_USER_API_KEY'],\n",
    "           'Content-Type': 'application/json'\n",
    "          }\n",
    "\n",
    "# Update each variable name, varibleType and valueType for your model:\n",
    "\n",
    "model_register_request = {\n",
    "    \"variables\": [\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"petal length (cm)\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"sepal length (cm)\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"petal width (cm)\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"numerical\",\n",
    "            \"variableType\": \"feature\",\n",
    "            \"name\": \"sepal width (cm)\"\n",
    "        },\n",
    "        {\n",
    "            \"valueType\": \"categorical\",\n",
    "            \"variableType\": \"prediction\",\n",
    "            \"name\": \"variety\"\n",
    "        }\n",
    "    ],\n",
    "    \"datasetDetails\": {\n",
    "        \"name\": training_dataset_name,\n",
    "        \"datasetType\": \"file\",\n",
    "        \"datasetConfig\": {\n",
    "            \"path\": training_dataset_path,\n",
    "            \"fileFormat\": training_dataset_fileFormat\n",
    "        },\n",
    "        \"datasourceName\": config['DMM_datasource_name'],\n",
    "        \"datasourceType\": config['datasource']['type'],\n",
    "    },\n",
    "    \"modelMetadata\": {\n",
    "        \"name\": model_name,\n",
    "        \"modelType\": \"classification\",\n",
    "        \"version\": \"1.01\",\n",
    "        \"description\": \"classification_iris_model\",\n",
    "        \"author\": os.environ['DOMINO_USER_NAME']\n",
    "    }\n",
    "}\n",
    "\n",
    "# Make api call\n",
    "ground_truth_response = requests.request(\"PUT\", datasource_url, headers=headers, data = json.dumps(model_register_request))\n",
    " \n",
    "# Print response\n",
    "response = json.loads(ground_truth_response.text.encode('utf8'))\n",
    "\n",
    "print(\"New model id is: {}\".format(response['id']))\n",
    "\n",
    "# Save the model ID for next steps\n",
    "new_model_id = response['id']\n",
    "\n",
    "# Save the new model ID to the config file\n",
    "config['external_model_id'] = new_model_id\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\", \"w\") as yamlfile:\n",
    "    config = yaml.dump(\n",
    "        config, stream=yamlfile, default_flow_style=False, sort_keys=False\n",
    "    )\n",
    "\n",
    "print('DONE!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b63076-f2a1-497d-9525-f42dbf74088d",
   "metadata": {},
   "source": [
    "*Dont forget to save and sync your workspace so that the config file has the new model id!*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0e2909a-85cc-4c80-9db3-28a77867e22b",
   "metadata": {},
   "source": [
    "### 3. Set up Drift Detection\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/86bc1f/set-up-drift-detection/\n",
    "\n",
    "\n",
    "While integrated models can capture prediction data using the DataCaptureClient, external models need to ingest prediction data from a connected Monitoring Data Source. Just like with the initial model registration, information needed to ingest the prediction data is provided to Domino using a **Prediction Config JSON**.\n",
    "\n",
    "\n",
    "There are two approaches to automating prediction data ingest from a Monitoring Data Source:\n",
    "\n",
    "\n",
    "(1) Append new data to the same file in your Monitoring Data Source.\n",
    "\n",
    "\n",
    "  -  Only register your Prediction Data config with the path to the prediction data file once. Domino will automatically retrieve new data every 24 hours from that file. You can schedule the daily check in the DMM UI.\n",
    "  -  This approach requires registering a **timestamp** variable so that DMM knows which prediction rows are new.\n",
    "\n",
    "\n",
    "(2) Upload prediction data as separate files to your Monitoring Data Source.\n",
    "\n",
    "\n",
    "  - This requires updating the datasetDetails in the Prediction Data Config everytime new prediction data is added. This is best automated through the API, using a Domino Job or some other scheduler.\n",
    "  - When you update the Prediction Data Config, only update the \"datasetDetails\" with the new prediction data file path. Variables are only set the first time, if you re-register variable names DMM will throw an error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a97e3c8-68f3-4acf-b283-404bfad5ed3c",
   "metadata": {},
   "source": [
    "#### Option 1\n",
    "\n",
    "Below is an example of an initial Prediction Data Config file, which be copied and pasted into the UI or automatically sent to Domino via APIs. Full docs here:\n",
    "\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/bb88ca/monitoring-config-json/\n",
    "\n",
    "\n",
    "At the end is an example of only updating the \"datasetDetails\" via the API if you choose to follow approach #2.\n",
    "\n",
    "**Pro Tip:** Domino recommends saving your config json as a file in your Project files for future reference and modification. See \"Example_Prediction_Config.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "854c970b-34b5-4747-8d10-a982f80a0f76",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b''\n",
      "DONE!\n"
     ]
    }
   ],
   "source": [
    "### Example to register the initial Prediction Config via the API\n",
    "# API Reference: https://docs.dominodatalab.com/en/latest/user_guide/a94c1c/model-monitoring-apis/#_model\n",
    "\n",
    "# The RowID Name (Optional, used for model quality monitoring. Do this only once.)\n",
    "Prediction_ID_name = 'event_id'\n",
    "\n",
    "# File names & format for the prediction data in your external datasource uploaded in Step 1.\n",
    "prediction_dataset_name = '{}_external_model_scoring_data.csv'.format(user_name)\n",
    "prediction_dataset_path = '{}_external_model_scoring_data.csv'.format(user_name)\n",
    "prediction_dataset_fileFormat = \"csv\"\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "prediction_data_url = \"https://{}/model-monitor/v2/api/model/{}/register-dataset/prediction\".format(config['url'], config['external_model_id'])\n",
    "\n",
    "\n",
    "# Set up call headers\n",
    "headers = {\n",
    "           'X-Domino-Api-Key': os.environ['DOMINO_USER_API_KEY'],\n",
    "           'Content-Type': 'application/json'\n",
    "          }\n",
    "\n",
    "# Update each variable name, varibleType and valueType for your model:\n",
    "\n",
    "prediction_registration_request = {\n",
    "    \"variables\": [\n",
    "        {\n",
    "            \"valueType\": \"string\",\n",
    "            \"variableType\": \"row_identifier\",\n",
    "            \"name\": Prediction_ID_name\n",
    "        }\n",
    "    ],\n",
    "    \"datasetDetails\": {\n",
    "        \"name\": prediction_dataset_name,\n",
    "        \"datasetType\": \"file\",\n",
    "        \"datasetConfig\": {\n",
    "            \"path\": prediction_dataset_path,\n",
    "            \"fileFormat\": prediction_dataset_fileFormat\n",
    "        },\n",
    "        \"datasourceName\": config['DMM_datasource_name'],\n",
    "        \"datasourceType\": config['datasource']['type']\n",
    "    }\n",
    "}\n",
    "\n",
    "# Make api call\n",
    "ground_truth_response = requests.request(\"PUT\", prediction_data_url, headers=headers, data = json.dumps(prediction_registration_request))\n",
    " \n",
    "# Print response\n",
    "print(ground_truth_response.text.encode('utf8'))\n",
    "\n",
    "print('DONE!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3224387-ecdc-419f-b523-bb3a0d783b13",
   "metadata": {},
   "source": [
    "#### Option 2: Upload additional prediction data as separate files to your Monitoring Data Source.\n",
    "\n",
    "Next is an example for updating the prediction data file via the API if you choose option (2).\n",
    "\n",
    "This step:\n",
    "1) Uploads a file with new scoring data to the DMM datasource\n",
    "2) Use the Domino Model Monitoring API to update the path to this new data, and ingest the new scoring data for drift detection.\n",
    "\n",
    "Example scripts to automate these steps using Domino Jobs are in the \"external_model_scripts\" folder, and are described in detail at the end of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5d0f2df1-ae4e-4989-8247-49b32b1e6b61",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a client and fetch the data source instance\n",
    "object_store = DataSourceClient().get_datasource(\"{}\".format(config['workbench_datasource_name']))\n",
    "\n",
    "# Upload to the data source (s3 in this case)\n",
    "object_store.upload_file('{}_external_model_scoring_data_update.csv'.format(user_name), '/mnt/code/data/external_model_scoring_data_update.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "09bc6736-63a0-46d2-8d0e-fb1beb462bee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'[\"Dataset already registered with the model.\"]'\n",
      "DONE!\n"
     ]
    }
   ],
   "source": [
    "### Example to update Prediction Config if uploading prediction data as separate files to your Monitoring Data Source.\n",
    "\n",
    "# File names & format for the new prediction data file.\n",
    "prediction_dataset_name = \"{}_external_model_scoring_data_update.csv\".format(user_name)\n",
    "prediction_dataset_path = \"{}_external_model_scoring_data_update.csv\".format(user_name)\n",
    "prediction_dataset_fileFormat = \"csv\"\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "prediction_data_url = \"https://{}/model-monitor/v2/api/model/{}/register-dataset/prediction\".format(config['url'], config['external_model_id'])\n",
    "\n",
    "\n",
    "# Set up call headers\n",
    "headers = {\n",
    "           'X-Domino-Api-Key': os.environ['DOMINO_USER_API_KEY'],\n",
    "           'Content-Type': 'application/json'\n",
    "          }\n",
    "\n",
    "prediction_registration_request = {\n",
    "    \"datasetDetails\": {\n",
    "        \"name\": prediction_dataset_name,\n",
    "        \"datasetType\": \"file\",\n",
    "        \"datasetConfig\": {\n",
    "            \"path\": prediction_dataset_path,\n",
    "            \"fileFormat\": prediction_dataset_fileFormat\n",
    "        },\n",
    "        \"datasourceName\": config['DMM_datasource_name'],\n",
    "        \"datasourceType\": config['datasource']['type']\n",
    "    }\n",
    "}\n",
    "\n",
    "# Make api call\n",
    "ground_truth_response = requests.request(\"PUT\", prediction_data_url, headers=headers, data = json.dumps(prediction_registration_request))\n",
    " \n",
    "# Print response\n",
    "print(ground_truth_response.text.encode('utf8'))\n",
    " \n",
    "print('DONE!')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a23c65-10c7-4217-80ab-1848f7a50b8a",
   "metadata": {},
   "source": [
    "### 4. Set up Model Quality Monitoring (Optional)\n",
    "\n",
    "There is very little difference in setting up Model Quality Monitoring between Internal and External models, since Domino Model APIs cannot capture actual outcomes after-the-fact. The process is nearly the same as registering prediction data for external models.\n",
    "\n",
    "For this example, we have a dummy ground truth dataset in the\"data folder \"external_model_ground_truth_data.csv\"\n",
    "\n",
    "1) Upload the sample ground truth data to your DMM data source.\n",
    "2) Use the Domino Model Monitoring API to set your ground truth labels column name, initial ground truth label file name, and DMM datasource to reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "17981520-45e2-4c06-9b06-6088bffeb897",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# instantiate a client and fetch the datasource instance\n",
    "object_store = DataSourceClient().get_datasource(\"{}\".format(config['workbench_datasource_name']))\n",
    "\n",
    "# Upload to the datasouce (s3 in this case)\n",
    "object_store.upload_file('{}_external_model_ground_truth_data.csv'.format(user_name), '/mnt/code/data/external_model_ground_truth_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36478b4f-53de-4a11-879e-b7a82875a892",
   "metadata": {},
   "source": [
    "The Ground Truth dataset needs 2 columns: \n",
    "\n",
    "1) The existing event ID column from the model predictions.\n",
    "   \n",
    "    This column has the join keys for joining ground truth labels to your model's predictions\n",
    "\n",
    "3) Your new column containing ground truth labels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "28e75973-c2f9-429e-aa4e-b8b1c029aaa0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering bryan_prosser_external_model_ground_truth_data.csv From S3 Bucket in DMM\n",
      "b''\n",
      "DONE!\n"
     ]
    }
   ],
   "source": [
    "# File names for the ground truth data in your external data source uploaded above.\n",
    "gt_file_name = \"{}_external_model_ground_truth_data.csv\".format(user_name)\n",
    "\n",
    "# Ground Truth column name\n",
    "GT_column_name = 'iris_ground_truth'\n",
    "\n",
    "# The original target column name\n",
    "target_column_name = 'variety'\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "ground_truth_url = \"https://{}/model-monitor/v2/api/model/{}/register-dataset/ground_truth\".format(config['url'], config['external_model_id'])\n",
    "\n",
    "print('Registering {} From S3 Bucket in DMM'.format(gt_file_name))\n",
    " \n",
    "# create GT payload    \n",
    " \n",
    "# Set up call headers\n",
    "headers = {\n",
    "           'X-Domino-Api-Key': os.environ['DOMINO_USER_API_KEY'],\n",
    "           'Content-Type': 'application/json'\n",
    "          }\n",
    "\n",
    " \n",
    "ground_truth_payload = \"\"\"\n",
    "{{\n",
    "    \"variables\": [{{\n",
    "    \n",
    "            \"valueType\": \"categorical\",\n",
    "            \"variableType\": \"ground_truth\",\n",
    "            \"name\": \"{2}\", \n",
    "            \"forPredictionOutput\": \"{3}\"\n",
    "        \n",
    "    }}],\n",
    "    \"datasetDetails\": {{\n",
    "            \"name\": \"{0}\",\n",
    "            \"datasetType\": \"file\",\n",
    "            \"datasetConfig\": {{\n",
    "                \"path\": \"{0}\",\n",
    "                \"fileFormat\": \"csv\"\n",
    "            }},\n",
    "            \"datasourceName\": \"{1}\",\n",
    "            \"datasourceType\": \"{4}\"\n",
    "        }}\n",
    "}}\n",
    "\"\"\".format(gt_file_name, config['DMM_datasource_name'], GT_column_name, target_column_name, config['datasource']['type'])\n",
    " \n",
    "# Make api call\n",
    "ground_truth_response = requests.request(\"PUT\", ground_truth_url, headers=headers, data = ground_truth_payload)\n",
    " \n",
    "# Print response\n",
    "print(ground_truth_response.text.encode('utf8'))\n",
    " \n",
    "print('DONE!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fafc660f-f094-4b83-bc2a-935b835e9c24",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Next Steps\n",
    "\n",
    "To periodically upload ground truth labels, repeat the previous step, but without the “variables” in the ground truth payload (this only needs to be done once). As new ground truth labels are added, point Domino to the path to the new labels in the monitoring data source by pinging the same Model Monitoring API:\n",
    "\n",
    "ground_truth_payload = \"\"\"\n",
    "\n",
    "{{\n",
    "\n",
    "       \"datasetDetails\": {{\n",
    "        \n",
    "            \"name\": \"{0}\",\n",
    "            \"datasetType\": \"file\",\n",
    "            \"datasetConfig\": {{\n",
    "                \"path\": \"{0}\",\n",
    "                \"fileFormat\": \"csv\"\n",
    "            }},\n",
    "            \"datasourceName\": \"{1}\",\n",
    "            \"datasourceType\": \"s3\"\n",
    "        }}\n",
    "}}\"\"\".format(gt_file_name, data_source, GT_column_name, target_column_name)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "679cd486-42bb-48e7-85e5-2d4ed3056f18",
   "metadata": {},
   "source": [
    "### Automation with Domino Jobs\n",
    "To simulate Domino Model Monitoring over time, you can try out running the following two scripts described above as scheduled Domino Jobs.\n",
    "\n",
    "Just like with this notebook, be sure to update & commit your workspace to update the config file.\n",
    "\n",
    "#### Your Model Parameters\n",
    "- workbench_datasource_name\n",
    "- datasourceType \n",
    "- DMM_datasource_name \n",
    "- domino_url \n",
    "- DMM_model_id \n",
    "- API_key \n",
    "\n",
    "**external_model_daily_batch_job.py** \n",
    "\n",
    "This script simulates the external model scoring step, using a Domino job for batch inference. Batch inference in Domino using Domino Jobs can be monitored the same way as an external model. For external models, the scoring data, external model predictions, and prediction ID must be captured manually.\n",
    "\n",
    "1) First, the script scores the data using a local model, and captures the model's predictions, and generates a row identifier used for model quality tracking (Step 4)\n",
    "2) Second, the script uploads the scoring data, including the external model predictions and prediction ID, to the DMM data source for our external model.\n",
    "3) Finally it updates the file paths for the scoring data using the DMM API so that DMM can find the new data when it ingests data from the DMM data source.\n",
    "\n",
    "**external_model_daily_ground_truth.py** \n",
    "\n",
    "This script simulated sending ground truth labels to the model after the predictions are made for model quality monitoring.\n",
    "\n",
    "1) It generates dummy ground truth labels (we unfortunately don't have a botanist to label the irises)\n",
    "2) Second, the script uploads the ground truth data to the DMM data source for our external model.\n",
    "3) Finally it updates the file paths for the ground truth data using the DMM API so that DMM can find the new labels when it ingests ground truth data from the DMM data source. \n",
    "\n",
    "\n",
    "To test these scripts the ensure batch job runs before daily scoring, and that DMM is scheduled to ingest the scoring and ground truth data after daily_scoring_upload has finished. \n",
    "\n",
    "If you schedule these two jobs, be sure that ground truth runs after the predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d74eb37f-c5d1-4927-82e3-2ca3abf165de",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "dca-init": "true",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}