1_Initial_Setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6d24f561-0872-4348-9c8c-82f0124b0cf9",
   "metadata": {},
   "source": [
    "# Initial Setup \n",
    "\n",
    "Domino model monitoring involves connecting your model and external data sources to Domino Model Monitoring. This starter notebook walks through setting up the components of Domino Model Monitor that will be used in the \"2_External_DMM_Quickstart\", \"3_Integrated_DMM_Quickstart\" and \"4_Custom_Metrics_Quickstart.ipynb\" notebooks.\n",
    "\n",
    "This notebook also saves the name and configs for your datasource in a config file under your Project Artifacts. This config file will streamline automated scoring & ground truth data checks later on, without committing any sensitive information to a git repo."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "688f5444-9720-46d5-b656-d318bd513c05",
   "metadata": {},
   "source": [
    "## Domino API Key\n",
    "\n",
    "The Quickstart notebooks have examples for interacting with Domino Model Monitoring via API (for example, registering a new datasource in Domino Model Monitoring's API).\n",
    "\n",
    "Your Domino User API key is stored by default as the user environment variable \"DOMINO_USER_API_KEY\". Treat your API key like a sensitive password, and do not save it anywhere in this notebook, as it could be committed to an external git service if you sync your workspace. Always access it using the user environment variable.\n",
    "\n",
    "Authenticating with your Domino API key is documented here:\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/40b91f/domino-api-authentication/#_authenticate_with_an_api_key\n",
    "\n",
    "**Note on your Domino API Key:** Note that Domino Model Monitoring API keys have been deprecated, there is now only one API key for your whole Domino account.\n",
    "\n",
    "*If you need to update or rotate your Domino API key, go to the Domino Workbench, then open up your User Account setting in the lower left. Regenerate your API key, save it securely, then also save to your Domino account as a User Environment Variable.*\n",
    "\n",
    "*Your Workspace will not yet know about your new User Environment Variable. Save your notebook, then save and restart your Workspace to make your workspace aware of the new environment variable. Also be sure to update the references to the API key in these notebooks to reference your new variable name.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8b5d0e23-2f8a-4f56-8f21-17adf5d658cf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Verify that you can access your API key. \n",
    "import os\n",
    "\n",
    "my_key = os.environ['DOMINO_USER_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a657b756-8bb1-4dc4-a58a-3d12b74be08f",
   "metadata": {},
   "source": [
    "## Register your Domino Model Monitoring Datasource\n",
    "\n",
    "Domino requires an external data source for external models. For integrated models, you only need a Monitoring Data Source if you are ingesting ground truth labels. A datasource can be set up in the Domino Model monitoring UI, or via the Domino Model Monitoring API.\n",
    "\n",
    "Both Quickstart Notebooks use an AWS s3 bucket for their Domino Model Monitoring datasource, but other data sources such as Snowflake or ADLS can be used. Domino Model Monitoring data sources are documented here:\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/user_guide/8c7833/connect-a-data-source/\n",
    "\n",
    "The Monitoring Data Sources are registered independently of the data sources used in Domino Workbench. The Quickstart Examples pass data from this Domino Project in the Domino Workbench to Domino Model Monitoring through the external datasource, so we will need to add the same external data source to both Domino Model Monitoring and this Project.\n",
    "\n",
    "### Steps\n",
    "\n",
    "(1) Set up your external storage outside of Domino (in AWS, Azure, GCP, Snowflake or HDFS). Note that to set up a Monitoring Datasource, you will need your key / password for that datsource. Note the location for monitoring files in your data source (e.g. bucket name, database name, container etc.).\n",
    "\n",
    "*If you do not have those credentials, contact your Admin and ask them to add the datasource to Domino Model Monitoring and grant you access to the data source in the Domino Workbench using a service account.*\n",
    "\n",
    "(2) Add the datasource to Domino Model Monitoring. Note the model monitoring \"Data Source Name\" when you create it, this will be used to access the datasource.\n",
    "\n",
    "<img src=\"readme_images/Add_DMM_Datasource.png\" width=\"600\" height=\"600\"> \n",
    "\n",
    "\n",
    "\n",
    "(3) Add the same datasource to this Project in the workbench. Note the workbench data source name in the data tab on the left.\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"readme_images/Add_DMM_Workbench.png\" width=\"600\" height=\"600\"> \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6beabd0a-6256-4442-938c-fa14b1eb0cc5",
   "metadata": {},
   "source": [
    "## Save Dataset names and API reference in a config file\n",
    "\n",
    "To reference the names of the data sources created above, as well as your Domino URL.\n",
    "\n",
    "We'll save this in a configuration yaml file in our Project Artifacts. The other Quick Setup notebook will reference these configs.\n",
    "\n",
    "Update each of the fields below, then run the cell, and commit your workspace. A new file called \"DMM_config.yaml\" should appear in your Project Artifacts folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "66052531-866c-4601-b061-dd9adce4791a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to config file successful\n",
      "Remember to save this notebook & sync your Workspace!\n"
     ]
    }
   ],
   "source": [
    "import yaml\n",
    "\n",
    "# UPDATE 1: Set the name for your new datasource in Domino Model Monitoring\n",
    "\n",
    "DMM_datasource_name = 'monitoring-workshop'\n",
    "\n",
    "# UPDATE 2: Set the name for your new data source in this Project (the name in the Data tab to the left)\n",
    "\n",
    "workbench_datasource_name = 'monitoring-workshop'\n",
    "\n",
    "# UPDATE 3: Datasource path. This will vary depending on the datasource type used. \n",
    "# For this s3 example, it is the bucket the monitoring files will be stored in.\n",
    "\n",
    "datasource_location = \"monitoring-workshop\"\n",
    "\n",
    "# UPDATE 4: Your Domino URL for API calls don't include the 'http://' or the trailing '/'\n",
    "\n",
    "your_domino_url = \"monitoring-url\"\n",
    "\n",
    "# UPDATE 5: The type of datasource (s3, ADLS, Snowflake etc). Names are documented here:\n",
    "# https://docs.dominodatalab.com/en/latest/user_guide/8c7833/connect-a-data-source/\n",
    "\n",
    "datasource_type = \"s3\"\n",
    "\n",
    "DMM_configs = {\n",
    "        'DMM_datasource_name': DMM_datasource_name,\n",
    "        'workbench_datasource_name': workbench_datasource_name,\n",
    "        'url': your_domino_url,\n",
    "        'datasource':{\n",
    "            'type': datasource_type,\n",
    "            'location': datasource_location     \n",
    "        },\n",
    "        'integrated_model_url': '_', # For 3_Integrated_DMM_Quickstart\n",
    "        'integrated_model_auth': '_', # For 3_Integrated_DMM_Quickstart\n",
    "        'integrated_model_id': '_' # For 3_Integrated_DMM_Quickstart\n",
    "    }\n",
    "\n",
    "\n",
    "# Create a yaml file in artifacts, and save configs above.\n",
    "!touch /mnt/artifacts/DMM_config.yaml\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\", 'w') as yamlfile:\n",
    "    data = yaml.dump(DMM_configs, yamlfile)\n",
    "    print(\"Write to config file successful\")\n",
    "\n",
    "print('Remember to save this notebook & sync your Workspace!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a39fbe-7662-418a-a679-fa7286b6d57b",
   "metadata": {},
   "source": [
    "### (Optional) Register Datasource via API\n",
    "\n",
    "You can register a datasource in Domino Model Monitoring using the UI (shown above) or using an API call.  The following example uses an AWS s3 bucket.\n",
    "\n",
    "The APIs for other data source types are documented here:\n",
    "\n",
    "https://docs.dominodatalab.com/en/latest/api_guide/f31cde/model-monitoring-api-reference/#_s3Request\n",
    "\n",
    "Like in the UI, registering a data source via API requires your secret key or password, which should be securely stored as user environment variables. \n",
    "\n",
    "Go to the Domino Workbench, then open up your User Account setting in the lower left. Save the required keys for your datasource as User Environment Variables. In this example, I’ve called the s3 keys 'AWS_ACCESS_KEY_ID' and 'AWS_SECRET_ACCESS_KEY'.\n",
    "\n",
    "Your Workspace will not yet know about your new User Environment Variable. Save your notebook, then save and restart your Workspace to make your workspace aware of the new user environment variable. Note that these keys will not be available to Project collaborators, only you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3797ccb8-7c94-4535-be24-4e3d16e78854",
   "metadata": {},
   "outputs": [],
   "source": [
    "# s3 Example\n",
    "\n",
    "import json\n",
    "import requests\n",
    "\n",
    "# UPDATE 1: Set the name for your new datasource in Domino Model Monitoring\n",
    "DMM_datasource_name = 'API-Datasource'\n",
    "\n",
    "# UPDATE 2: Cloud region, needed for s3 bucket registration\n",
    "S3_Region = \"us-west-2\"\n",
    "\n",
    "# AWS Keys, saved as User Environment Variables\n",
    "AWS_Access_Key = os.environ.get(\"AWS_ACCESS_KEY_ID\")\n",
    "AWS_Secret_Key = os.environ.get(\"AWS_SECRET_ACCESS_KEY\")\n",
    "\n",
    "# Fetch configs\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\") as yamlfile:\n",
    "    config = yaml.safe_load(yamlfile)\n",
    "\n",
    "datasource_url = \"https://{}/model-monitor/v2/api/datasource\".format(config['url'])\n",
    "\n",
    "# Set up call headers\n",
    "headers = {\n",
    "           'X-Domino-Api-Key': os.environ['DOMINO_USER_API_KEY'],\n",
    "           'Content-Type': 'application/json'\n",
    "          }\n",
    "\n",
    "data_source_request = {\n",
    "    \"name\": DMM_datasource_name,\n",
    "    \"type\": config['datasource']['type'],\n",
    "    \"config\" : {\n",
    "        \"bucket\": config['datasource']['location'],\n",
    "        \"region\": S3_Region,\n",
    "        \"instance_role\" : False,\n",
    "        \"access_key\": AWS_Access_Key,\n",
    "        \"secret_key\": AWS_Secret_Key\n",
    "    }\n",
    "}\n",
    "\n",
    "# Make api call\n",
    "ground_truth_response = requests.request(\"PUT\", datasource_url, headers=headers, data = json.dumps(data_source_request))\n",
    " \n",
    "# Print response\n",
    "print(ground_truth_response.text.encode('utf8'))\n",
    "\n",
    "# Finally, add datasource-specific configs to the yaml file. These will vary depending on your datasource type.\n",
    "config['datasource']['S3_Region'] = S3_Region\n",
    "config['DMM_datasource_name'] = DMM_datasource_name\n",
    "\n",
    "with open(\"/mnt/artifacts/DMM_config.yaml\", \"w\") as yamlfile:\n",
    "    config = yaml.dump(\n",
    "        config, stream=yamlfile, default_flow_style=False, sort_keys=False\n",
    "    )\n",
    "\n",
    "print('DONE!')\n",
    "\n",
    "print('Remember to save this notebook & sync your Workspace!')"
   ]
  }
 ],
 "metadata": {
  "dca-init": "true",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}