sdk/python/endpoints/online/managed/online-endpoints-inference-schema.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference Schema\n",
    "In this example, we use the [Inference Schema](https://github.com/Azure/InferenceSchema) package to facilitate automatic Swagger generation and parameter casting for Managed Online Endpoints.\n",
    "\n",
    "## Background\n",
    "\n",
    "[Inference Schema](https://github.com/Azure/InferenceSchema) is an open source library for ML applications maintained by Travis Angevine [@trangevi](https://github.com/trangevi) that streamlines schema design and development for ML applications and offers features such as parameter type definition, automatic type conversion, and schema/swagger file generation. Using Inference Schema, users can easily define parameter types and associate them to functions with input and output decorators. \n",
    "\n",
    "Inference Schema integrates directly with AzureML endpoints. User `run` functions with Inference Schema decorators can be defined with an arbitrary number of arguments and receive automatic swagger file generation at `/swagger.json`.\n",
    "\n",
    "## Function Decorators\n",
    "The decorators `input_schema` and `output_schema` are used to define the schema. The `input_schema` decorator can be stacked multiple times as in [score-standard](code/score-standard.py) to correspond to multiple function arguments in the run function. \n",
    "\n",
    "## Parameter Types\n",
    "There are 4 core parameter types available:\n",
    "- StandardPythonParameterType\n",
    "- PandasParameterType\n",
    "- NumpyParameterType\n",
    "- SparkParameterType\n",
    "\n",
    "It is possible to nest parameter types by wrapping them in a list or dict and a `StandardParameterType`.\n",
    "\n",
    "## Swagger\n",
    "The automatically-generated Swagger can be retrieved by default at `/swagger.json` with an optional `version` HTTP parameter. See the [online-endpoints-openapi](online-endpoints-openapi.ipynb) example for more details about generating and accessing Swagger files.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Configure parameters, assets, and clients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Set workspace details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace_name = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Set asset details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "rand = random.randint(0, 10000)\n",
    "\n",
    "endpoint_name = f\"infsrv-{rand}\"\n",
    "model_name = \"infsrv\"\n",
    "model_version = str(rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Import required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    Model,\n",
    "    CodeConfiguration,\n",
    "    Environment,\n",
    ")\n",
    "from azure.identity import DefaultAzureCredential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Create an MLClient instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "credential = DefaultAzureCredential()\n",
    "ml_client = MLClient(\n",
    "    credential,\n",
    "    subscription_id=subscription_id,\n",
    "    resource_group_name=resource_group,\n",
    "    workspace_name=workspace_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Create endpoint and model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Define and create the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = ManagedOnlineEndpoint(name=endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = ml_client.online_endpoints.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Get endpoint key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "key = ml_client.online_endpoints.get_keys(endpoint_name).primary_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Register the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(name=model_name, version=model_version, path=\"inference-schema/models\")\n",
    "model = ml_client.models.create_or_update(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create Deployment: Numpy\n",
    "\n",
    "Each of the deployments in this example use the same model and produce the same output, however they each accept input in different formats.\n",
    "\n",
    "For the first deployment, we will use the [score-numpy.py](../../../../../cli/endpoints/online/managed/inference-schema/code/score-numpy.py) scoring script, which declares a single input parameter `iris` as a `NumpyParameterType` and a nested `StandardParameterType` as output.\n",
    "\n",
    "```python\n",
    "@input_schema(\n",
    "    param_name=\"iris\",\n",
    "    param_type=NumpyParameterType(np.array([[7.2, 3.2, 6.0, 1.8]]))\n",
    ")\n",
    "@output_schema(\n",
    "    output_type=StandardPythonParameterType({\n",
    "        \"Category\" : [\"Virginica\"]\n",
    "    })\n",
    ")\n",
    "``` \n",
    "When the following [sample input](../../../../../cli/endpoints/online/managed/inference-schema/sample-inputs/numpy.json) is sent, it is automatically validated and converted to a Numpy array. Validation can be toggled using the `enforce_column_type` and `enforce_shape` arguments to `NumpyParameterType`.\n",
    "\n",
    "```json\n",
    "{\"iris\": [[7.2, 3.2, 6.0, 1.8], [4.2, 3.5, 1.0, 3.0]]}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Define the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ManagedOnlineDeployment(\n",
    "    name=\"infsrv-numpy\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    model=f\"azureml:{model.name}:{model.version}\",\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"inference-schema/code\", scoring_script=\"score-numpy.py\"\n",
    "    ),\n",
    "    environment=Environment(\n",
    "        image=\"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference\",\n",
    "        conda_file=\"inference-schema/env.yml\",\n",
    "    ),\n",
    "    instance_type=\"Standard_DS2_v2\",\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ml_client.online_deployments.begin_create_or_update(deployment).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Set traffic to 100%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"infsrv-numpy\": 100}\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Test the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=endpoint_name,\n",
    "    request_file=\"inference-schema/sample-inputs/numpy.json\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Get swagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "res = requests.get(url=endpoint.openapi_uri, headers={\"Authorization\": f\"Bearer {key}\"})\n",
    "print(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create Deployment: Standard / Multiple Parameters\n",
    "\n",
    "This deployment uses a different scoring script - [score-standard.py](../../../../../cli/endpoints/online/managed/inference-schema/code/score-numpy.py) - in which the run function accepts multiple list arguments. This is done by using the `StandardPythonParameter` type and apploying the `input_schema` decorator multiple times. \n",
    "\n",
    "```python\n",
    "@input_schema(\n",
    "    param_name=\"sepal_length\",\n",
    "    param_type=StandardPythonParameterType([7.2])\n",
    ")\n",
    "@input_schema(\n",
    "    param_name=\"sepal_width\",\n",
    "    param_type=StandardPythonParameterType([3.2])\n",
    ")\n",
    "@input_schema(\n",
    "    param_name=\"petal_length\",\n",
    "    param_type=StandardPythonParameterType([6.0])\n",
    ")\n",
    "@input_schema(\n",
    "    param_name=\"petal_width\",\n",
    "    param_type=StandardPythonParameterType([1.8])\n",
    ")\n",
    "@output_schema(\n",
    "    output_type=StandardPythonParameterType({\n",
    "        \"Category\" : [\"Virginica\"]\n",
    "    })\n",
    ")\n",
    "def run(sepal_length, sepal_width, petal_length, petal_width):\n",
    "``` \n",
    "This script produces the same output as before using the following [sample input](../../../../../cli/endpoints/online/managed/inference-schema/sample-inputs/standard.json).\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"sepal_length\": [7.2, 4.2],\n",
    "    \"sepal_width\": [3.2,3.5], \n",
    "    \"petal_length\": [6.0,1.0],\n",
    "    \"petal_width\": [1.8,3.0]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Define the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ManagedOnlineDeployment(\n",
    "    name=\"infsrv-standard\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    model=f\"azureml:{model.name}:{model.version}\",\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"inference-schema/code/\", scoring_script=\"score-standard.py\"\n",
    "    ),\n",
    "    environment=Environment(\n",
    "        image=\"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference\",\n",
    "        conda_file=\"inference-schema/env.yml\",\n",
    "    ),\n",
    "    instance_type=\"Standard_DS2_v2\",\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Create the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ml_client.online_deployments.begin_create_or_update(deployment).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Set traffic to 100%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"infsrv-standard\": 100}\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Test the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=endpoint_name,\n",
    "    request_file=\"inference-schema/sample-inputs/standard.json\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Get swagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = requests.get(url=endpoint.openapi_uri, headers={\"Authorization\": f\"Bearer {key}\"})\n",
    "print(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Create Deployment: Standard / Multiple Parameters\n",
    "\n",
    "This scoring script - [score-pandas.py](../../../../../cli/endpoints/online/managed/inference-schema/code/score-pandas.py) - accepts a single Pandas dataframe by using the following decorators\n",
    "```python\n",
    "@input_schema(\n",
    "    param_name=\"iris\",\n",
    "    param_type=PandasParameterType(pd.DataFrame({\n",
    "        \"sepal_length\": [7.2],\n",
    "        \"sepal_width\": [3.2],\n",
    "        \"petal_length\": [6.0],\n",
    "        \"petal_width\": [1.8]})\n",
    "    )\n",
    ")\n",
    "@output_schema(\n",
    "    output_type=StandardPythonParameterType({\n",
    "        \"Category\" : [\"Virginica\"]\n",
    "    })\n",
    ")\n",
    "``` \n",
    "This script produces the same output as before using the following [sample input](../../../../../cli/endpoints/online/managed/inference-schema/sample-inputs/pandas.json).\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"iris\": {\n",
    "        \"sepal_length\": {\n",
    "            \"0\": 7.2,\n",
    "            \"1\": 4.2\n",
    "        },\n",
    "        \"sepal_width\": {\n",
    "            \"0\": 3.2,\n",
    "            \"1\": 3.5\n",
    "        },\n",
    "        \"petal_length\": {\n",
    "            \"0\": 6.0,\n",
    "            \"1\": 1.0\n",
    "        },\n",
    "        \"petal_width\": {\n",
    "            \"0\": 1.8,\n",
    "            \"1\": 3.0\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Define the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ManagedOnlineDeployment(\n",
    "    name=\"infsrv-pandas\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    model=f\"azureml:{model.name}:{model.version}\",\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"inference-schema/code\", scoring_script=\"score-pandas.py\"\n",
    "    ),\n",
    "    environment=Environment(\n",
    "        image=\"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference\",\n",
    "        conda_file=\"inference-schema/env.yml\",\n",
    "    ),\n",
    "    instance_type=\"Standard_DS2_v2\",\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Create the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ml_client.online_deployments.begin_create_or_update(deployment).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Set traffic to 100%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"infsrv-pandas\": 100}\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Test the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=endpoint_name,\n",
    "    request_file=\"inference-schema/sample-inputs/pandas.json\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Get swagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = requests.get(url=endpoint.openapi_uri, headers={\"Authorization\": f\"Bearer {key}\"})\n",
    "print(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Delete assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "551a8cf72b4d3415950b8ca8640992b04259286a8f1f283af90604d953b514a0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}