bqml/07_model_inference.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aa560ef-ebcb-4de6-88e7-5dee8f1d8f9e",
   "metadata": {
    "id": "ur8xi4C7S06n",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Copyright 2022 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d1c73d-9e36-45ec-9a6f-f49c23f90241",
   "metadata": {
    "id": "JAPoU8Sm5E6e",
    "tags": []
   },
   "source": [
    "# Fraudfinder - Model Inference\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/fraudfinder/raw/main/07_model_inference.ipynb\">\n",
    "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Cloud Notebook\n",
    "    </a>\n",
    "  </td> \n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/fraudfinder/blob/main/07_model_inference.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/fraudfinder/blob/main/07_model_inference.ipynb\">\n",
    "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a7ffa73-2e9f-44a9-b88e-3d5eb156bcc6",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "[Fraudfinder](https://github.com/googlecloudplatform/fraudfinder) is a series of labs on how to build a real-time fraud detection system on Google Cloud. Throughout the Fraudfinder labs, you will learn how to read historical bank transaction data stored in data warehouse, read from a live stream of new transactions, perform exploratory data analysis (EDA), do feature engineering, ingest features into a feature store, train a model using feature store, register your model in a model registry, evaluate your model, deploy your model to an endpoint, do real-time inference on your model with feature store, and monitor your model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee88f6f-ee5f-4664-b308-f8a0fc854cf9",
   "metadata": {
    "id": "tvgnzT1CKxrO",
    "tags": []
   },
   "source": [
    "### Objective\n",
    "\n",
    "In this notebook, you will create a Cloud Run app to perform model inference on the endpoint deployed in the previous notebooks. This Cloud Run app will be triggered by the Pub/Sub subscriber for live transactions, perform a look-up on feature values from the feature store you created in earlier notebooks, and send the prediction request to the Vertex AI endpoint. You can then view the resulting prediction-response logs in BigQuery.\n",
    "\n",
    "This lab uses the following Google Cloud services and resources:\n",
    "\n",
    "- [Vertex AI](https://cloud.google.com/vertex-ai/)\n",
    "- [BigQuery](https://cloud.google.com/bigquery/)\n",
    "- [Cloud Run](https://cloud.google.com/run)\n",
    "- [Pub/Sub](https://cloud.google.com/pubsub/)\n",
    "\n",
    "Steps performed in this notebook:\n",
    "\n",
    "- Build and deploy a Cloud Run app for model inference\n",
    "- Create and use a Pub/Sub push subscription to invoke the Cloud Run model inference app\n",
    "- Inspect the prediction-responses of the endpoint in BigQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "706e3e1e-69d0-451f-a092-960fc88d22c9",
   "metadata": {
    "id": "ze4-nDLfK4pw"
   },
   "source": [
    "### Load config settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "125f883f-eebc-4607-8009-6cc34a2d7c1e",
   "metadata": {
    "id": "gCuSR8GkAgzl"
   },
   "outputs": [],
   "source": [
    "GCP_PROJECTS = !gcloud config get-value project\n",
    "PROJECT_ID = GCP_PROJECTS[0]\n",
    "BUCKET_NAME = f\"{PROJECT_ID}-fraudfinder\"\n",
    "config = !gsutil cat gs://{BUCKET_NAME}/config/notebook_env.py\n",
    "print(config.n)\n",
    "exec(config.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfceee97-a7b9-4323-ba10-6171a02362c2",
   "metadata": {},
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c32029-7664-442c-a58b-93f30ecc39d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "PAYLOAD_SCHEMA = {\n",
    "    \"tx_amount\": \"float64\",\n",
    "    \"customer_id_nb_tx_1day_window\": \"int64\",\n",
    "    \"customer_id_nb_tx_7day_window\": \"int64\",\n",
    "    \"customer_id_nb_tx_14day_window\": \"int64\",\n",
    "    \"customer_id_avg_amount_1day_window\": \"float64\",\n",
    "    \"customer_id_avg_amount_7day_window\": \"float64\",\n",
    "    \"customer_id_avg_amount_14day_window\": \"float64\",\n",
    "    \"customer_id_nb_tx_15min_window\": \"int64\",\n",
    "    \"customer_id_avg_amount_15min_window\": \"float64\",\n",
    "    \"customer_id_nb_tx_30min_window\": \"int64\",\n",
    "    \"customer_id_avg_amount_30min_window\": \"float64\",\n",
    "    \"customer_id_nb_tx_60min_window\": \"int64\",\n",
    "    \"customer_id_avg_amount_60min_window\": \"float64\",\n",
    "    \"terminal_id_nb_tx_1day_window\": \"int64\",\n",
    "    \"terminal_id_nb_tx_7day_window\": \"int64\",\n",
    "    \"terminal_id_nb_tx_14day_window\": \"int64\",\n",
    "    \"terminal_id_risk_1day_window\": \"float64\",\n",
    "    \"terminal_id_risk_7day_window\": \"float64\",\n",
    "    \"terminal_id_risk_14day_window\": \"float64\",\n",
    "    \"terminal_id_nb_tx_15min_window\": \"int64\",\n",
    "    \"terminal_id_avg_amount_15min_window\": \"float64\",\n",
    "    \"terminal_id_nb_tx_30min_window\": \"int64\",\n",
    "    \"terminal_id_avg_amount_30min_window\": \"float64\",\n",
    "    \"terminal_id_nb_tx_60min_window\": \"int64\",\n",
    "    \"terminal_id_avg_amount_60min_window\": \"float64\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5aa00a6-7682-44a1-87ad-0613b1fd984b",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbe4b059-0018-4466-8bc0-8482e70ee21c",
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "from google.cloud.aiplatform import Featurestore, EntityType, Feature\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud import aiplatform as vertex_ai\n",
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb54a14-d9f0-4cb2-814c-4662739d8b9d",
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "source": [
    "### Initialize Vertex AI for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df2729ca-f3b0-43e1-b30c-e9daa26a592f",
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d030e8ec-6b29-4477-a92c-122775ed9925",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Build and deploy a Cloud Run app for model inference\n",
    "To formalize the process of prediction, you will use a Cloud Run app that takes in live transactions as a trigger, then fetches feature values from Vertex AI Feature Store, then sends the prediction payload to an endpoint. To clarify, to invoke the Cloud Run app, you will create a Pub/Sub push subscription that reads live transactions from the public Pub/Sub topic to invoke the Cloud Run app.\n",
    "\n",
    "[Cloud Run](https://cloud.google.com/run) is a serverless compute platform that enables you to deploy containers that can be executed every time it is triggered. \n",
    "\n",
    "#### Steps to build and deploy the Cloud Run app\n",
    "\n",
    "To deploy a Cloud Run app, you must:\n",
    "1. Build a Docker container with your code\n",
    "2. Deploy your container to Cloud Run"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e547a9bc-7fd9-4d05-8bd1-96034e79eced",
   "metadata": {},
   "source": [
    "### 1. Build a Docker container with your code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d97d86-0585-43b0-8050-875343bbb5aa",
   "metadata": {},
   "source": [
    "The container code has been prepared for you in the `cloud_run_model_inference/` folder, which you can use to build and submit to automatically to Google Container Registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f6213f-195e-4e27-bf6d-6db48d7064ac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!gcloud builds submit ../scripts/cloud_run_model_inference --tag gcr.io/$PROJECT_ID/cloud_run_model_inference --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c25b5bd-9463-4544-b072-757c0acad108",
   "metadata": {},
   "source": [
    "### 2. Deploy your container to Cloud Run\n",
    "\n",
    "With your container built on Container Registry, you can now deploy it to Cloud Run.\n",
    "\n",
    "To do so, you will need some environment variables to make sure your Cloud Run app knows which Vertex AI endpoint to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9aa8e82-0b39-4f40-a059-ef8d056a13de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve your Vertex AI endpoint name\n",
    "endpoints = vertex_ai.Endpoint.list(\n",
    "    filter=f'display_name={ENDPOINT_NAME}', # optional: filter by specific endpoint name\n",
    "    order_by=\"update_time\"\n",
    ")\n",
    "ENDPOINT_ID = endpoints[-1].name\n",
    "print(ENDPOINT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d7dc061-1298-4055-a33f-c07dd525977d",
   "metadata": {},
   "source": [
    "The following cell will deploy the container as an app on Cloud Run, which you can then check on https://console.cloud.google.com/run. \n",
    "\n",
    "Note that once deployed, if you try to visit the Service URL (which may look like http://cloud-run-model-inference-app-XXXXXX-a.run.app), you should expect to see `Error: Forbidden\n",
    "Your client does not have permission to get URL / from this server`, which is normal, as you don't want the public internet to invoke your app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df81ca0c-b11e-46e1-8b5e-2243732ab9dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!gcloud run deploy cloud-run-model-inference-app \\\n",
    "--image gcr.io/{PROJECT_ID}/cloud_run_model_inference \\\n",
    "--no-allow-unauthenticated \\\n",
    "--region $REGION \\\n",
    "--update-env-vars FEATURESTORE_ID=$FEATURESTORE_ID,ENDPOINT_ID=$ENDPOINT_ID,PROJECT_ID=$PROJECT_ID,REGION=$REGION \\\n",
    "--quiet --verbosity=none"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16f682ea-003d-42c6-a8e9-3fbfb80eb3e8",
   "metadata": {},
   "source": [
    "You have now deployed a Cloud Run app to do model inference. However, it is not currently triggered by anything. In the next section, you will connect your Cloud Run app to the live transactions so you can continuously trigger your model inference app."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6154b978-c5f9-4503-963c-17df78ed9d2f",
   "metadata": {},
   "source": [
    "## Create and use a Pub/Sub push subscription to invoke the Cloud Run model inference app\n",
    "\n",
    "In this section, you will connect the live transactions to trigger your Cloud Run app. To do so, you will need to create a Pub/Sub push subscription from the live transactions (the public Pub/Sub topic), then use a service account to trigger your Cloud Run app.\n",
    "\n",
    "#### There are a few steps needed:\n",
    "1. Create a service account that can invoke your Cloud Run app with appropriate IAM policies\n",
    "2. Create the Pub/Sub subscription from the live transactions to invoke the Cloud Run app"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e0b5985-1ad6-44bf-b080-ad883fb038b0",
   "metadata": {},
   "source": [
    "### 1. Create a service account that can invoke your Cloud Run app with appropriate IAM policies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7968aa44-0ab4-43bb-96ef-5c9a62767d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure IAM API is enabled (if it is not, then enable in Cloud Shell: gcloud services enable iam.googleapis.com)\n",
    "\n",
    "# Create a service account\n",
    "!gcloud iam service-accounts create cloud-run-invoker --display-name \"Cloud Run Pub/Sub Invoker\"\n",
    "\n",
    "# Retrieve your project number\n",
    "PROJECT_NUMBER = !gcloud projects list --filter=\"$PROJECT_ID\" --format=\"value(PROJECT_NUMBER)\"\n",
    "PROJECT_NUMBER = PROJECT_NUMBER[0]\n",
    "\n",
    "# Bind the service account with an IAM policy to invoke the Cloud Run app\n",
    "!gcloud run services add-iam-policy-binding cloud-run-model-inference-app \\\n",
    "   --member=serviceAccount:cloud-run-invoker@{PROJECT_ID}.iam.gserviceaccount.com \\\n",
    "   --role=roles/run.invoker \\\n",
    "   --region=us-central1\n",
    "\n",
    "# Add another IAM policy to the service account to provide authentication needed to invoke Cloud Run\n",
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "     --member=serviceAccount:service-{PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com \\\n",
    "     --role=roles/iam.serviceAccountTokenCreator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a299c429-60be-45b4-937a-25b048b2ec73",
   "metadata": {},
   "source": [
    "### 2. Create the Pub/Sub subscription from the live transactions to invoke the Cloud Run app"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0ebd39c-5275-4dfb-8b11-5f2027b718da",
   "metadata": {},
   "source": [
    "With the service account read, now you can create a Pub/Sub push subscription to connect the live transactions (from the public Pub/Sub topic `ff-tx`) to trigger your Cloud Run app.\n",
    "\n",
    "In other words, as new transactions are received in the Pub/Sub topic, the push subscription will then automatically trigger the Cloud Run app, which processes the live data, retrieves values from Vertex AI Feature Store, then sends the prediction request to the Vertex AI endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad286fd5-2a11-4cd2-8b53-ca7ad01d9fc4",
   "metadata": {},
   "source": [
    "To create the Pub/Sub push subscription, you will first need to retrieve your Cloud Run service URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac18d705-b62d-4ba5-8569-0ab10726d293",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to get the service URL programmatically\n",
    "SERVICE_URL = !gcloud run services describe cloud-run-model-inference-app \\\n",
    "  --platform managed \\\n",
    "  --region $REGION \\\n",
    "  --format \"value(status.url)\"\n",
    "SERVICE_URL = SERVICE_URL[0]\n",
    "\n",
    "print(SERVICE_URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6839d5-6f11-4de5-be2c-0a426fdd2387",
   "metadata": {},
   "source": [
    "Now you can create your Pub/Sub push subscription:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebf2e721-cadc-44dc-93f4-149027e8a6b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud pubsub subscriptions create push-live-tx-to-cloudrun --topic projects/cymbal-fraudfinder/topics/ff-tx \\\n",
    "   --ack-deadline=600 \\\n",
    "   --push-endpoint=$SERVICE_URL \\\n",
    "   --push-auth-service-account=cloud-run-invoker@{PROJECT_ID}.iam.gserviceaccount.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df358d43-83f4-4155-8002-23636ba625cb",
   "metadata": {},
   "source": [
    "Once created, you can do some checks to make sure everything worked successfully:\n",
    "- On the [Pub/Sub page](https://console.cloud.google.com/cloudpubsub/subscription/list), inspect your new Pub/Sub subscription `push-live-tx-to-cloudrun`\n",
    "- On the [Cloud Run logs page](https://console.cloud.google.com/run/detail/us-central1/cloud-run-model-inference-app/logs), check the logs of your Cloud Run app to confirm that you see model prediction requests and responses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f81c197-f999-4494-a354-0db8850b4960",
   "metadata": {},
   "source": [
    "## Inspecting model prediction requests and responses in BigQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e93faa-dd02-46d8-87ff-bea7e51dd4e7",
   "metadata": {},
   "source": [
    "With Model Monitoring enabled on your Vertex AI endpoint, your endpoint will now automatically store all of your model predictions and responses in BigQuery.\n",
    "\n",
    "You may need to wait a few minutes before you start to see new rows updated in BigQuery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e761ef17-7dff-46d0-93ca-37aefddd9a8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "bq_client = bigquery.Client(project=PROJECT_ID)\n",
    "\n",
    "sql = f\"\"\"\n",
    "SELECT\n",
    "  *\n",
    "FROM\n",
    "  `model_deployment_monitoring_{ENDPOINT_ID}.serving_predict`\n",
    "ORDER BY\n",
    "  TIMESTAMP(logging_time) DESC\n",
    "LIMIT\n",
    "  100\n",
    "\"\"\"\n",
    "\n",
    "client_result = bq_client.query(sql, \n",
    "                                job_config=bigquery.QueryJobConfig()\n",
    "                               )\n",
    "df = client_result.result().to_arrow().to_pandas()\n",
    "df"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-cpu.2-10.m100",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-cpu.2-10:m100"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}