From 842428c256e2d2e538745cab2f39cf249b5e491c Mon Sep 17 00:00:00 2001 From: Morgan Du Date: Mon, 26 Jul 2021 11:06:21 -0700 Subject: [PATCH] add notebooks official model monitoring (#15) * add notebooks official model monitoring * fix: update repo --- .../model_monitoring/model_monitoring.ipynb | 1090 +++++++++++++++++ 1 file changed, 1090 insertions(+) create mode 100644 notebooks/official/model_monitoring/model_monitoring.ipynb diff --git a/notebooks/official/model_monitoring/model_monitoring.ipynb b/notebooks/official/model_monitoring/model_monitoring.ipynb new file mode 100644 index 000000000..8449e4f90 --- /dev/null +++ b/notebooks/official/model_monitoring/model_monitoring.ipynb @@ -0,0 +1,1090 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# @title Copyright & License (click to expand)\n", + "# Copyright 2021 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fsv4jGuU89rX" + }, + "source": [ + "# Vertex Model Monitoring\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google Open in GCP Notebooks\n", + " \n", + " \n", + " \n", + " \"Colab Open in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lA32H1oKGgpf" + }, + "source": [ + "## Overview" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t6Cd51FkG09E" + }, + "source": [ + "### What is Model Monitoring?\n", + "\n", + "Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:\n", + "\n", + "* software versioning\n", + "* rigorous deployment processes\n", + "* event logging\n", + "* alerting/notication of situations requiring intervention\n", + "* on-demand and automated diagnostic tracing\n", + "* automated performance and functional testing\n", + "\n", + "You should be able to manage your ML services with the same degree of power and flexibility with which you can manage your applications. That's what MLOps is all about - managing ML services with the best practices Google and the broader computing industry have learned from generations of experience deploying well engineered, reliable, and scalable services.\n", + "\n", + "Model monitoring is only one piece of the ML Ops puzzle - it helps answer the following questions:\n", + "\n", + "* How well do recent service requests match the training data used to build your model? This is called **training-serving skew**.\n", + "* How significantly are service requests evolving over time? This is called **drift detection**.\n", + "\n", + "If production traffic differs from training data, or varies substantially over time, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yG7FcXWKHOhC" + }, + "source": [ + "### Objective\n", + "\n", + "In this notebook, you will learn how to... \n", + "\n", + "* deploy a pre-trained model\n", + "* configure model monitoring\n", + "* generate some artificial traffic\n", + "* understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8yVpQt-JHKPF" + }, + "source": [ + "### Costs \n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertext AI\n", + "* BigQuery\n", + "\n", + "Learn about [Vertext AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "### The example model\n", + "\n", + "The model you'll use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n", + "\n", + "- identity - unique player identitity numbers\n", + "- demographic features - information about the player, such as the geographic region in which a player is located\n", + "- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level\n", + "- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n", + "\n", + "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process it for use in machine learning, and train a model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Google Cloud Storage. In the next section, you will setup your environment and import this model into your own project." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ze4-nDLfK4pw" + }, + "source": [ + "## Before you begin" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i7EUnXsZhAGF" + }, + "source": [ + "### Setup your dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "020040f91150" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "assert sys.version_info.major == 3, \"This notebook requires Python 3.\"\n", + "\n", + "# Google Cloud Notebook requires dependencies to be installed with '--user'\n", + "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n", + "USER_FLAG = \"\"\n", + "if IS_GOOGLE_CLOUD_NOTEBOOK:\n", + " USER_FLAG = \"--user\"\n", + "\n", + "if 'google.colab' in sys.modules: \n", + " from google.colab import auth\n", + " auth.authenticate_user()\n", + "\n", + "# Install Python package dependencies.\n", + "! pip3 install {USER_FLAG} --quiet --upgrade google-api-python-client google-auth-oauthlib \\\n", + " google-auth-httplib2 oauth2client requests \\\n", + " google-cloud-aiplatform google-cloud-storage==1.32.0\n", + "\n", + "if not os.getenv(\"IS_TESTING\"): \n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + " app = IPython.Application.instance() \n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BF1j6f9HApxa" + }, + "source": [ + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "1. Enter your project id in the first line of the cell below.\n", + "\n", + "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", + "\n", + "1. You'll use the *gcloud* command throughout this notebook. In the following cell, enter your project name and run the cell to authenticate yourself with the Google Cloud and initialize your *gcloud* configuration settings.\n", + "\n", + "**Model monitoring is currently supported in regions us-central1, europe-west4, asia-east1, and asia-southeast1. To keep things simple for this lab, we're going to use region us-central1 for all our resources (BigQuery training data, Cloud Storage bucket, model and endpoint locations, etc.). You can use any supported region, so long as all resources are co-located.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "wxiE6dEWOFm3" + }, + "outputs": [], + "source": [ + "# Import globally needed dependencies here, after kernel restart.\n", + "import copy\n", + "import numpy as np\n", + "import os\n", + "import pprint as pp\n", + "import random\n", + "import sys\n", + "import time\n", + "\n", + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "REGION = \"us-central1\" # @param {type:\"string\"}\n", + "SUFFIX = \"aiplatform.googleapis.com\"\n", + "API_ENDPOINT = f\"{REGION}-{SUFFIX}\"\n", + "PREDICT_API_ENDPOINT = f\"{REGION}-prediction-{SUFFIX}\"\n", + "if os.getenv(\"IS_TESTING\"):\n", + " !gcloud --quiet components install beta\n", + " !gcloud --quiet components update\n", + "!gcloud config set project $PROJECT_ID\n", + "!gcloud config set ai/region $REGION" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sECqTau7Oh6M" + }, + "source": [ + "### Login to your Google Cloud account and enable AI services" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "C6H1vZYjvT6w" + }, + "outputs": [], + "source": [ + "# The Google Cloud Notebook product has specific requirements\n", + "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n", + "\n", + "# If on Google Cloud Notebooks, then don't execute this code\n", + "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''\n", + "\n", + "!gcloud services enable aiplatform.googleapis.com" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "btZeLzqQ7pXc" + }, + "source": [ + "### Define utilities\n", + "\n", + "Run the following cells to define some utility functions and distributions used later in this notebook. Although these utilities are not critical to understand the main concepts, feel free to expand the cells\n", + "in this section if you're curious or want to dive deeper into how some of your API requests are made." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "yhDFSB2YDvfT" + }, + "outputs": [], + "source": [ + "# @title Utility imports and constants\n", + "\n", + "from google.cloud.aiplatform_v1beta1.services.endpoint_service import \\\n", + " EndpointServiceClient\n", + "from google.cloud.aiplatform_v1beta1.services.job_service import \\\n", + " JobServiceClient\n", + "from google.cloud.aiplatform_v1beta1.services.prediction_service import \\\n", + " PredictionServiceClient\n", + "from google.cloud.aiplatform_v1beta1.types.io import BigQuerySource\n", + "from google.cloud.aiplatform_v1beta1.types.model_deployment_monitoring_job import (\n", + " ModelDeploymentMonitoringJob, ModelDeploymentMonitoringObjectiveConfig,\n", + " ModelDeploymentMonitoringScheduleConfig)\n", + "from google.cloud.aiplatform_v1beta1.types.model_monitoring import (\n", + " ModelMonitoringAlertConfig, ModelMonitoringObjectiveConfig,\n", + " SamplingStrategy, ThresholdConfig)\n", + "from google.cloud.aiplatform_v1beta1.types.prediction_service import \\\n", + " PredictRequest\n", + "from google.protobuf import json_format\n", + "from google.protobuf.duration_pb2 import Duration\n", + "from google.protobuf.struct_pb2 import Value\n", + "\n", + "# This is the default value at which you would like the monitoring function to trigger an alert.\n", + "# In other words, this value fine tunes the alerting sensitivity. This threshold can be customized\n", + "# on a per feature basis but this is the global default setting.\n", + "DEFAULT_THRESHOLD_VALUE = 0.001" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# @title Utility functions\n", + "\n", + "def create_monitoring_job(objective_configs):\n", + " # Create sampling configuration.\n", + " random_sampling = SamplingStrategy.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)\n", + " sampling_config = SamplingStrategy(random_sample_config=random_sampling)\n", + "\n", + " # Create schedule configuration.\n", + " duration = Duration(seconds=MONITOR_INTERVAL)\n", + " schedule_config = ModelDeploymentMonitoringScheduleConfig(monitor_interval=duration)\n", + "\n", + " # Create alerting configuration.\n", + " emails = [USER_EMAIL]\n", + " email_config = ModelMonitoringAlertConfig.EmailAlertConfig(user_emails=emails)\n", + " alerting_config = ModelMonitoringAlertConfig(email_alert_config=email_config)\n", + "\n", + " # Create the monitoring job.\n", + " endpoint = f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\"\n", + " predict_schema = \"\"\n", + " analysis_schema = \"\"\n", + " job = ModelDeploymentMonitoringJob(\n", + " display_name=JOB_NAME,\n", + " endpoint=endpoint,\n", + " model_deployment_monitoring_objective_configs=objective_configs,\n", + " logging_sampling_strategy=sampling_config,\n", + " model_deployment_monitoring_schedule_config=schedule_config,\n", + " model_monitoring_alert_config=alerting_config,\n", + " predict_instance_schema_uri=predict_schema,\n", + " analysis_instance_schema_uri=analysis_schema,\n", + " )\n", + " options = dict(api_endpoint=API_ENDPOINT)\n", + " client = JobServiceClient(client_options=options)\n", + " parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n", + " response = client.create_model_deployment_monitoring_job(\n", + " parent=parent, model_deployment_monitoring_job=job\n", + " )\n", + " print(\"Created monitoring job:\")\n", + " print(response)\n", + " return response\n", + "\n", + "\n", + "def get_thresholds(default_thresholds, custom_thresholds):\n", + " thresholds = {}\n", + " default_threshold = ThresholdConfig(value=DEFAULT_THRESHOLD_VALUE)\n", + " for feature in default_thresholds.split(\",\"):\n", + " feature = feature.strip()\n", + " thresholds[feature] = default_threshold\n", + " for custom_threshold in custom_thresholds.split(\",\"):\n", + " pair = custom_threshold.split(\":\")\n", + " if len(pair) != 2:\n", + " print(f\"Invalid custom skew threshold: {custom_threshold}\")\n", + " return\n", + " feature, value = pair\n", + " thresholds[feature] = ThresholdConfig(value=float(value))\n", + " return thresholds\n", + "\n", + "\n", + "def get_deployed_model_ids(endpoint_id):\n", + " client_options = dict(api_endpoint=API_ENDPOINT)\n", + " client = EndpointServiceClient(client_options=client_options)\n", + " parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n", + " response = client.get_endpoint(name=f\"{parent}/endpoints/{endpoint_id}\")\n", + " model_ids = []\n", + " for model in response.deployed_models:\n", + " model_ids.append(model.id)\n", + " return model_ids\n", + "\n", + "\n", + "def set_objectives(model_ids, objective_template):\n", + " # Use the same objective config for all models.\n", + " objective_configs = []\n", + " for model_id in model_ids:\n", + " objective_config = copy.deepcopy(objective_template)\n", + " objective_config.deployed_model_id = model_id\n", + " objective_configs.append(objective_config)\n", + " return objective_configs\n", + "\n", + "\n", + "def send_predict_request(endpoint, input):\n", + " client_options = {\"api_endpoint\": PREDICT_API_ENDPOINT}\n", + " client = PredictionServiceClient(client_options=client_options)\n", + " params = {}\n", + " params = json_format.ParseDict(params, Value())\n", + " request = PredictRequest(endpoint=endpoint, parameters=params)\n", + " inputs = [json_format.ParseDict(input, Value())]\n", + " request.instances.extend(inputs)\n", + " response = client.predict(request)\n", + " return response\n", + "\n", + "\n", + "def list_monitoring_jobs():\n", + " client_options = dict(api_endpoint=API_ENDPOINT)\n", + " parent = f\"projects/{PROJECT_ID}/locations/us-central1\"\n", + " client = JobServiceClient(client_options=client_options)\n", + " response = client.list_model_deployment_monitoring_jobs(parent=parent)\n", + " print(response)\n", + "\n", + "\n", + "def pause_monitoring_job(job):\n", + " client_options = dict(api_endpoint=API_ENDPOINT)\n", + " client = JobServiceClient(client_options=client_options)\n", + " response = client.pause_model_deployment_monitoring_job(name=job)\n", + " print(response)\n", + "\n", + "\n", + "def delete_monitoring_job(job):\n", + " client_options = dict(api_endpoint=API_ENDPOINT)\n", + " client = JobServiceClient(client_options=client_options)\n", + " response = client.delete_model_deployment_monitoring_job(name=job)\n", + " print(response)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# @title Utility distributions\n", + "\n", + "# This cell containers parameters enabling us to generate realistic test data that closely \n", + "# models the feature distributions found in the training data.\n", + "DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}\n", + "LANGUAGE = {\n", + " \"en-us\": 4807,\n", + " \"en-gb\": 678,\n", + " \"ja-jp\": 419,\n", + " \"en-au\": 310,\n", + " \"en-ca\": 299,\n", + " \"de-de\": 147,\n", + " \"en-in\": 130,\n", + " \"en\": 127,\n", + " \"fr-fr\": 94,\n", + " \"pt-br\": 81,\n", + " \"es-us\": 65,\n", + " \"zh-tw\": 64,\n", + " \"zh-hans-cn\": 55,\n", + " \"es-mx\": 53,\n", + " \"nl-nl\": 37,\n", + " \"fr-ca\": 34,\n", + " \"en-za\": 29,\n", + " \"vi-vn\": 29,\n", + " \"en-nz\": 29,\n", + " \"es-es\": 25,\n", + "}\n", + "OS = {\"IOS\": 3980, \"ANDROID\": 3798, \"null\": 253}\n", + "MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}\n", + "COUNTRY = {\n", + " \"United States\": 4395,\n", + " \"India\": 486,\n", + " \"Japan\": 450,\n", + " \"Canada\": 354,\n", + " \"Australia\": 327,\n", + " \"United Kingdom\": 303,\n", + " \"Germany\": 144,\n", + " \"Mexico\": 102,\n", + " \"France\": 97,\n", + " \"Brazil\": 93,\n", + " \"Taiwan\": 72,\n", + " \"China\": 65,\n", + " \"Saudi Arabia\": 49,\n", + " \"Pakistan\": 48,\n", + " \"Egypt\": 46,\n", + " \"Netherlands\": 45,\n", + " \"Vietnam\": 42,\n", + " \"Philippines\": 39,\n", + " \"South Africa\": 38,\n", + "}\n", + "\n", + "# Means and standard deviations for numerical features...\n", + "MEAN_SD = {\n", + " \"julianday\": (204.6, 34.7),\n", + " \"cnt_user_engagement\": (30.8, 53.2),\n", + " \"cnt_level_start_quickplay\": (7.8, 28.9),\n", + " \"cnt_level_end_quickplay\": (5.0, 16.4),\n", + " \"cnt_level_complete_quickplay\": (2.1, 9.9),\n", + " \"cnt_level_reset_quickplay\": (2.0, 19.6),\n", + " \"cnt_post_score\": (4.9, 13.8),\n", + " \"cnt_spend_virtual_currency\": (0.4, 1.8),\n", + " \"cnt_ad_reward\": (0.1, 0.6),\n", + " \"cnt_challenge_a_friend\": (0.0, 0.3),\n", + " \"cnt_completed_5_levels\": (0.1, 0.4),\n", + " \"cnt_use_extra_steps\": (0.4, 1.7),\n", + "}\n", + "\n", + "DEFAULT_INPUT = {\n", + " \"cnt_ad_reward\": 0,\n", + " \"cnt_challenge_a_friend\": 0,\n", + " \"cnt_completed_5_levels\": 1,\n", + " \"cnt_level_complete_quickplay\": 3,\n", + " \"cnt_level_end_quickplay\": 5,\n", + " \"cnt_level_reset_quickplay\": 2,\n", + " \"cnt_level_start_quickplay\": 6,\n", + " \"cnt_post_score\": 34,\n", + " \"cnt_spend_virtual_currency\": 0,\n", + " \"cnt_use_extra_steps\": 0,\n", + " \"cnt_user_engagement\": 120,\n", + " \"country\": \"Denmark\",\n", + " \"dayofweek\": 3,\n", + " \"julianday\": 254,\n", + " \"language\": \"da-dk\",\n", + " \"month\": 9,\n", + " \"operating_system\": \"IOS\",\n", + " \"user_pseudo_id\": \"104B0770BAE16E8B53DF330C95881893\",\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lAOk8UqvCL0S" + }, + "source": [ + "## Import your model\n", + "\n", + "The churn propensity model you'll be using in this notebook has been trained in BigQuery ML and exported to a Google Cloud Storage bucket. This illustrates how you can easily export a trained model and move a model from one cloud service to another. \n", + "\n", + "Run the next cell to import this model into your project. **If you've already imported your model, you can skip this step.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Se5DxJoK3AMz" + }, + "outputs": [], + "source": [ + "MODEL_NAME = \"churn\"\n", + "IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-4:latest\"\n", + "ARTIFACT = \"gs://mco-mm/churn\"\n", + "output = !gcloud --quiet beta ai models upload --container-image-uri=$IMAGE --artifact-uri=$ARTIFACT --display-name=$MODEL_NAME --format=\"value(model)\"\n", + "MODEL_ID = output[1].split(\"/\")[-1]\n", + "if _exit_code == 0:\n", + " print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")\n", + "else:\n", + " print(f\"Error creating model: {output}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "L1HzDTh-JRsr" + }, + "source": [ + "## Deploy your endpoint\n", + "\n", + "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet. Your endpoint is also serverless, in the sense that Google ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n", + "\n", + "Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete. If you've already deployed a model to an endpoint, you can reuse your endpoint by running the cell after the next one.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "M4nHQTVh7PIi" + }, + "outputs": [], + "source": [ + "ENDPOINT_NAME = \"churn\"\n", + "output = !gcloud --quiet beta ai endpoints create --display-name=$ENDPOINT_NAME --format=\"value(name)\"\n", + "if _exit_code == 0:\n", + " print(\"Endpoint created.\")\n", + "else:\n", + " print(f\"Error creating endpoint: {output}\")\n", + "ENDPOINT = output[-1]\n", + "ENDPOINT_ID = ENDPOINT.split(\"/\")[-1]\n", + "output = !gcloud --quiet beta ai endpoints deploy-model $ENDPOINT_ID --display-name=$ENDPOINT_NAME --model=$MODEL_ID --traffic-split=\"0=100\"\n", + "DEPLOYED_MODEL_ID = output[-1].split()[-1][:-1]\n", + "if _exit_code == 0:\n", + " print(\n", + " f\"Model {MODEL_NAME}/{MODEL_ID} deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}.\"\n", + " )\n", + "else:\n", + " print(f\"Error deploying model to endpoint: {output}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lAOk8UqvCL0S" + }, + "source": [ + "### If you already have a deployed endpoint\n", + "\n", + "You can reuse your existing endpoint by filling in the value of your endpoint ID in the next cell and running it. **If you've just deployed an endpoint in the previous cell, you should skip this step.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QzZS5piBeBFI" + }, + "outputs": [], + "source": [ + "# @title Run this cell only if you want to reuse an existing endpoint.\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " ENDPOINT_ID = \"\" # @param {type:\"string\"}\n", + " if ENDPOINT_ID:\n", + " ENDPOINT = f\"projects/{PROJECT_ID}/locations/us-central1/endpoints/{ENDPOINT_ID}\"\n", + " print(f\"Using endpoint {ENDPOINT}\")\n", + " else:\n", + " print(\"If you want to reuse an existing endpoint, you must specify the endpoint id above.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NKsA_lfl9Ryw" + }, + "source": [ + "## Run a prediction test\n", + "\n", + "Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON.\n", + "\n", + "**Try this now by running the next cell and examine the results.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QNEb7fDJ9NXc" + }, + "outputs": [], + "source": [ + "print(ENDPOINT)\n", + "print(\"request:\")\n", + "pp.pprint(DEFAULT_INPUT)\n", + "try:\n", + " resp = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n", + " print(\"response\")\n", + " pp.pprint(resp)\n", + "except Exception:\n", + " print(\"prediction request failed\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lN6ADMJqeWtR" + }, + "source": [ + "Taking a closer look at the results, we see the following elements:\n", + "\n", + "- **churned_values** - a set of possible values (0 and 1) for the target field\n", + "- **churned_probs** - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)\n", + "- **predicted_churn** - based on the probabilities, the predicted value of the target field (1)\n", + "\n", + "This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8rF5iLuXCT7i" + }, + "source": [ + "## Start your monitoring job\n", + "\n", + "Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.\n", + "\n", + "In this section, you will configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wW2gLBQ3Zkhq" + }, + "source": [ + "### Configure the following fields:\n", + "\n", + "1. User email - The email address to which you would like monitoring alerts sent.\n", + "1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n", + "1. Monitor interval - The time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds).\n", + "1. Target field - The prediction target column name in training dataset.\n", + "1. Skew detection threshold - The skew threshold for each feature you want to monitor.\n", + "1. Prediction drift threshold - The drift threshold for each feature you want to monitor." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "plpASmM2YIVO" + }, + "outputs": [], + "source": [ + "USER_EMAIL = \"[your-email-address]\" # @param {type:\"string\"}\n", + "JOB_NAME = \"churn\"\n", + "\n", + "# Sampling rate (optional, default=.8)\n", + "LOG_SAMPLE_RATE = 0.8 # @param {type:\"number\"}\n", + "\n", + "# Monitoring Interval in seconds (optional, default=3600).\n", + "MONITOR_INTERVAL = 3600 # @param {type:\"number\"}\n", + "\n", + "# URI to training dataset.\n", + "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\" # @param {type:\"string\"}\n", + "# Prediction target column name in training dataset.\n", + "TARGET = \"churned\"\n", + "\n", + "# Skew and drift thresholds.\n", + "SKEW_DEFAULT_THRESHOLDS = \"country,language\" # @param {type:\"string\"}\n", + "SKEW_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" # @param {type:\"string\"}\n", + "DRIFT_DEFAULT_THRESHOLDS = \"country,language\" # @param {type:\"string\"}\n", + "DRIFT_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mjVSViZR-dP2" + }, + "source": [ + "### Create your monitoring job\n", + "\n", + "The following code uses the Google Python client library to translate your configuration settings into a programmatic request to start a model monitoring job. To do this successfully, you need to specify your alerting thresholds (for both skew and drift), your training data source, and apply those settings to all deployed models on your new endpoint (of which there should only be one at this point).\n", + "\n", + "Instantiating a monitoring job can take some time. If everything looks good with your request, you'll get a successful API response. Then, you'll need to check your email to receive a notification that the job is running." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-62TYm2iYv3K" + }, + "outputs": [], + "source": [ + "# Set thresholds specifying alerting criteria for training/serving skew and create config object.\n", + "skew_thresholds = get_thresholds(SKEW_DEFAULT_THRESHOLDS, SKEW_CUSTOM_THRESHOLDS)\n", + "skew_config = ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(\n", + " skew_thresholds=skew_thresholds\n", + ")\n", + "\n", + "# Set thresholds specifying alerting criteria for serving drift and create config object.\n", + "drift_thresholds = get_thresholds(DRIFT_DEFAULT_THRESHOLDS, DRIFT_CUSTOM_THRESHOLDS)\n", + "drift_config = ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(\n", + " drift_thresholds=drift_thresholds\n", + ")\n", + "\n", + "# Specify training dataset source location (used for schema generation).\n", + "training_dataset = ModelMonitoringObjectiveConfig.TrainingDataset(target_field=TARGET)\n", + "training_dataset.bigquery_source = BigQuerySource(input_uri=DATASET_BQ_URI)\n", + "\n", + "# Aggregate the above settings into a ModelMonitoringObjectiveConfig object and use\n", + "# that object to adjust the ModelDeploymentMonitoringObjectiveConfig object.\n", + "objective_config = ModelMonitoringObjectiveConfig(\n", + " training_dataset=training_dataset,\n", + " training_prediction_skew_detection_config=skew_config,\n", + " prediction_drift_detection_config=drift_config,\n", + ")\n", + "objective_template = ModelDeploymentMonitoringObjectiveConfig(\n", + " objective_config=objective_config\n", + ")\n", + "\n", + "# Find all deployed model ids on the created endpoint and set objectives for each.\n", + "model_ids = get_deployed_model_ids(ENDPOINT_ID)\n", + "objective_configs = set_objectives(model_ids, objective_template)\n", + "\n", + "# Create the monitoring job for all deployed models on this endpoint.\n", + "monitoring_job = create_monitoring_job(objective_configs)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OiwOVR4D_xhl" + }, + "outputs": [], + "source": [ + "# Run a prediction request to generate schema, if necessary.\n", + "try:\n", + " _ = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n", + " print(\"prediction succeeded\")\n", + "except Exception:\n", + " print(\"prediction failed\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SaXYVFFslRru" + }, + "source": [ + "After a minute or two, you should receive email at the address you configured above for USER_EMAIL. This email confirms successful deployment of your monitoring job. Here's a sample of what this email might look like:\n", + "
\n", + "
\n", + "\n", + "
\n", + "As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to take a look at your measurements in Cloud Storage.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XV-vru2Pm1oX" + }, + "outputs": [], + "source": [ + "!gsutil ls gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/*/*" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XgUwU0sDpUUD" + }, + "source": [ + "You will notice the following components in these Cloud Storage paths:\n", + "\n", + "- **cloud-ai-platform-..** - This is a bucket created for you and assigned to capture your service's prediction data. Each monitoring job you create will trigger creation of a new folder in this bucket.\n", + "- **[model_monitoring|instance_schemas]/job-..** - This is your unique monitoring job number, which you can see above in both the response to your job creation requesst and the email notification. \n", + "- **instance_schemas/job-../analysis** - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).\n", + "- **instance_schemas/job-../predict** - This is the first prediction made to your model after the current monitoring job was enabled.\n", + "- **model_monitoring/job-../serving** - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.\n", + "- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8V2zo7-MMd7G" + }, + "source": [ + "### You can create monitoring jobs with other user interfaces\n", + "\n", + "In the previous cells, you created a monitoring job using the Python client library. You can also use the *gcloud* command line tool to create a model monitoring job and, in the near future, you will be able to use the Cloud Console, as well for this function. \n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Q106INuFCXKX" + }, + "source": [ + "## Generate test data to trigger alerting\n", + "\n", + "Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. It takes about five minutes to run this cell and at least an hour to assess and report anamolies in skew or drift so after running this cell, feel free to proceed with the notebook and you'll see how to examine the resulting alert later." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "obZYLLAuKmG8" + }, + "outputs": [], + "source": [ + "def random_uid():\n", + " digits = [str(i) for i in range(10)] + [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\n", + " return \"\".join(random.choices(digits, k=32))\n", + "\n", + "\n", + "def monitoring_test(count, sleep, perturb_num={}, perturb_cat={}):\n", + " # Use random sampling and mean/sd with gaussian distribution to model\n", + " # training data. Then modify sampling distros for two categorical features\n", + " # and mean/sd for two numerical features.\n", + " mean_sd = MEAN_SD.copy()\n", + " country = COUNTRY.copy()\n", + " for k, (mean_fn, sd_fn) in perturb_num.items():\n", + " orig_mean, orig_sd = MEAN_SD[k]\n", + " mean_sd[k] = (mean_fn(orig_mean), sd_fn(orig_sd))\n", + " for k, v in perturb_cat.items():\n", + " country[k] = v\n", + " for i in range(0, count):\n", + " input = DEFAULT_INPUT.copy()\n", + " input[\"user_pseudo_id\"] = str(random_uid())\n", + " input[\"country\"] = random.choices([*country], list(country.values()))[0]\n", + " input[\"dayofweek\"] = random.choices([*DAYOFWEEK], list(DAYOFWEEK.values()))[0]\n", + " input[\"language\"] = str(random.choices([*LANGUAGE], list(LANGUAGE.values()))[0])\n", + " input[\"operating_system\"] = str(random.choices([*OS], list(OS.values()))[0])\n", + " input[\"month\"] = random.choices([*MONTH], list(MONTH.values()))[0]\n", + " for key, (mean, sd) in mean_sd.items():\n", + " sample_val = round(float(np.random.normal(mean, sd, 1)))\n", + " val = max(sample_val, 0)\n", + " input[key] = val\n", + " print(f\"Sending prediction {i}\")\n", + " try:\n", + " send_predict_request(ENDPOINT, input)\n", + " except Exception:\n", + " print(\"prediction request failed\")\n", + " time.sleep(sleep)\n", + " print(\"Test Completed.\")\n", + "\n", + "\n", + "test_time = 300\n", + "tests_per_sec = 1\n", + "sleep_time = 1 / tests_per_sec\n", + "iterations = test_time * tests_per_sec\n", + "perturb_num = {\"cnt_user_engagement\": (lambda x: x * 3, lambda x: x / 3)}\n", + "perturb_cat = {\"Japan\": max(COUNTRY.values()) * 2}\n", + "monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bQohDTJgLQlW" + }, + "source": [ + "## Interpret your results\n", + "\n", + "While waiting for your results, which, as noted, may take up to an hour, you can read ahead to get sense of the alerting experience." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uGPI92qbOFUR" + }, + "source": [ + "### Here's what a sample email alert looks like...\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HoaqsxpaRs1m" + }, + "source": [ + "This email is warning you that the *cnt_user_engagement*, *country* and *language* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* feature value is drifting significantly over time, again, as per your threshold specification." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "w4jVIVq4VzB_" + }, + "source": [ + "### Monitoring results in the Cloud Console\n", + "\n", + "You can examine your model monitoring data from the Cloud Console. Below is a screenshot of those capabilities." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2OdIMVBAPZi_" + }, + "source": [ + "#### Monitoring Status\n", + "\n", + "\n", + "\n", + "#### Monitoring Alerts\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TpV-iwP9qw9c" + }, + "source": [ + "## Clean up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TPP_ImwJDFJf" + }, + "outputs": [], + "source": [ + "out = !gcloud ai endpoints undeploy-model $ENDPOINT_ID --deployed-model-id $DEPLOYED_MODEL_ID\n", + "if _exit_code == 0:\n", + " print(\"Model undeployed.\")\n", + "else:\n", + " print(\"Error undeploying model:\", out)\n", + "\n", + "out = !gcloud ai endpoints delete $ENDPOINT_ID --quiet\n", + "if _exit_code == 0:\n", + " print(\"Endpoint deleted.\")\n", + "else:\n", + " print(\"Error deleting endpoint:\", out)\n", + "\n", + "out = !gcloud ai models delete $MODEL_ID --quiet\n", + "if _exit_code == 0:\n", + " print(\"Model deleted.\")\n", + "else:\n", + " print(\"Error deleting model:\", out)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "j3Dh15h3-NoO" + }, + "source": [ + "## Learn more about model monitoring\n", + "\n", + "**Congratulations!** You've now learned what model monitoring is, how to configure and enable it, and how to find and interpret the results. Check out the following resources to learn more about model monitoring and ML Ops.\n", + "\n", + "- [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv)\n", + "- [Data Understanding, Validation, and Monitoring At Scale](https://blog.tensorflow.org/2018/09/introducing-tensorflow-data-validation.html)\n", + "- [Vertex Product Documentation](https://cloud.google.com/vertex)\n", + "- [Model Monitoring Reference Docs](https://cloud.google.com/vertex/docs/reference)\n", + "- [Model Monitoring blog article]()" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [ + "lA32H1oKGgpf" + ], + "name": "model_monitoring.ipynb", + "toc_visible": true + }, + "environment": { + "name": "tf2-gpu.2-3.m71", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m71" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}