model_garden/notebooks/official/model_monitoring/model_monitoring.ipynb

In [ ]:

# @title Copyright & License (click to expand)
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Vertex AI Model Monitoring with Explainable AI Feature Attributions

Run in Colab

View on GitHub

Open in Vertex AI Workbench

Overview

What is Vertex AI Model Monitoring?

Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:

• software versioning
• rigorous deployment processes
• event logging
• alerting/notication of situations requiring intervention
• on-demand and automated diagnostic tracing
• automated performance and functional testing

You should be able to manage your ML services with the same degree of power and flexibility with which you can manage your applications. That's what MLOps is all about - managing ML services with the best practices Google and the broader computing industry have learned from generations of experience deploying well engineered, reliable, and scalable services.

Model monitoring is only one piece of the ML Ops puzzle - it helps answer the following questions:

• How well do recent service requests match the training data used to build your model? This is called training-serving skew.
• How significantly are service requests evolving over time? This is called drift detection.

Vertex Explainable AI adds another facet to model monitoring, which we call feature attribution monitoring. Explainable AI enables you to understand the relative contribution of each feature to a resulting prediction. In essence, it assesses the magnitude of each feature's influence.

If production traffic differs from training data, or varies substantially over time, either in terms of model predictions or feature attributions, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that you can anticipate problems before they affect your customer experiences or your revenue streams.

Learn more about Vertex AI Model Monitoring.

Objective

In this notebook, you learn to use the Vertex AI Model Monitoring service to detect drift and anomalies in prediction requests from a deployed Vertex AI Model resource.

This tutorial uses the following Google Cloud ML services:

• Vertex AI Model Monitoring
• Vertex AI Prediction
• Vertex AI Model resource
• Vertex AI Endpoint resource

The steps performed include:

• Upload a pre-trained model as a Vertex AI Model resource.
• Create an Vertex AI Endpoint resource.
• Deploy the Model resource to the Endpoint resource.
• Configure the Endpoint resource for model monitoring.
• Initialize the baseline distribution for model monitoring.
• Generate synthetic prediction requests.
• Understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature.

Model

This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. The model predicts for an online gaming site, the probability that a player may churn, i.e. stop being an active player.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• BigQuery
• Cloud Storage

Learn about Vertext AI pricing, Cloud Storage pricing, and BigQuery pricing and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Before you begin

Set up your local development environment

If you are using Colab or Vertex AI Workbench notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.

Otherwise, make sure your environment meets this notebook's requirements. You need the following:

• The Google Cloud SDK
• Git
• Python 3
• virtualenv
• Jupyter notebook running in a virtual environment with Python 3

The Google Cloud guide to setting up a Python development environment and the Jupyter installation guide provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:

1. Install and initialize the Cloud SDK.

2. Install Python 3.

3. Install virtualenv and create a virtual environment that uses Python 3. Activate the virtual environment.

4. To install Jupyter, run pip install jupyter on the command-line in a terminal shell.

5. To launch Jupyter, run jupyter notebook on the command-line in a terminal shell.

6. Open this notebook in the Jupyter Notebook dashboard.

Installation

Install the packages required for executing this notebook.

In [ ]:

import os
import sys

assert sys.version_info.major == 3, "This notebook requires Python 3."

# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME") and not os.getenv("VIRTUAL_ENV")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)

# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"

# Don't bother installing tensorflow or explainable_ai_sdk on Colab
extra_pkgs = "tensorflow==2.7 explainable_ai_sdk"
if "google.colab" in sys.modules:
  extra_pkgs = ""

# Install required packages.
! pip3 install --upgrade -q {USER_FLAG} \
    google-cloud-aiplatform \
    google-cloud-bigquery \
    explainable_ai_sdk \
    $extra_pkgs

Restart the kernel

After you install the SDK, you need to restart the notebook kernel so it can find the packages. You can restart kernel from Kernel -> Restart Kernel, or running the following:

In [ ]:

# Automatically restart kernel after installs
import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

Set up your Google Cloud project

The following steps are required, regardless of your notebook environment.

1. Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.

2. Make sure that billing is enabled for your project.

3. Enable the Vertex AI API and Compute Engine API.

4. If you are running this notebook locally, you will need to install the Cloud SDK.

5. Enter your project ID in the cell below. Then run the cell to make sure the Cloud SDK uses the right project for all the commands in this notebook.

Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.

Set your project ID

If you don't know your project ID, you may be able to get your project ID using gcloud.

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}

In [ ]:

if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)

In [ ]:

! gcloud config set project $PROJECT_ID

Region

You can also change the REGION variable, which is used for operations throughout the rest of this notebook. Below are regions supported for Vertex AI.

• Americas: us-central1
• Europe: europe-west4
• Asia Pacific: asia-east1

For this notebook, we recommend that you leave the region set to the default value us-central1.

You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.

Learn more about Vertex AI regions

In [ ]:

REGION = "[your-region]"  # @param {type: "string"}

if REGION == "[your-region]":
    REGION = "us-central1"

UUID

If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial.

In [ ]:

import random
import string


# Generate a uuid of a specifed length(default=8)
def generate_uuid(length: int = 8) -> str:
    return "".join(random.choices(string.ascii_lowercase + string.digits, k=length))


UUID = generate_uuid()

Authenticate your Google Cloud account

If you are using Vertex AI Workbench notebooks, your environment is already authenticated.

If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.

Otherwise, follow these steps:

1. In the Cloud Console, go to the Create service account key page.

2. Click Create service account.

3. In the Service account name field, enter a name, and click Create.

4. In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex AI" into the filter box, and select Vertex AI Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.

5. Click Create. A JSON file that contains your key downloads to your local environment.

6. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.

Login to your Google Cloud account and enable AI services

In [ ]:

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = "google.colab" in sys.modules
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''

Notes about service account and permission

By default no configuration is required, if you run into any permission related issue, please make sure the service accounts above have the required roles:

Service account email	Description	Roles
PROJECT_NUMBER-compute@developer.gserviceaccount.com	Compute Engine default service account	Dataflow Admin, Dataflow Worker, Storage Admin, BigQuery Admin, Vertex AI User
service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com	AI Platform Service Agent	Vertex AI Service Agent

1. Goto https://console.cloud.google.com/iam-admin/iam.
2. Check the "Include Google-provided role grants" checkbox.
3. Find the above emails.
4. Grant the corresponding roles.

Using data source from a different project

• For the BQ data source, grant both service accounts the "BigQuery Data Viewer" role.
• For the CSV data source, grant both service accounts the "Storage Object Viewer" role.

Import libraries and define constants

In [ ]:

# Import required packages.
import os
import pprint as pp
import sys

import google.cloud.aiplatform as aiplatform
import matplotlib.pyplot as plt
from google.cloud import bigquery
from google.cloud.aiplatform import model_monitoring
from google.cloud.aiplatform.explain.metadata.tf.v2 import \
    saved_model_metadata_builder

In [ ]:

if os.getenv("IS_TESTING"):
    ! gcloud --quiet components install beta
    ! gcloud --quiet components update

! gcloud config set ai/region $REGION
os.environ["GOOGLE_CLOUD_PROJECT"] = PROJECT_ID

Initialize Vertex AI SDK for Python

Initialize the Vertex AI SDK for Python for your project and corresponding bucket.

In [ ]:

aiplatform.init(project=PROJECT_ID, location=REGION)

Create BigQuery client

In this tutorial, you use data from the same public BigQuery table that was used to train the pre-trained model. You create a client interface, which you subsequently use to access the data.

In [ ]:

bqclient = bigquery.Client(project=PROJECT_ID)

The example model

The model you use in this notebook is based on this blog post. The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:

• identity - unique player identitity numbers
• demographic features - information about the player, such as the geographic region in which a player is located
• behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level
• churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.

The blog article referenced above explains how to use BigQuery to store the raw data, pre-process the data for machine learning, and train the corresponding model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Cloud Storage. In the next section, you will setup your environment and import this model into your own project.

Define some helper data structures

Run the following cell to define some data structures used throughout this notebook.

In [ ]:

# @title Utility data structures

# Sampling distributions for categorical features...
DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}

LANGUAGE = {
    "en-us": 4807,
    "en-gb": 678,
    "ja-jp": 419,
    "en-au": 310,
    "en-ca": 299,
    "de-de": 147,
    "en-in": 130,
    "en": 127,
    "fr-fr": 94,
    "pt-br": 81,
    "es-us": 65,
    "zh-tw": 64,
    "zh-hans-cn": 55,
    "es-mx": 53,
    "nl-nl": 37,
    "fr-ca": 34,
    "en-za": 29,
    "vi-vn": 29,
    "en-nz": 29,
    "es-es": 25,
}

OS = {"IOS": 3980, "ANDROID": 3798, "null": 253}

MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}

COUNTRY = {
    "United States": 4395,
    "India": 486,
    "Japan": 450,
    "Canada": 354,
    "Australia": 327,
    "United Kingdom": 303,
    "Germany": 144,
    "Mexico": 102,
    "France": 97,
    "Brazil": 93,
    "Taiwan": 72,
    "China": 65,
    "Saudi Arabia": 49,
    "Pakistan": 48,
    "Egypt": 46,
    "Netherlands": 45,
    "Vietnam": 42,
    "Philippines": 39,
    "South Africa": 38,
}

# Means and standard deviations for numerical features...
MEAN_SD = {
    "julianday": (204.6, 34.7),
    "cnt_user_engagement": (30.8, 53.2),
    "cnt_level_start_quickplay": (7.8, 28.9),
    "cnt_level_end_quickplay": (5.0, 16.4),
    "cnt_level_complete_quickplay": (2.1, 9.9),
    "cnt_level_reset_quickplay": (2.0, 19.6),
    "cnt_post_score": (4.9, 13.8),
    "cnt_spend_virtual_currency": (0.4, 1.8),
    "cnt_ad_reward": (0.1, 0.6),
    "cnt_challenge_a_friend": (0.0, 0.3),
    "cnt_completed_5_levels": (0.1, 0.4),
    "cnt_use_extra_steps": (0.4, 1.7),
}

DEFAULT_INPUT = {
    "cnt_ad_reward": 0,
    "cnt_challenge_a_friend": 0,
    "cnt_completed_5_levels": 1,
    "cnt_level_complete_quickplay": 3,
    "cnt_level_end_quickplay": 5,
    "cnt_level_reset_quickplay": 2,
    "cnt_level_start_quickplay": 6,
    "cnt_post_score": 34,
    "cnt_spend_virtual_currency": 0,
    "cnt_use_extra_steps": 0,
    "cnt_user_engagement": 120,
    "country": "Denmark",
    "dayofweek": 3,
    "julianday": 254,
    "language": "da-dk",
    "month": 9,
    "operating_system": "IOS",
    "user_pseudo_id": "104B0770BAE16E8B53DF330C95881893",
}

Generate model metadata for Vertex Explainable AI

Run the following cell to extract metadata from the exported model, which is needed for generating the explanations for a prediction request.

In [ ]:

MODEL_PATH = "gs://mco-mm/churn"

params = {"sampled_shapley_attribution": {"path_count": 10}}
EXPLAIN_PARAMS = aiplatform.explain.ExplanationParameters(params)

builder = saved_model_metadata_builder.SavedModelMetadataBuilder(
    model_path=MODEL_PATH, outputs_to_explain=["churned_probs"]
)
EXPLAIN_META = builder.get_metadata_protobuf()

Upload your model

The churn propensity model you use in this notebook has been trained in BigQuery ML and exported to a Cloud Storage bucket. This illustrates how you can easily export a trained model and move a model from one cloud service to another.

Run the next cell to import this model into your project. If you've already imported your model, you can skip this step.

In [ ]:

MODEL_NAME = "churn"
IMAGE = "us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-5:latest"

model = aiplatform.Model.upload(
    display_name=MODEL_NAME,
    artifact_uri=MODEL_PATH,
    serving_container_image_uri=IMAGE,
    explanation_parameters=EXPLAIN_PARAMS,
    explanation_metadata=EXPLAIN_META,
    sync=True,
)

MODEL_ID = model.resource_name.split("/")[-1]

Once the above cell completes, you should see a new model on the Vertex AI Model Registry page on the Cloud Console.

Deploy your Model resource to an Endpoint resource

Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you can make online prediction requests on your model via the public internet. Your endpoint is also serverless, in the sense that Google Cloud ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.

Run the next cell to deploy your model to an endpoint. This will take about ten minutes to complete.

In [ ]:

endpoint = model.deploy(machine_type="n1-standard-4")
print(f"endpoint display name: {endpoint.display_name}")
print(f"endpoint resource name: {endpoint.resource_name}")
ENDPOINT = endpoint.resource_name
ENDPOINT_ID = ENDPOINT.split("/")[-1]

Once the above cell completes, you should see a new endoint on the Vertex AI Endpoints page on the Cloud Console.

Run a prediction test

Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON, along with a pie chart summarizing the results.

Try this now by running the next cell.

In [ ]:

try:
    resp = endpoint.predict([DEFAULT_INPUT])
    for i in resp.predictions:
        vals = i["churned_values"]
        probs = i["churned_probs"]
    for i in range(len(vals)):
        print(vals[i], probs[i])
    plt.pie(probs, labels=vals)
    pp.pprint(resp)
except Exception as ex:
    print("prediction request failed", ex)

Test results

Taking a look at the results, you see the following elements:

• churned_values - a set of possible values (0 and 1) for the target field
• churned_probs - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)
• predicted_churn - based on the probabilities, the predicted value of the target field (1)

This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application.

Run an explanation test

You can run a test of Explainable AI on this endpoint. Run the next cell to send a test explanation request. The response you receive encodes the feature importance of this prediction in a text representation called JSON, along with a bar chart summarizing the results.

Try this now by running the next cell.

In [ ]:

try:
    features = []
    scores = []
    resp = endpoint.explain([DEFAULT_INPUT])
    for i in resp.explanations:
        for j in i.attributions:
            for k in j.feature_attributions:
                features.append(k)
                scores.append(j.feature_attributions[k])
    features = [x for _, x in sorted(zip(scores, features))]
    scores = sorted(scores)
    fig, ax = plt.subplots()
    fig.set_size_inches(9, 9)
    ax.barh(features, scores)
    fig.show()
except Exception as ex:
    print("explanation request failed", ex)

Start your monitoring job

Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.

In this section, you configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML.

Configure the following fields:

1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.
2. Monitor interval - time window over which to analyze your data and report anomalies. The minimum window is one hour (1)
3. Target field - prediction target column name in training dataset
4. Skew detection threshold - skew threshold for each feature you want to monitor
5. Prediction drift threshold - drift threshold for each feature you want to monitor
6. Attribution Skew detection threshold - feature importance skew threshold
7. Attribution Prediction drift threshold - feature importance drift threshold

In [ ]:

USER_EMAIL = "[your-email-address]"  # @param {type:"string"}
JOB_NAME = "churn"

# Sampling rate (optional, default=.8)
LOG_SAMPLE_RATE = 0.8  # @param {type:"number"}

# Monitoring Interval in hours (optional, default=1).
MONITOR_INTERVAL = 1  # @param {type:"number"}

# URI to training dataset.
DATASET_BQ_URI = "bq://mco-mm.bqmlga4.train"  # @param {type:"string"}
# Prediction target column name in training dataset.
TARGET = "churned"

# # Skew and drift thresholds.

DEFAULT_THRESHOLD_VALUE = 0.001

SKEW_THRESHOLDS = {
    "country": DEFAULT_THRESHOLD_VALUE,
    "cnt_user_engagement": DEFAULT_THRESHOLD_VALUE,
}
DRIFT_THRESHOLDS = {
    "country": DEFAULT_THRESHOLD_VALUE,
    "cnt_user_engagement": DEFAULT_THRESHOLD_VALUE,
}
ATTRIB_SKEW_THRESHOLDS = {
    "country": DEFAULT_THRESHOLD_VALUE,
    "cnt_user_engagement": DEFAULT_THRESHOLD_VALUE,
}
ATTRIB_DRIFT_THRESHOLDS = {
    "country": DEFAULT_THRESHOLD_VALUE,
    "cnt_user_engagement": DEFAULT_THRESHOLD_VALUE,
}

You can change the threshold values and the configuration settings, so that you can monitor other features in the model as well.

Create your monitoring job

The following code uses the Google Python client library to translate your configuration settings into a programmatic request to start a model monitoring job. Instantiating a monitoring job can take some time. If everything looks good with your request, you'll get a successful API response. Then, you'll need to check your email to receive a notification that the job is running.

In [ ]:

skew_config = model_monitoring.SkewDetectionConfig(
    data_source=DATASET_BQ_URI,
    skew_thresholds=SKEW_THRESHOLDS,
    attribute_skew_thresholds=ATTRIB_SKEW_THRESHOLDS,
    target_field=TARGET,
)

drift_config = model_monitoring.DriftDetectionConfig(
    drift_thresholds=DRIFT_THRESHOLDS,
    attribute_drift_thresholds=ATTRIB_DRIFT_THRESHOLDS,
)

explanation_config = model_monitoring.ExplanationConfig()
objective_config = model_monitoring.ObjectiveConfig(
    skew_config, drift_config, explanation_config
)

# Create sampling configuration
random_sampling = model_monitoring.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)

# Create schedule configuration
schedule_config = model_monitoring.ScheduleConfig(monitor_interval=MONITOR_INTERVAL)

# Create alerting configuration.
emails = [USER_EMAIL]
alerting_config = model_monitoring.EmailAlertConfig(
    user_emails=emails, enable_logging=True
)

# Create the monitoring job.
job = aiplatform.ModelDeploymentMonitoringJob.create(
    display_name=JOB_NAME,
    logging_sampling_strategy=random_sampling,
    schedule_config=schedule_config,
    alert_config=alerting_config,
    objective_configs=objective_config,
    project=PROJECT_ID,
    location=REGION,
    endpoint=endpoint,
)

Receiving email alert

After a minute or two, you should receive email at the address you configured above for USER_EMAIL. This email confirms successful deployment of your monitoring job. Here's a sample of what this email might look like:


As your monitoring job collects data, measurements are stored in Cloud Storage and you are free to examine your data at any time. The "Statistics and Anomalies Root Path" specifies the location of your measurements in Cloud Storage. Run the following cell to see an example of the layout of these measurements in Cloud Storage. If you substitute the Cloud Storage URL in your job creation email, you can view the structure and content of the data files for your own monitoring job.

Create the sampling distribution

Next, you send a first test prediction request. The model monitoring service will analyze the distribution of features and automatically create a baseline to monitor deviations from the baseline.

Note: You need to wait for the email notification before making the first prediction request.

Initialize the parsing for automatically generating the input schema

After your Endpoint receives a 1000 prediction requests, the modeling service will automatically parse and create the input schema.

Create the 1000 instance data

In this example, the first 1000 entries in the BigQuery training data are used as the first 1000 prediction requests.

Note: In this context, each instance is a prediction request. In otherwords, sending 1000 prediction requests of a single instance is the same as sending a single prediction request with 1000 instances.

In [ ]:

# Download the table.
table = bigquery.TableReference.from_string(DATASET_BQ_URI[5:])

rows = bqclient.list_rows(table, max_results=1000)

instances = []
for row in rows:
    instance = {}
    for key, value in row.items():
        if key == TARGET:
            continue
        if value is None:
            value = ""
        instance[key] = value
    instances.append(instance)

print(len(instances))

Make the initial prediction request

Next, you send the the 1000 prediction request to your Vertex AI Endpoint resource using the predict() method.

In [ ]:

response = endpoint.predict(instances=instances)

prediction = response[0]

# print the prediction for the first instance
print(prediction[0])

Automatic generation of the input schema

After the model monitoring service receives 1000 instances of prediction requests, the monitoring will start analyzing the prediction requests to automatically generate an input schema for the feature inputs.

Automatic generation of the baseline distribution

After the input schema is generated, the monitoring service creates a batch job to analyze the training data to determine the baseline distribution.

In [ ]:

# Pause a bit for the baseline distribution to be calculated
if os.getenv("IS_TESTING"):
    import time

    time.sleep(120)

Example of monitoring data stored in Cloud Storage

In [ ]:

!gsutil ls gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/*/*

Cloud storage layout

Notice the following components in these Cloud Storage paths:

• cloud-ai-platform-.. - This is a bucket created for you and assigned to capture your service's prediction data. Each monitoring job you create will trigger creation of a new folder in this bucket.
• [model_monitoring|instance_schemas]/job-.. - This is your unique monitoring job number, which you can see above in both the response to your job creation requesst and the email notification.
• instance_schemas/job-../analysis - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).
• instance_schemas/job-../predict - This is the first prediction made to your model after the current monitoring job was enabled.
• model_monitoring/job-../serving - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.
• model_monitoring/job-../training - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data.
• model_monitoring/job-../feature_attribution_score - This folder is used to record data relevant to feature attribution calculations. It contains an ongoing summary of feature attribution scores relative to training data.

You can create monitoring jobs with other user interfaces

In the previous cells, you created a monitoring job using the Python client library. Alternatively, you can use the gcloud command line tool or the Cloud Console to create a model monitoring job.

Interpret your results

Vertex AI Model Monitoring detects an anomaly when the threshold set for a feature is exceeded. The following cells give you a sense of the alerting and reporting experience after model monitoring anomalies have been detected.

Vertex AI Model Monitoring automatically notifies you of detected anomalies through email, but you can also set up alerts through Cloud Logging.

Here's what a sample email alert looks like...

This email is warning you that the cnt_level_start_quickplay, cnt_user_engagement, and country feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the cnt_user_engagement and country feature attribution values are skewed relative to your training data, again, as per your threshold specification.

Monitoring results in the Cloud Console

You can examine your model monitoring data from the Cloud Console. Below is a screenshot of those capabilities.

Monitoring Status

You can verify that a given endpoint has an active model monitoring job via the Endpoint summary page:

Monitoring Alerts

You can examine the alert details by clicking into the endpoint of interest, and selecting the alerts panel:

Feature Value Distributions

You can also examine the recorded training and production feature distributions by drilling down into a given feature, like this:

which yields graphical representations of the feature distrubution during both training and production, like this:

Clean up

When you are finished with this notebook, you can clean up all Google Cloud resources used in this project, by deleting the Google Cloud project you used for the tutorial.

Alternatively, you can preserve the project and delete the individual resources you created in this tutorial by executing the following cell.

In [ ]:

# Undeploy the model and delete the endpoint
endpoint.undeploy_all()
endpoint.delete()

model.delete()

# Delete BQ table and dataset
rmtable = f"bq rm -f model_deployment_monitoring_{ENDPOINT_ID}.serving_predict"
! $rmtable
rmdataset = f"bq rm -f model_deployment_monitoring_{ENDPOINT_ID}"
! $rmdataset

Learn more about model monitoring

Congratulations! You've now learned what model monitoring is, how to configure and enable it, and how to find and interpret the results. Check out the following resources to learn more about model monitoring and ML Ops.

• TensorFlow Data Validation
• Data Understanding, Validation, and Monitoring At Scale
• Vertex Product Documentation
• Vertex AI Model Monitoring Reference Docs
• Vertex AI Model Monitoring blog article
• Explainable AI Whitepaper