model_garden/notebooks/official/pipelines/challenger_vs_blessed_deployment_method.ipynb

In [ ]:

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Challenger vs Blessed methodology for model deployment into production

Open in Colab

Open in Colab Enterprise

Open in Workbench

View on GitHub

Overview

This tutorial shows how to use Vertex AI Pipelines for deploying the next version of a model into production using the challenger vs blessed method.

Learn more about Vertex AI Pipelines and Model evaluation in Vertex AI.

NOTE: This notebook uses KFP 1.x and GCPC 1.x. It's recommended to use 2.x versions.

NOTE: This notebook has been tested in the following environment:

• Python version = 3.9

Objective

In this tutorial, you learn how to construct a Vertex AI pipeline, which trains a new challenger version of a model, evaluates the model and compares the evaluation to the existing blessed model in production. Then, it determines whether the challenger model becomes the blessed model for replacement in production.

This tutorial uses the following Vertex AI services and resources:

• Vertex AI Pipeline
• Vertex AI Model Evaluation
• Vertex AI Model Registry
• Vertex AI Endpoints

The steps performed include:

• Import a pretrained (blessed) model to the Vertex AI Model Registry.
• Import synthetic model evaluation metrics to the corresponding (blessed) model.
• Create a Vertex AI endpoint resource
• Deploy the blessed model to the endpoint resource.
• Create a Vertex AI Pipeline that runs the following steps:
  • Get the blessed model.
  • Import another instance (challenger) of the pretrained model.
  • Register the pretrained (challenger) model as a new version of the existing blessed model.
  • Create a synthetic model evaluation.
  • Import the synthetic model evaluation metrics to the corresponding challenger model.
  • Compare the evaluations and set the blessed or challenger model as the default.
  • Deploy the new blessed model.

Learn more about Vertex AI Pipelines.

Dataset

The dataset used for this tutorial is the CIFAR10 dataset from TensorFlow Datasets. The version of the dataset you use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage

Learn about Vertex AI pricing, and Cloud Storage pricing, and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Get started

Install Vertex AI SDK for Python and other required packages

In [ ]:

# Install the packages
! pip3 install --upgrade google-cloud-aiplatform \
                                'google-cloud-pipeline-components<2'
! pip3 install tensorflow==2.5 \
                tensorflow_hub
  
! pip3 install --upgrade 'kfp<2'

Restart runtime (Colab only)

To use the newly installed packages, you must restart the runtime on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️

Authenticate your notebook environment (Colab only)

Authenticate your environment on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information

Learn more about setting up a project and a development environment.

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

Create a Cloud Storage bucket

Create a storage bucket to store intermediate artifacts such as datasets.

In [ ]:

BUCKET_URI = f"gs://your-bucket-name-{PROJECT_ID}-unique"  # @param {type:"string"}

If your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gcloud storage buckets create --location=$LOCATION --project=$PROJECT_ID $BUCKET_URI

Service Account

You use a service account to create Vertex AI Pipeline jobs.

If you don't want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID.

In [ ]:

SERVICE_ACCOUNT = "[your-service-account]"  # @param {type:"string"}

In [ ]:

import sys

IS_COLAB = "google.colab" in sys.modules

if (
    SERVICE_ACCOUNT == ""
    or SERVICE_ACCOUNT is None
    or SERVICE_ACCOUNT == "[your-service-account]"
):
    # Get your service account from gcloud
    if not IS_COLAB:
        shell_output = !gcloud auth list 2>/dev/null
        SERVICE_ACCOUNT = shell_output[2].replace("*", "").strip()

    else:  # IS_COLAB:
        shell_output = ! gcloud projects describe  $PROJECT_ID
        project_number = shell_output[-1].split(":")[1].strip().replace("'", "")
        SERVICE_ACCOUNT = f"{project_number}-compute@developer.gserviceaccount.com"

    print("Service Account:", SERVICE_ACCOUNT)

Set service account access for Vertex AI Pipelines

Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run these once per service account.

In [ ]:

! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectCreator

! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer

Import libraries

In [ ]:

import kfp
import tensorflow as tf
import tensorflow_hub as hub
from google.cloud import aiplatform
from google.cloud.aiplatform import gapic
from kfp.v2 import compiler
from kfp.v2.dsl import component

Initialize Vertex AI SDK for Python

To get started using Vertex AI, you must enable the Vertex AI API in your Google Cloud project.

Then, initialize the Vertex AI SDK for Python for your project.

In [ ]:

aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)

Set hardware accelerators

You can set hardware accelerators for training and prediction.

Set the variables DEPLOY_GPU/DEPLOY_NGPU to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa T4 GPUs allocated to each VM, you would specify:

(aip.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)

Otherwise specify (None, None) to use a container image to run on a CPU.

Learn more about hardware accelerator support for your region.

In [ ]:

DEPLOY_GPU, DEPLOY_NGPU = (None, None)

Set prebuilt containers

Set the prebuilt Docker container image for prediction.

For the latest list, see Prebuilt containers for prediction.

In [ ]:

TF = "2.5".replace(".", "-")

if DEPLOY_GPU:
    DEPLOY_VERSION = "tf2-gpu.{}".format(TF)
else:
    DEPLOY_VERSION = "tf2-cpu.{}".format(TF)

DEPLOY_IMAGE = "{}-docker.pkg.dev/vertex-ai/prediction/{}:latest".format(
    LOCATION.split("-")[0], DEPLOY_VERSION
)

print("Deployment:", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)

Set machine type

Next, set the machine type to use for deployment.

• Set the variable DEPLOY_COMPUTE to configure the compute resources for the VMs you use for deployment.
• machine_type
  • n1-standard: 3.75GB of memory per vCPU
  • n1-highmem: 6.5GB of memory per vCPU
  • n1-highcpu: 0.9 GB of memory per vCPU
• vCPUs: number of [2, 4, 8, 16, 32, 64, 96 ]

Note: The following aren't supported.

• standard: 2 vCPUs
• highcpu: 2, 4 and 8 vCPUs

Note: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs.

In [ ]:

machine_type = "n1-standard"
vCPUs = "4"
DEPLOY_COMPUTE = f"{machine_type}-{vCPUs}"
print("Deploy machine type", DEPLOY_COMPUTE)

Get pretrained model from TensorFlow Hub

For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a Vertex AI model resource. Once you have a Vertex AI model resource, the model can be deployed to a Vertex AI endpoint resource.

Download the pretrained model

First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a Sequential() model with the downloaded TFHub model as a layer, and specify the input shape to the model.

In [ ]:

tfhub_model = tf.keras.Sequential(
    [hub.KerasLayer("https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5")]
)

tfhub_model.build([None, 32, 32, 3])

tfhub_model.summary()

Save the model artifacts

At this point, the model is in memory. Next, you save the model artifacts to a Cloud Storage location.

In [ ]:

MODEL_DIR = BUCKET_URI + "/model"
tfhub_model.save(MODEL_DIR)

Upload the TensorFlow Hub model to Vertex AI Model Registry

Finally, you upload the model artifacts from the TFHub model into Vertex AI Model Registry and get a model resource object using the upload() method, with the following parameters:

• display_name: A human readable name for the model resource.
• artifact_uri: The Cloud Storage location of the model package.
• serving_container_image_uri: The serving container image.

Uploading a model into Vertex AI Model Registry returns a long running operation, since it may take a few moments.

Note: When you upload the model artifacts to Vertex AI Model Registry, you specify the corresponding deployment container image.

In [ ]:

blessed_model = aiplatform.Model.upload(
    display_name="resnet",
    artifact_uri=MODEL_DIR,
    serving_container_image_uri=DEPLOY_IMAGE,
    is_default_version=True,
    version_aliases=["v1"],
)

print(blessed_model)

Create a model evaluation

First, you create a model evaluation in a format that corresponds to one of the predefined schemas for model evaluations. In this example, you use the schema for a classification metric, and specify the following subset of evaluation metrics as a dictionary:

• logLoss: The log loss.
• auPrc: The accuracy.

You then construct the ModelEvaluation object with the following parameters:

• display_name: The human readable name for the evaluation metric.
• metrics_schema_uri: The schema for the specific type of evaluation metrics.
• metrics: The dictionary with the evaluation metrics.

Learn more about Schemas for evaluation metrics.

In [ ]:

metrics = {"logLoss": 1.4, "auPrc": 0.85}
print(metrics)

blessed_eval = gapic.ModelEvaluation(
    display_name="eval",
    metrics_schema_uri="gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
    metrics=metrics,
)

Upload the evaluation metrics to the Model Registry

Next, upload the model's evaluation from the custom training job to the corresponding entry in the Vertex AI Model Registry.

Currently, the Vertex AI Python SDK has no support for this method. Instead, you use the lower level GAPIC API interface.

In [ ]:

API_ENDPOINT = f"{LOCATION}-aiplatform.googleapis.com"
client = gapic.ModelServiceClient(client_options={"api_endpoint": API_ENDPOINT})

client.import_model_evaluation(
    parent=blessed_model.resource_name, model_evaluation=blessed_eval
)

Creating an endpoint resource

You create an endpoint resource using the Endpoint.create() method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location (region); otherwise the settings are inherited by the values you set when you initialized the Vertex AI Python SDK with the init() method.

In this example, the following parameters are specified:

• display_name: A human readable name for the endpoint resource.
• project: Your project ID.
• location: Your region.

This method returns an Endpoint object.

Learn more about Vertex AI Endpoints.

In [ ]:

endpoint = aiplatform.Endpoint.create(
    display_name="resnet", project=PROJECT_ID, location=LOCATION
)

print(endpoint)

Deploy the model resource to the endpoint resource

Next, you deploy the blessed Vertex AI model resource to a Vertex AI endpoint resource. The container image defined for the Vertex AI model resource is used for deployment. To deploy, you specify the following additional configuration settings:

• The machine type.
• The (if any) type and number of GPUs.
• Static, manual or auto-scaling of VM instances.

In this example, you deploy the model with the minimal amount of specified parameters, as follows:

• model: The model resource.
• deployed_model_displayed_name: The human readable name for the deployed model instance.
• machine_type: The machine type for each VM instance.

This may take upto a few minutes due to provisioning of the configured resources.

In [ ]:

response = endpoint.deploy(
    model=blessed_model,
    deployed_model_display_name="resnet",
    machine_type=DEPLOY_COMPUTE,
)

print(response)

Create custom components for pipeline

Next, you create several custom components that you use in your pipeline.

Create component to upload the next version of the model

First, you define a component to upload the trained challenger model as a version to the blessed model in the Vertex AI Model Registry. The component takes the following arguments:

• parent_model: The full resource name of the blessed model.
• artifact_uri: The Cloud Storage location of the model artifacts for the challenger model.
• serving_container: The serving container for the challenger model.
• project: Your Project ID.
• region: The region where you want to create or use the resources.

In [ ]:

@component(packages_to_install=["google-cloud-aiplatform"])
def create_next_model_version(
    parent_model: str,
    artifact_uri: str,
    serving_container: str,
    project: str,
    region: str,
) -> str:
    from google.cloud import aiplatform

    aiplatform.init(project=project, location=region)

    model = aiplatform.Model.upload(
        display_name="resnet",
        artifact_uri=artifact_uri,
        serving_container_image_uri=serving_container,
        parent_model=parent_model,
        is_default_version=True,
        version_aliases=["v2"],
        version_description="This is the second version of the model",
    )

    return model.resource_name

Create component to import classification metrics

Next, you define a component to import the evaluation metrics for the challenger model to the Model Registry. The component takes the following arguments:

• display_name: Human readable name for the evaluation metrics.
• metrics: The evaluation metrics formatted for classification.
• parent_model_resource: The full resource name for the challenger model version.
• region: The region where you want to create or use the resources.

In [ ]:

@component(packages_to_install=["google-cloud-aiplatform"])
def import_classification_metrics(
    display_name: str, metrics: dict, parent_model_resource: str, region: str
):
    from google.cloud.aiplatform import gapic

    evaluation = gapic.ModelEvaluation(
        display_name=display_name,
        metrics_schema_uri="gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml",
        metrics=metrics,
    )

    API_ENDPOINT = f"{region}-aiplatform.googleapis.com"
    client = gapic.ModelServiceClient(client_options={"api_endpoint": API_ENDPOINT})
    client.import_model_evaluation(
        parent=parent_model_resource, model_evaluation=evaluation
    )

Create component to compare metrics

Next, you define a component to compare the auPrc metric between two versions of a model. In this case, you compare between the current blessed model and the new challenger model. Whichever of the two has the best auPrc value is set as the default model. When you subsequently deploy, the default model is deployed. The component takes the following arguments:

• blessed_model_resource_name: The full resource name of the blessed model.
• challenger_model_resource_name: The full resource name of the challenger model.

In [ ]:

@component(packages_to_install=["google-cloud-aiplatform"])
def compare_metrics(
    blessed_model_resource_name: str, challenger_model_resource_name: str
):
    from google.cloud import aiplatform

    # Get the metrics for the blessed model
    blessed_model = aiplatform.Model(blessed_model_resource_name)
    blessed_eval = blessed_model.list_model_evaluations()[0]
    blessed_auPrc = blessed_eval.metrics["auPrc"]

    # Get the metrics for the challenger model
    challenger_model = aiplatform.Model(challenger_model_resource_name)
    challenger_eval = challenger_model.list_model_evaluations()[0]
    challenger_auPrc = challenger_eval.metrics["auPrc"]

    # Which model has the best accuracy becomes the default model
    if challenger_auPrc > blessed_auPrc:
        challenger_model.versioning_registry.add_version_aliases(
            new_aliases=["default"], version=challenger_model.version_id
        )
    else:
        blessed_model.versioning_registry.add_version_aliases(
            new_aliases=["default"], version=blessed_model.version_id
        )

Blessed vs challenger pipeline

In this section, you construct a pipeline to fetch the blessed, and challenger models and compare them to deploy the best one to the production endpoint.

Define the pipeline

Your pipeline runs the following tasks:

• Get the blessed version of a model.
• Get the endpoint for the deployed blessed model.
• Train (faked) the challenger model.
• Upload the challenger model as the next version of the model.
• Import the challenger model's evaluation metrics.
• Compare the blessed and challenger evaluation metrics, and set the default accordingly.
• Import the existing production endpoint.
• Deploy the new default model to the production endpoint.

In [ ]:

@kfp.dsl.pipeline(name="blessed-vs-challenger")
def pipeline(
    blessed_model_resource: str,
    serving_container: str,
    machine_type: str,
    endpoint_resource_name: str,
    endpoint_resource_uri: str,
    project: str = PROJECT_ID,
    region: str = LOCATION,
):
    from google_cloud_pipeline_components.experimental.evaluation import \
        GetVertexModelOp
    from google_cloud_pipeline_components.types import artifact_types
    from google_cloud_pipeline_components.v1.endpoint import ModelDeployOp
    from kfp.v2.components import importer_node

    # Get the Vertex AI model resource of the blessed model
    model = GetVertexModelOp(model_resource_name=blessed_model_resource)

    # pretend that you trained a new version of the model (artifacts at MODEL_DIR)

    # create the next version in the Model Registry as the challenger model
    next_version = create_next_model_version(
        parent_model=blessed_model_resource,
        artifact_uri=MODEL_DIR,
        serving_container=serving_container,
        project=project,
        region=region,
    ).after(model)

    # pretend to evaluate the challenger
    challenger_metrics = {"logLoss": 1.3, "auPrc": 0.88}

    # upload the metrics for the challenger version
    import_metrics = import_classification_metrics(
        display_name="challenger",
        metrics=challenger_metrics,
        parent_model_resource=next_version.output,
        region=region,
    ).after(next_version)

    # test metrics
    compare = compare_metrics(blessed_model_resource, next_version.output).after(
        import_metrics
    )

    # import the production Endpoint
    endpoint = importer_node.importer(
        artifact_uri=endpoint_resource_uri,
        artifact_class=artifact_types.VertexEndpoint,
        metadata={"resourceName": endpoint_resource_name},
    )

    # deploy model to endpoint
    _ = ModelDeployOp(
        model=model.outputs["model"],
        endpoint=endpoint.output,  # .outputs["endpoint"],
        dedicated_resources_min_replica_count=1,
        dedicated_resources_max_replica_count=1,
        dedicated_resources_machine_type=machine_type,
        traffic_split={"0": 100},
    ).after(compare)

Compile the pipeline

Next, you compile the pipeline to a JSON file.

In [ ]:

# Compile the pipeline to a json file
compiler.Compiler().compile(
    pipeline_func=pipeline, package_path="challenger_vs_blessed.json"
)

Execute the pipeline

Finally, you execute your pipeline, passing the following pipeline parameter values:

• blessed_model_resource: The full resource name of the current blessed version of the model.
• serving_container: The serving container for deploying the model.
• machine_type: The machine type for deploying the model.
• endpoint_resource_name: The full resource name of the production endpoint.
• endpoint_resource_uri: The full URI for the production endpoint.
• project:The project ID.
• region: The region where you want to create or use the resources.

In [ ]:

# Define the root folder for your pipeline artifacts
PIPELINE_ROOT = "{}/pipeline_root/control".format(BUCKET_URI)

# Define the pipeline job
job = aiplatform.PipelineJob(
    display_name="challenger_vs_blessed",
    template_path="challenger_vs_blessed.json",
    pipeline_root=PIPELINE_ROOT,
    parameter_values={
        "blessed_model_resource": blessed_model.resource_name,
        "serving_container": DEPLOY_IMAGE,
        "machine_type": DEPLOY_COMPUTE,
        "endpoint_resource_name": endpoint.resource_name,
        "endpoint_resource_uri": "https://us-central1-aiplatform.googleapis.com/v1/"
        + endpoint.resource_name,
        "project": PROJECT_ID,
        "region": LOCATION,
    },
    enable_caching=False,
)

# Run the pipeline job
job.run()

Get the latest state of the production endpoint

Once your pipeline execution is finished, notice that the challenger (version 2) model has replaced the previous blessed model on the production endpoint.

Now, display the latest details of the deployed models from the production endpoint, and then display the traffic split. Notice the resource ID for the 100% entry is the resource ID for the challenger (v2) model.

In [ ]:

gca_resource = endpoint.list(filter="display_name=resnet")[0].gca_resource
print("Deployed Models", gca_resource.deployed_models)
print("\n")
print("Traffic Split", gca_resource.traffic_split)

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial.

In [ ]:

# Undeploy the model from endpoint
endpoint.undeploy_all()

# Delete the endpoint resource
endpoint.delete()

# Delete the model resource(s)
blessed_model.delete()

# Delete the pipeline job
job.delete()

# Delete Cloud Storage objects that were created
delete_bucket = True
if delete_bucket:
    ! gcloud storage rm --recursive $BUCKET_URI

# Delete the locally saved pipeline package file
! rm challenger_vs_blessed.json