model_garden/notebooks/official/prediction/get_started_with_raw_predict.ipynb

In [ ]:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Get started with TensorFlow serving functions with Vertex AI Raw Prediction

Open in Colab

Open in Colab Enterprise

Open in Workbench

View on GitHub

Overview

This tutorial demonstrates how to use Vertex AI Raw Prediction to send raw HTTP content directly to a model deployed to a Vertex AI Endpoint.

For example, the HTTP server for pre-built Vertex AI deployment containers does not support the HTTP request body for TensorFlow 1.x estimators. Using raw predict, one can send raw content through the HTTP server that is presented to the model input as-is -- no canonical processing.

Learn more about Raw Predict.

Objective

In this tutorial, you learn how to use Vertex AI Raw Prediction on a Vertex AI Endpoint resource.

This tutorial uses the following Google Cloud ML services and resources:

• Vertex AI Raw Prediction
• Vertex AI Models
• Vertex AI Endpoints

The steps performed include:

• Download pretrained tabular classification model artifacts for a TensorFlow 1.x estimator.
• Upload the TensorFlow estimator model as a Vertex AI Model resource.
• Create an Endpoint resource.
• Deploy the Model resource to an Endpoint resource.
• Make an online raw prediction to the Model resource instance deployed to the Endpoint resource.

Dataset

This tutorial uses a pre-trained tabular classification model from a public Cloud Storage bucket, which is trained on the Penguins dataset (https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage

Learn about Vertex AI pricing and Cloud Storage pricing, and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Get started

Install Vertex AI SDK for Python and other required packages

In [ ]:

! pip3 install --upgrade --quiet google-cloud-aiplatform 
! pip3 install tensorflow-hub

Restart runtime (Colab only)

To use the newly installed packages, you must restart the runtime on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️

Authenticate your notebook environment (Colab only)

Authenticate your environment on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information and initialize Vertex AI SDK for Python

To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API. Learn more about setting up a project and a development environment.

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

Create a Cloud Storage bucket

Create a storage bucket to store intermediate artifacts such as datasets.

In [ ]:

BUCKET_URI = f"gs://your-bucket-name-{PROJECT_ID}-unique"  # @param {type:"string"}

If your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}

Set Google Cloud project

In [ ]:

from google.cloud import aiplatform

aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)

Set hardware accelerators

You can set hardware accelerators for training and prediction.

Set the variables DEPLOY_GPU/DEPLOY_NGPU to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Tesla K80 GPUs allocated to each VM, you would specify:

(aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)

Otherwise specify (None, None) to use a container image to run on a CPU.

Learn more about hardware accelerator support for your region.

In [ ]:

DEPLOY_GPU, DEPLOY_NGPU = (None, None)

Set pre-built containers

Set the pre-built Docker container image for prediction.

For the latest list, see Pre-built containers for prediction.

In [ ]:

TF = "2-5"

if DEPLOY_GPU:
    DEPLOY_VERSION = "tf2-gpu.{}".format(TF)
else:
    DEPLOY_VERSION = "tf2-cpu.{}".format(TF)


DEPLOY_IMAGE = "{}-docker.pkg.dev/vertex-ai/prediction/{}:latest".format(
    LOCATION.split("-")[0], DEPLOY_VERSION
)

print("Deployment:", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)

Set machine type

Next, set the machine type to use for prediction.

• Set the variable DEPLOY_COMPUTE to configure the compute resources for the VMs you will use for prediction.
• machine type
  • n1-standard: 3.75GB of memory per vCPU.
  • n1-highmem: 6.5GB of memory per vCPU
  • n1-highcpu: 0.9 GB of memory per vCPU
• vCPUs: number of [2, 4, 8, 16, 32, 64, 96 ]

Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs.

In [ ]:

MACHINE_TYPE = "n1-standard"

VCPU = "4"
DEPLOY_COMPUTE = MACHINE_TYPE + "-" + VCPU
print("Train machine type", DEPLOY_COMPUTE)

Get pretrained model from the public Cloud Storage location

For demonstration purposes, this tutorial uses a pretrained model TensorFlow 1.x estimator tabular classification Model, which is then uploaded to a Vertex AI Model resource. Once you have a Vertex AI Model resource, the model can be deployed to a Vertex AI Endpoint resource.

Download the pretrained model

Download the pretrained TensorFlow estimator model artifacts from the public Cloud Storage, and then upload the model artifacts to your own Cloud Storage bucket.

In [ ]:

MODEL_DIR = BUCKET_URI + "/model"

! gsutil cp -r gs://cloud-samples-data/vertex-ai/google-cloud-aiplatform-ci-artifacts/models/penguins/estimator/ {MODEL_DIR}

Upload the TensorFlow estimator model to a Vertex AI Model resource

Finally, you upload the model artifacts from the TFHub model and serving function into a Vertex AI Model resource.

Note: When you upload the model artifacts to a Vertex AI Model resource, you specify the corresponding deployment container image.

In [ ]:

model = aiplatform.Model.upload(
    display_name="example",
    artifact_uri=MODEL_DIR,
    serving_container_image_uri=DEPLOY_IMAGE,
)

print(model)

Creating an Endpoint resource

You create an Endpoint resource using the Endpoint.create() method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location; otherwise the settings are inherited by the values you set when you initialized the Vertex AI SDK with the init() method.

In this example, the following parameters are specified:

• display_name: A human readable name for the Endpoint resource.
• project: Your project ID.
• location: Your location.
• labels: (optional) User defined metadata for the Endpoint in the form of key/value pairs.

This method returns an Endpoint object.

Learn more about Vertex AI Endpoints.

In [ ]:

endpoint = aiplatform.Endpoint.create(
    display_name="example",
    project=PROJECT_ID,
    location=LOCATION,
    labels={"your_key": "your_value"},
)

print(endpoint)

Deploying Model resources to an Endpoint resource.

You can deploy one of more Vertex AI Model resource instances to the same endpoint. Each Vertex AI Model resource that is deployed will have its own deployment container for the serving binary.

In the next example, you deploy the Vertex AI Model resource to a Vertex AI Endpoint resource. The Vertex AI Model resource already has defined the deployment container image. To deploy, you specify the following additional configuration settings:

• The machine type.
• The (if any) type and number of GPUs.
• Static, manual or auto-scaling of VM instances.

In this example, you deploy the model with the minimal amount of specified parameters, as follows:

• model: The Model resource.
• deployed_model_displayed_name: The human readable name for the deployed model instance.
• machine_type: The machine type for each VM instance.

Do to the requirements to provision the resource, this may take upto a few minutes.

In [ ]:

response = endpoint.deploy(
    model=model,
    deployed_model_display_name="example",
    machine_type=DEPLOY_COMPUTE,
)

print(endpoint)

Make prediction instances

Next, you prepare a prediction request using a synthetic example. In this example, the model format is a TensorFlow 1.x estimator format. This model format takes a request signature not supported by the HTTP server in the Vertex AI prebuilt TensorFlow serving containers.

For this model format, you use the raw_predict() to pass as-is a request that matches directly the serving interface of the model, with the following request format:

http_body -> {
    'signature_name' : serving_signature,
    'instances': [ {instance_1}, {instance_2}, ... ]
}

instance -> { 'feature_1': value_1, 'feature_2': value_2, ... }

serving_signature -> "predict"

In [ ]:

import json

from google.api import httpbody_pb2
from google.cloud import aiplatform_v1

DATA = {
    "signature_name": "predict",
    "instances": [
        {
            "island": "DREAM",
            "culmen_length_mm": 36.6,
            "culmen_depth_mm": 18.4,
            "flipper_length_mm": 184.0,
            "body_mass_g": 3475.0,
            "sex": "FEMALE",
        }
    ],
}

http_body = httpbody_pb2.HttpBody(
    data=json.dumps(DATA).encode("utf-8"),
    content_type="application/json",
)

req = aiplatform_v1.RawPredictRequest(
    http_body=http_body, endpoint=endpoint.resource_name
)

Make a prediction

Finally, you make the prediction request using Vertex AI Prediction service.

In [ ]:

API_ENDPOINT = "{}-aiplatform.googleapis.com".format(LOCATION)
client_options = {"api_endpoint": API_ENDPOINT}

pred_client = aiplatform.gapic.PredictionServiceClient(client_options=client_options)

response = pred_client.raw_predict(req)
print(response)

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial:

In [ ]:

delete_bucket = False
delete_model = True
delete_endpoint = True

if delete_endpoint:
    try:
        endpoint.undeploy_all()
        endpoint.delete()
    except Exception as e:
        print(e)

if delete_model:
    try:
        model.delete()
    except Exception as e:
        print(e)

if delete_bucket:
    ! gsutil rm -rf {BUCKET_URI}