model_garden/notebooks/official/custom/SDK_Custom_Container_Prediction.ipynb

In [ ]:

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Deploying Iris-detection model using FastAPI and Vertex AI custom container serving

Open in Colab

Open in Colab Enterprise

Open in Workbench

View on GitHub

Overview

In this tutorial, you build a scikit-learn model and deploy it on Vertex AI using the custom container method. You use the FastAPI Python web server framework to create a prediction endpoint. You also incorporate a preprocessor from training pipeline into your online serving application.

Learn more about Custom training and Vertex AI Prediction.

Objective

In this notebook, you learn how to create, deploy and serve a custom classification model on Vertex AI. This notebook focuses more on deploying the model than on the design of the model itself.

This tutorial uses the following Vertex AI services and resources:

• Vertex AI models
• Vertex AI endpoints

The steps performed include:

• Train a model that uses flower's measurements as input to predict the class of iris.
• Save the model and its serialized pre-processor.
• Build a FastAPI server to handle predictions and health checks.
• Build a custom container with model artifacts.
• Upload and deploy custom container to Vertex AI Endpoints.

Dataset

This tutorial uses R.A. Fisher's Iris dataset, a small and popular dataset for machine learning experiments. Each instance has four numerical features, which are different measurements of a flower, and a target label that categorizes the flower into: Iris setosa, Iris versicolour and Iris virginica.

This tutorial uses a version of the Iris dataset available in the scikit-learn library.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage
• Artifact Registry
• Cloud Build

Learn about Vertex AI pricing, Cloud Storage pricing, Artifact Registry pricing and Cloud Build pricing and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Get started

Install Vertex AI SDK for Python and other required packages

Write the requirements needed for building container into a file.

In [ ]:

%%writefile requirements.txt
joblib~=1.0
numpy~=1.20
scikit-learn~=0.24
google-cloud-storage>=1.26.0,<2.0.0dev

Install the dependencies.

In [ ]:

# Required in Docker serving container
! pip3 install -U  -r requirements.txt -q

# For local FastAPI development and running
! pip3 install -U  "uvicorn[standard]>=0.12.0,<0.14.0" fastapi~=0.63 -q

# Vertex SDK for Python
! pip3 install --upgrade --quiet  google-cloud-aiplatform

Restart runtime (Colab only)

To use the newly installed packages, you must restart the runtime on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️

Authenticate your notebook environment (Colab only)

Authenticate your environment on Google Colab.

In [ ]:

import sys

if "google.colab" in sys.modules:

    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information

Learn more about setting up a project and a development environment.

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

Create a Cloud Storage bucket

Create a storage bucket to store intermediate artifacts such as datasets.

In [ ]:

BUCKET_URI = f"gs://your-bucket-name-{PROJECT_ID}-unique"  # @param {type:"string"}

If your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}

Initialize Vertex AI SDK for Python

To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API.

In [ ]:

from google.cloud import aiplatform

aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)

Import the required libraries

In [ ]:

import os
import sys

Configure resource names

Set a name for the following parameters:

MODEL_ARTIFACT_DIR - Folder directory path to your model artifacts within a Cloud Storage bucket, for example: "my-models/fraud-detection/trial-4"

REPOSITORY - Name of the Artifact Repository to create or use.

IMAGE - Name of the container image that is pushed to the repository.

MODEL_DISPLAY_NAME - Display name of Vertex AI model resource.

In [ ]:

MODEL_ARTIFACT_DIR = "[your-artifact-directory]"  # @param {type:"string"}
REPOSITORY = "[your-repository-name]"  # @param {type:"string"}
IMAGE = "[your-image-name]"  # @param {type:"string"}
MODEL_DISPLAY_NAME = "[your-model-display-name]"  # @param {type:"string"}

# Set the defaults if no names were specified
if MODEL_ARTIFACT_DIR == "[your-artifact-directory]":
    MODEL_ARTIFACT_DIR = "custom-container-prediction-model"

if REPOSITORY == "[your-repository-name]":
    REPOSITORY = "custom-container-prediction"

if IMAGE == "[your-image-name]":
    IMAGE = "sklearn-fastapi-server"

if MODEL_DISPLAY_NAME == "[your-model-display-name]":
    MODEL_DISPLAY_NAME = "sklearn-custom-container"

Write your pre-processor

Standardize the training data so that each numerical feature column has a mean of 0 and a standard deviation of 1 can improve your model.

Define an app folder and create preprocess.py, which contains a class to perform standardization.

In [ ]:

%mkdir app

In [ ]:

%%writefile app/preprocess.py
import numpy as np

class MySimpleScaler(object):
    def __init__(self):
        self._means = None
        self._stds = None

    def preprocess(self, data):
        if self._means is None:  # during training only
            self._means = np.mean(data, axis=0)

        if self._stds is None:  # during training only
            self._stds = np.std(data, axis=0)
            if not self._stds.all():
                raise ValueError("At least one column has standard deviation of 0.")

        return (data - self._means) / self._stds

Train and store model with pre-processor

Use preprocess.MySimpleScaler to preprocess the iris data, and then train a model using scikit-learn.

After training, export your trained model as a joblib (.joblib) file and export your MySimpleScaler instance as a pickle (.pkl) file.

In [ ]:

%cd app/

import pickle

import joblib
from preprocess import MySimpleScaler
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
scaler = MySimpleScaler()

X = scaler.preprocess(iris.data)
y = iris.target

model = RandomForestClassifier()
model.fit(X, y)

joblib.dump(model, "model.joblib")
with open("preprocessor.pkl", "wb") as f:
    pickle.dump(scaler, f)

Upload model artifacts and custom code to Cloud Storage

Before you can deploy your model for serving, Vertex AI needs access to the following files in Cloud Storage:

• model.joblib (model artifact)
• preprocessor.pkl (model artifact)

Run the following commands to upload your files:

In [ ]:

!gcloud storage cp model.joblib preprocessor.pkl {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/
%cd ..

Build a FastAPI server

To serve predictions from the classification model, build a FastAPI server application.

In [ ]:

%%writefile app/main.py
from fastapi import FastAPI, Request

import joblib
import json
import numpy as np
import pickle
import os

from google.cloud import storage
from preprocess import MySimpleScaler
from sklearn.datasets import load_iris


app = FastAPI()
gcs_client = storage.Client()

with open("preprocessor.pkl", 'wb') as preprocessor_f, open("model.joblib", 'wb') as model_f:
    gcs_client.download_blob_to_file(
        f"{os.environ['AIP_STORAGE_URI']}/preprocessor.pkl", preprocessor_f
    )
    gcs_client.download_blob_to_file(
        f"{os.environ['AIP_STORAGE_URI']}/model.joblib", model_f
    )

with open("preprocessor.pkl", "rb") as f:
    preprocessor = pickle.load(f)

_class_names = load_iris().target_names
_model = joblib.load("model.joblib")
_preprocessor = preprocessor


@app.get(os.environ['AIP_HEALTH_ROUTE'], status_code=200)
def health():
    return {}


@app.post(os.environ['AIP_PREDICT_ROUTE'])
async def predict(request: Request):
    body = await request.json()

    instances = body["instances"]
    inputs = np.asarray(instances)
    preprocessed_inputs = _preprocessor.preprocess(inputs)
    outputs = _model.predict(preprocessed_inputs)

    return {"predictions": [_class_names[class_num] for class_num in outputs]}

Add pre-start script

FastAPI executes the following script before starting up the server. Set the environment variable PORT to AIP_HTTP_PORT for running the FastAPI server.

In [ ]:

%%writefile app/prestart.sh
#!/bin/bash
export PORT=$AIP_HTTP_PORT

Create test instances

To learn more about formatting input instances in JSON, read the documentation.

In [ ]:

%%writefile instances.json
{
    "instances": [
        [6.7, 3.1, 4.7, 1.5],
        [4.6, 3.1, 1.5, 0.2]
    ]
}

Push the container image to Artifact Registry

Write the Dockerfile, using tiangolo/uvicorn-gunicorn-fastapi as a base image. This automatically runs FastAPI for you using Gunicorn and Uvicorn. Visit the FastAPI docs about deploying with Docker to learn more.

In [ ]:

%%writefile Dockerfile

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.9

COPY ./app /app
COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

Test the image locally (optional)

Build the image locally (optional)

Build the image using docker to test it locally.

Note: In this tutorial, docker is only being used to test the container locally. For deployment to Artifact Registry, Cloud-Build is used.

In [ ]:

IS_COLAB = "google.colab" in sys.modules

if not IS_COLAB and not os.getenv("IS_TESTING"):
    ! sudo docker build \
            --tag="{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}" \
            .

Run and test the container locally (optional)

Test running the container locally in detached mode and provide the environment variables that the container requires. These variables are provided to the container by Vertex AI once deployed. Test the /health and /predict routes and then stop the running image.

In [ ]:

if not IS_COLAB and not os.getenv("IS_TESTING"):
    ! sudo docker stop local-iris
    ! sudo docker rm local-iris
    ! docker run -d -p 80:8080 \
            --name=local-iris \
            -e AIP_HTTP_PORT=8080 \
            -e AIP_HEALTH_ROUTE=/health \
            -e AIP_PREDICT_ROUTE=/predict \
            -e AIP_STORAGE_URI={BUCKET_URI}/{MODEL_ARTIFACT_DIR} \
            "{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}"

Ping the health route.

In [ ]:

if not IS_COLAB and not os.getenv("IS_TESTING"):
    ! curl localhost/health

Pass the instances.json and test the predict route.

In [ ]:

if not IS_COLAB and not os.getenv("IS_TESTING"):
    ! curl -X POST \
      -d @instances.json \
      -H "Content-Type: application/json; charset=utf-8" \
      localhost/predict

Stop and delete the container locally.

In [ ]:

if not IS_COLAB and not os.getenv("IS_TESTING"):
    ! sudo docker stop local-iris
    ! sudo docker rm local-iris

Create a repository in Artifact Registry

Enable Artifact Registry API

You must enable the Artifact Registry API service for your project.

Learn more about Enabling service.

In [ ]:

! gcloud services enable artifactregistry.googleapis.com

if os.getenv("IS_TESTING"):
    ! sudo apt-get update --yes && sudo apt-get --only-upgrade --yes install google-cloud-sdk-cloud-run-proxy google-cloud-sdk-harbourbridge google-cloud-sdk-cbt google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-kpt google-cloud-sdk-local-extract google-cloud-sdk-minikube google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-python google-cloud-sdk-spanner-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-nomos google-cloud-sdk-package-go-module google-cloud-sdk-firestore-emulator kubectl google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-kubectl-oidc google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-grpc google-cloud-sdk-pubsub-emulator google-cloud-sdk-datalab google-cloud-sdk-skaffold google-cloud-sdk google-cloud-sdk-terraform-tools google-cloud-sdk-config-connector
    ! gcloud components update --quiet

Create a docker repository in Artifact Registry

Your first step is to create your own Docker repository in Google Artifact Registry.

1 - Run the gcloud artifacts repositories create command to create a new Docker repository with your region and the description as "docker repository".

2 - Run the gcloud artifacts repositories list command to verify that your repository was created.

In [ ]:

REPOSITORY = "my-docker-repo-unique"

! gcloud artifacts repositories create {REPOSITORY} --repository-format=docker --location={LOCATION} --description="Docker repository"

! gcloud artifacts repositories list

Submit the image

Push the image to the created artifact repository using Cloud build.

Note: The following command automatically considers the Dockerfile from the directory it's being run from.

In [ ]:

!gcloud builds submit --region={LOCATION} --tag={LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}

Deploy to Vertex AI

Create Vertex AI model using artifact uri

Use the Python SDK to upload your model artifact in Vertex AI.

In [ ]:

model = aiplatform.Model.upload(
    display_name=MODEL_DISPLAY_NAME,
    artifact_uri=f"{BUCKET_URI}/{MODEL_ARTIFACT_DIR}",
    serving_container_image_uri=f"{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}",
)

Deploy the model to Vertex AI Endpoint

Once the deployment process gets done, the model is deployed and ready for serving online predictions.

In [ ]:

endpoint = model.deploy(machine_type="n1-standard-4")

Request predictions

Send online requests to the endpoint and get predictions.

Using Python SDK

Get predictions from the endpoint for a sample input using python SDK.

In [ ]:

# Send some sample data to the endpoint
endpoint.predict(instances=[[6.7, 3.1, 4.7, 1.5], [4.6, 3.1, 1.5, 0.2]])

Using REST

Get predictions from the endpoint using curl request.

In [ ]:

# Fetch the endpoint name
ENDPOINT_ID = endpoint.name

In [ ]:

# Send a prediction request using sample data 
! curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d @instances.json \
https://{LOCATION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}:predict

Using gcloud CLI

Get predictions from the endpoint using gcloud CLI.

In [ ]:

!gcloud ai endpoints predict $ENDPOINT_ID \
  --region=$LOCATION \
  --json-request=instances.json

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial:

• Model
• Endpoint
• Artifact Registry Image
• Artifact Repository: Set delete_art_repo to True to delete the repository created in this tutorial.
• Cloud Storage bucket: Set delete_bucket to True to delete the Cloud Storage bucket used in this tutorial.

In [ ]:

delete_bucket = False
delete_art_repo = False
    
# Undeploy model and delete endpoint
endpoint.undeploy_all()
endpoint.delete()

#Delete the model resource
model.delete()

# Delete the container image from Artifact Registry
!gcloud artifacts docker images delete \
    --quiet \
    --delete-tags \
    {LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}

# Delete the Artifact Repository
if delete_art_repo:
    ! gcloud artifacts repositories delete {REPOSITORY} --location=$LOCATION -q
    
# Delete the Cloud Storage bucket
if delete_bucket:
    ! gcloud storage rm --recursive $BUCKET_URI

Clean up the locally created files.

In [ ]:

! rm -rf app/
! rm requirements.txt
! rm instances.json
! rm Dockerfile