model_garden/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb

In [ ]:

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Vertex AI: Track parameters and metrics for custom training jobs

Run in Colab

View on GitHub

Open in Vertex AI Workbench

Overview

This notebook demonstrates how to track metrics and parameters for Vertex AI custom training jobs, and how to perform detailed analysis using this data.

Objective

In this notebook, you learn how to use Vertex AI SDK for Python to:

This tutorial uses the following Google Cloud ML services and resources:

• Vertex AI Dataset
• Vertex AI Model
• Vertex AI Endpoint
• Vertex AI Custom Training Job

The steps performed include:

• Track training parameters and prediction metrics for a custom training job.
• Extract and perform analysis for all parameters and metrics within an Experiment.

Dataset

This example uses the Abalone Dataset. For more information about this dataset please visit: https://archive.ics.uci.edu/ml/datasets/abalone

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage

Learn about Vertex AI pricing and Cloud Storage pricing, and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Installation

Install the packages required for executing this notebook.

In [ ]:

! pip3 install --upgrade tensorflow \
                         google-cloud-aiplatform \
                         scikit-learn -q

Colab only: Uncomment the following cell to restart the kernel.

In [ ]:

# Automatically restart kernel after installs so that your environment can access the new packages
# import IPython

# app = IPython.Application.instance()
# app.kernel.do_shutdown(True)

Before you begin

Set your project ID

If you don't know your project ID, try the following:

• Run gcloud config list.
• Run gcloud projects list.
• See the support page: Locate the project ID

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}

# Set the project id
! gcloud config set project {PROJECT_ID}

Region

You can also change the REGION variable used by Vertex AI. Learn more about Vertex AI regions.

In [ ]:

REGION = "us-central1"  # @param {type: "string"}

Authenticate your Google Cloud account

Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.

1. Vertex AI Workbench

• Do nothing as you are already authenticated.

2. Local JupyterLab instance, uncomment and run:

In [ ]:

# ! gcloud auth login

3. Colab, uncomment and run:

In [ ]:

# from google.colab import auth
# auth.authenticate_user()

4. Service account or other

• See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples.

Create a Cloud Storage bucket

Create a storage bucket to store intermediate artifacts such as datasets.

In [ ]:

BUCKET_URI = "gs://your-bucket-name-unique"  # @param {type:"string"}

Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI

Import libraries and define constants

Import required libraries.

In [ ]:

import os

import pandas as pd
from google.cloud import aiplatform
from sklearn.metrics import mean_absolute_error, mean_squared_error
from tensorflow.python.keras.utils import data_utils

Initialize Vertex AI and set an experiment

Define experiment name.

In [ ]:

EXPERIMENT_NAME = "my-experiment-unique"

Initialize the client for Vertex AI.

In [ ]:

aiplatform.init(
    project=PROJECT_ID,
    location=REGION,
    staging_bucket=BUCKET_URI,
    experiment=EXPERIMENT_NAME,
)

Tracking parameters and metrics in Vertex AI custom training jobs

Download the Dataset to Cloud Storage

In [ ]:

!wget https://storage.googleapis.com/download.tensorflow.org/data/abalone_train.csv
!gsutil cp abalone_train.csv {BUCKET_URI}/data/

gcs_csv_path = f"{BUCKET_URI}/data/abalone_train.csv"

Create a Vertex AI Tabular dataset from CSV data

A Vertex AI dataset can be used to create an AutoML model or a custom model.

In [ ]:

ds = aiplatform.TabularDataset.create(display_name="abalone", gcs_source=[gcs_csv_path])

ds.resource_name

Write the training script

Next, you create the training script that is used in the sample custom training job.

In [ ]:

%%writefile training_script.py

import pandas as pd
import argparse
import os
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

parser = argparse.ArgumentParser()
parser.add_argument('--epochs', dest='epochs',
                    default=10, type=int,
                    help='Number of epochs.')
parser.add_argument('--num_units', dest='num_units',
                    default=64, type=int,
                    help='Number of unit for first layer.')
args = parser.parse_args()

col_names = ["Length", "Diameter", "Height", "Whole weight", "Shucked weight", "Viscera weight", "Shell weight", "Age"]
target = "Age"

def aip_data_to_dataframe(wild_card_path):
    return pd.concat([pd.read_csv(fp.numpy().decode(), names=col_names)
                      for fp in tf.data.Dataset.list_files([wild_card_path])])

def get_features_and_labels(df):
    return df.drop(target, axis=1).values, df[target].values

def data_prep(wild_card_path):
    return get_features_and_labels(aip_data_to_dataframe(wild_card_path))


model = tf.keras.Sequential([layers.Dense(args.num_units), layers.Dense(1)])
model.compile(loss='mse', optimizer='adam')

model.fit(*data_prep(os.environ["AIP_TRAINING_DATA_URI"]),
          epochs=args.epochs ,
          validation_data=data_prep(os.environ["AIP_VALIDATION_DATA_URI"]))
print(model.evaluate(*data_prep(os.environ["AIP_TEST_DATA_URI"])))

# save as Vertex AI Managed model
tf.saved_model.save(model, os.environ["AIP_MODEL_DIR"])

Launch a custom training job and track its trainig parameters on Vertex ML Metadata

In [ ]:

job = aiplatform.CustomTrainingJob(
    display_name="train-abalone-dist-1-replica",
    script_path="training_script.py",
    container_uri="us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-8:latest",
    requirements=["gcsfs==0.7.1"],
    model_serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-8:latest",
)

Start a new experiment run to track training parameters and start the training job. Note that this operation will take around 10 mins.

In [ ]:

aiplatform.start_run(
    "custom-training-run-unique"
)  # Change this to your desired run name
parameters = {"epochs": 10, "num_units": 64}
aiplatform.log_params(parameters)

model = job.run(
    ds,
    replica_count=1,
    model_display_name="abalone-model",
    args=[f"--epochs={parameters['epochs']}", f"--num_units={parameters['num_units']}"],
)

Deploy model and calculate prediction metrics

Next, deploy your Vertex AI Model resource to a Vertex AI Endpoint resource. This operation will take 10-20 mins.

In [ ]:

endpoint = model.deploy(machine_type="n1-standard-4")

Prediction dataset preparation and online prediction

Once model is deployed, perform online prediction using the abalone_test dataset and calculate prediction metrics.

Prepare the prediction dataset.

In [ ]:

def read_data(uri):
    dataset_path = data_utils.get_file("abalone_test.data", uri)
    col_names = [
        "Length",
        "Diameter",
        "Height",
        "Whole weight",
        "Shucked weight",
        "Viscera weight",
        "Shell weight",
        "Age",
    ]
    dataset = pd.read_csv(
        dataset_path,
        names=col_names,
        na_values="?",
        comment="\t",
        sep=",",
        skipinitialspace=True,
    )
    return dataset


def get_features_and_labels(df):
    target = "Age"
    return df.drop(target, axis=1).values, df[target].values


test_dataset, test_labels = get_features_and_labels(
    read_data(
        "https://storage.googleapis.com/download.tensorflow.org/data/abalone_test.csv"
    )
)

Perform online prediction.

In [ ]:

prediction = endpoint.predict(test_dataset.tolist())
prediction

Calculate and track prediction evaluation metrics.

In [ ]:

mse = mean_squared_error(test_labels, prediction.predictions)
mae = mean_absolute_error(test_labels, prediction.predictions)

aiplatform.log_metrics({"mse": mse, "mae": mae})

Extract all parameters and metrics created during this experiment.

In [ ]:

aiplatform.get_experiment_df()

View data in the Cloud Console

Parameters and metrics can also be viewed in the Cloud Console.

In [ ]:

print("Vertex AI Experiments:")
print(
    f"https://console.cloud.google.com/ai/platform/experiments/experiments?folder=&organizationId=&project={PROJECT_ID}"
)

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial: Training Job Model Cloud Storage Bucket

• Vertex AI Dataset
• Training Job
• Model
• Endpoint
• Cloud Storage Bucket

In [ ]:

# Warning: Setting this to true will delete everything in your bucket
delete_bucket = False

# Delete dataset
ds.delete()

# Delete experiment
experiment = aiplatform.Experiment(
    experiment_name=EXPERIMENT_NAME, project=PROJECT_ID, location=REGION
)
experiment.delete()

# Delete the training job
job.delete()

# Undeploy model from endpoint
endpoint.undeploy_all()

# Delete the endpoint
endpoint.delete()

# Delete the model
model.delete()


if delete_bucket or os.getenv("IS_TESTING"):
    ! gsutil -m rm -r $BUCKET_URI