model_garden/notebooks/community/hyperparameter_tuning/distributed-hyperparameter-tuning.ipynb

In [ ]:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Vertex Training: Distributed Hyperparameter Tuning

Run in Colab

View on GitHub

Open in Vertex AI Workbench

Overview

This notebook demonstrates how to run a hyperparameter tuning job with Vertex Training to discover optimal hyperparameter values for an ML model. To speed up the training process, MirroredStrategy from the tf.distribute module is used to distribute training across multiple GPUs on a single machine.

Dataset

The dataset used for this tutorial is the horses or humans dataset from TensorFlow Datasets. The trained model predicts if an image is of a horse or a human.

Objective

In this notebook, you create a custom-trained model from a Python script in a Docker container. You learn how to modify training application code for hyperparameter tuning and submit a Vertex Training hyperparameter tuning job with the Python SDK.

The steps performed include:

• Create a Vertex AI custom job for training a model.
• Launch hyperparameter tuning job with the Python SDK.
• Cleanup resources.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage

Learn about Vertex AI pricing and Cloud Storage pricing, and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Set up your local development environment

If you are using Colab or Google Cloud Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.

Otherwise, make sure your environment meets this notebook's requirements. You need the following:

• The Google Cloud SDK
• Git
• Python 3
• virtualenv
• Jupyter notebook running in a virtual environment with Python 3

The Google Cloud guide to Setting up a Python development environment and the Jupyter installation guide provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:

1. Install and initialize the Cloud SDK.

2. Install Python 3.

3. Install virtualenv and create a virtual environment that uses Python 3. Activate the virtual environment.

4. To install Jupyter, run pip3 install jupyter on the command-line in a terminal shell.

5. To launch Jupyter, run jupyter notebook on the command-line in a terminal shell.

6. Open this notebook in the Jupyter Notebook Dashboard.

Install additional packages

Install the latest version of Vertex SDK for Python.

In [ ]:

import os

# The Google Cloud Notebook product has specific requirements
IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists("/opt/deeplearning/metadata/env_version")

# Google Cloud Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_GOOGLE_CLOUD_NOTEBOOK:
    USER_FLAG = "--user"

In [ ]:

! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform

Restart the kernel

After you install the additional packages, you need to restart the notebook kernel so it can find the packages.

In [ ]:

# Automatically restart kernel after installs
import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

Before you begin

Select a GPU runtime

Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select "Runtime --> Change runtime type > GPU"

Set up your Google Cloud project

The following steps are required, regardless of your notebook environment.

1. Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.

2. Make sure that billing is enabled for your project.

3. Enable the Vertex AI API and Compute Engine API.

4. If you are running this notebook locally, you will need to install the Cloud SDK.

5. Enter your project ID in the cell below. Then run the cell to make sure the Cloud SDK uses the right project for all the commands in this notebook.

Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.

Set your project ID

If you don't know your project ID, you may be able to get your project ID using gcloud.

In [ ]:

import os

PROJECT_ID = ""

# Get your Google Cloud project ID from gcloud
if not os.getenv("IS_TESTING"):
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID: ", PROJECT_ID)

Otherwise, set your project ID here.

In [ ]:

if PROJECT_ID == "" or PROJECT_ID is None:
    PROJECT_ID = ""  # @param {type:"string"}

Set project ID

In [ ]:

! gcloud config set project $PROJECT_ID

Timestamp

If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial.

In [ ]:

from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")

Authenticate your Google Cloud account

If you are using Google Cloud Notebooks, your environment is already authenticated. Skip this step.

If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.

Otherwise, follow these steps:

1. In the Cloud Console, go to the Create service account key page.

2. Click Create service account.

3. In the Service account name field, enter a name, and click Create.

4. In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex AI" into the filter box, and select Vertex AI Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.

5. Click Create. A JSON file that contains your key downloads to your local environment.

6. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.

In [ ]:

import os
import sys

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

# The Google Cloud Notebook product has specific requirements
IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists("/opt/deeplearning/metadata/env_version")

# If on Google Cloud Notebooks, then don't execute this code
if not IS_GOOGLE_CLOUD_NOTEBOOK:
    if "google.colab" in sys.modules:
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''

Create a Cloud Storage bucket

The following steps are required, regardless of your notebook environment.

When you submit a custom training job using the Cloud SDK, you will need to provide a staging bucket.

Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets.

You may also change the REGION variable, which is used for operations throughout the rest of this notebook. Make sure to choose a region where Vertex AI services are available. You may not use a Multi-Regional Storage bucket for training with Vertex AI.

In [ ]:

BUCKET_URI = "gs://[your-bucket-name]"  # @param {type:"string"}
REGION = "us-central1"  # @param {type:"string"}

In [ ]:

if BUCKET_URI == "" or BUCKET_URI is None or BUCKET_URI == "gs://[your-bucket-name]":
    BUCKET_URI = "gs://" + PROJECT_ID + "aip-" + TIMESTAMP

In [ ]:

print(BUCKET_URI)

Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gcloud storage buckets create --location=$REGION $BUCKET_URI

Finally, validate access to your Cloud Storage bucket by examining its contents:

In [ ]:

! gcloud storage ls --all-versions --long $BUCKET_URI

Import libraries and define constants

In [ ]:

import os
import sys

from google.cloud import aiplatform
from google.cloud.aiplatform import hyperparameter_tuning as hpt

Write Dockerfile

The first step in containerizing your code is to create a Dockerfile. In the Dockerfile, you'll include all the commands needed to run the image such as installing the necessary libraries and setting up the entry point for the training code.

This Dockerfile uses the Deep Learning Container TensorFlow Enterprise 2.5 GPU Docker image. The Deep Learning Containers on Google Cloud come with many common ML and data science frameworks pre-installed. After downloading that image, this Dockerfile installs the CloudML Hypertune library and sets up the entrypoint for the training code.

In [ ]:

%%writefile Dockerfile

FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5
WORKDIR /

# Installs hypertune library
RUN pip install cloudml-hypertune

# Copies the trainer code to the docker image.
COPY trainer /trainer

# Sets up the entry point to invoke the trainer.
ENTRYPOINT ["python", "-m", "trainer.task"]

Create training application code

Next, you create a trainer directory with a task.py script that contains the code for your training application.

In [ ]:

# Create trainer directory

! mkdir trainer

In the next cell, you write the contents of the training script, task.py. This file downloads the horses or humans dataset from TensorFlow datasets and trains a tf.keras functional model using MirroredStrategy from the tf.distribute module.

There are a few components that are specific to using the hyperparameter tuning service:

• The script imports the hypertune library. Note that the Dockerfile included instructions to pip install the hypertune library.
• The function get_args() defines a command-line argument for each hyperparameter you want to tune. In this example, the hyperparameters that will be tuned are the learning rate, the momentum value in the optimizer, and the number of units in the last hidden layer of the model. The value passed in those arguments is then used to set the corresponding hyperparameter in the code.
• At the end of the main() function, the hypertune library is used to define the metric to optimize. In this example, the metric that will be optimized is the the validation accuracy. This metric is passed to an instance of HyperTune.

In [ ]:

%%writefile trainer/task.py

import argparse
import hypertune
import tensorflow as tf
import tensorflow_datasets as tfds

def get_args():
  """Parses args. Must include all hyperparameters you want to tune."""

  parser = argparse.ArgumentParser()
  parser.add_argument(
      '--learning_rate', required=True, type=float, help='learning rate')
  parser.add_argument(
      '--momentum', required=True, type=float, help='SGD momentum value')
  parser.add_argument(
      '--units',
      required=True,
      type=int,
      help='number of units in last hidden layer')
  parser.add_argument(
      '--epochs',
      required=False,
      type=int,
      default=10,
      help='number of training epochs')
  args = parser.parse_args()
  return args


def preprocess_data(image, label):
  """Resizes and scales images."""

  image = tf.image.resize(image, (150, 150))
  return tf.cast(image, tf.float32) / 255., label


def create_dataset(batch_size):
  """Loads Horses Or Humans dataset and preprocesses data."""

  data, info = tfds.load(
      name='horses_or_humans', as_supervised=True, with_info=True)

  # Create train dataset
  train_data = data['train'].map(preprocess_data)
  train_data = train_data.shuffle(1000)
  train_data = train_data.batch(batch_size)

  # Create validation dataset
  validation_data = data['test'].map(preprocess_data)
  validation_data = validation_data.batch(64)

  return train_data, validation_data


def create_model(units, learning_rate, momentum):
  """Defines and compiles model."""

  inputs = tf.keras.Input(shape=(150, 150, 3))
  x = tf.keras.layers.Conv2D(16, (3, 3), activation='relu')(inputs)
  x = tf.keras.layers.MaxPooling2D((2, 2))(x)
  x = tf.keras.layers.Conv2D(32, (3, 3), activation='relu')(x)
  x = tf.keras.layers.MaxPooling2D((2, 2))(x)
  x = tf.keras.layers.Conv2D(64, (3, 3), activation='relu')(x)
  x = tf.keras.layers.MaxPooling2D((2, 2))(x)
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(units, activation='relu')(x)
  outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)
  model = tf.keras.Model(inputs, outputs)
  model.compile(
      loss='binary_crossentropy',
      optimizer=tf.keras.optimizers.SGD(
          learning_rate=learning_rate, momentum=momentum),
      metrics=['accuracy'])
  return model


def main():
  args = get_args()

  # Create Strategy
  strategy = tf.distribute.MirroredStrategy()

  # Scale batch size
  GLOBAL_BATCH_SIZE = 64 * strategy.num_replicas_in_sync  
  train_data, validation_data = create_dataset(GLOBAL_BATCH_SIZE)

  # Wrap model variables within scope
  with strategy.scope():
    model = create_model(args.units, args.learning_rate, args.momentum)

  # Train model
  history = model.fit(
      train_data, epochs=args.epochs, validation_data=validation_data)

  # Define Metric
  hp_metric = history.history['val_accuracy'][-1]

  hpt = hypertune.HyperTune()
  hpt.report_hyperparameter_tuning_metric(
      hyperparameter_metric_tag='accuracy',
      metric_value=hp_metric,
      global_step=args.epochs)


if __name__ == '__main__':
  main()

Build the Container

In the next cells, you build the container and push it to Google Container Registry.

In [ ]:

# Set the IMAGE_URI
IMAGE_URI = f"gcr.io/{PROJECT_ID}/horse-human:hypertune"

In [ ]:

# Build the docker image
! docker build -f Dockerfile -t $IMAGE_URI ./

In [ ]:

# Push it to Google Container Registry:
! docker push $IMAGE_URI

Create and run hyperparameter tuning job on Vertex AI

Once your container is pushed to Google Container Registry, you use the Vertex SDK to create and run the hyperparameter tuning job.

You define the following specifications:

• worker_pool_specs: Dictionary specifying the machine type and Docker image. This example defines a single node cluster with one n1-standard-4 machine with two NVIDIA_TESLA_T4 GPUs.
• parameter_spec: Dictionary specifying the parameters to optimize. The dictionary key is the string assigned to the command line argument for each hyperparameter in your training application code, and the dictionary value is the parameter specification. The parameter specification includes the type, min/max values, and scale for the hyperparameter.
• metric_spec: Dictionary specifying the metric to optimize. The dictionary key is the hyperparameter_metric_tag that you set in your training application code, and the value is the optimization goal.

In [ ]:

worker_pool_specs = [
    {
        "machine_spec": {
            "machine_type": "n1-standard-4",
            "accelerator_type": "NVIDIA_TESLA_T4",
            "accelerator_count": 2,
        },
        "replica_count": 1,
        "container_spec": {"image_uri": IMAGE_URI},
    }
]

metric_spec = {"accuracy": "maximize"}

parameter_spec = {
    "learning_rate": hpt.DoubleParameterSpec(min=0.001, max=1, scale="log"),
    "momentum": hpt.DoubleParameterSpec(min=0, max=1, scale="linear"),
    "units": hpt.DiscreteParameterSpec(values=[64, 128, 512], scale=None),
}

Create a CustomJob.

In [ ]:

print(BUCKET_URI)

In [ ]:

# Create a CustomJob

JOB_NAME = "horses-humans-hyperparam-job" + TIMESTAMP

my_custom_job = aiplatform.CustomJob(
    display_name=JOB_NAME,
    project=PROJECT_ID,
    worker_pool_specs=worker_pool_specs,
    staging_bucket=BUCKET_URI,
)

Then, create and run a HyperparameterTuningJob.

There are a few arguments to note:

• max_trial_count: Sets an upper bound on the number of trials the service will run. The recommended practice is to start with a smaller number of trials and get a sense of how impactful your chosen hyperparameters are before scaling up.

• parallel_trial_count: If you use parallel trials, the service provisions multiple training processing clusters. The worker pool spec that you specify when creating the job is used for each individual training cluster. Increasing the number of parallel trials reduces the amount of time the hyperparameter tuning job takes to run; however, it can reduce the effectiveness of the job overall. This is because the default tuning strategy uses results of previous trials to inform the assignment of values in subsequent trials.

• search_algorithm: The available search algorithms are grid, random, or default (None). The default option applies Bayesian optimization to search the space of possible hyperparameter values and is the recommended algorithm.

In [ ]:

# Create and run HyperparameterTuningJob

hp_job = aiplatform.HyperparameterTuningJob(
    display_name=JOB_NAME,
    custom_job=my_custom_job,
    metric_spec=metric_spec,
    parameter_spec=parameter_spec,
    max_trial_count=15,
    parallel_trial_count=3,
    project=PROJECT_ID,
    search_algorithm=None,
)

hp_job.run()

Click on the generated link in the output to see your run in the Cloud Console. When the job completes, you will see the results of the tuning trials.

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial:

In [ ]:

# Set this to true only if you'd like to delete your bucket
delete_bucket = False

if delete_bucket or os.getenv("IS_TESTING"):
    ! gcloud storage rm --recursive $BUCKET_URI