mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
feat: migrate hpt distributed (#1672)
* feat: migrate hpt distributed * fix: lint * Update distributed_hyperparameter_tuning.ipynb
This commit is contained in:
Andrew Ferlitsch
@@ -0,0 +1,762 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "9Nmi2KIicB7S"
|
||||
},
|
||||
"source": [
|
||||
"# Distributed Vertex AI Hyperparameter Tuning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "JAPoU8Sm5E6e"
|
||||
},
|
||||
"source": [
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/distributed_hyperparameter_tuning.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/distributed_hyperparameter_tuning.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/training/distributed_hyperparameter_tuning.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c582cc111a01"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to run a hyperparameter tuning job with Vertex AI Training to discover optimal hyperparameter values for an ML model. To speed up the training process, `MirroredStrategy` from the `tf.distribute` module is used to distribute training across multiple GPUs on a single machine.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Hyperparameter Tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "60d57f1c2ae0"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this notebook, you create a custom trained model from a Python script in a Docker container. You learn how to modify training application code for hyperparameter tuning and submit a Vertex AI Hyperparameter Tuning job with the Python SDK.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
"- `Vertex AI Training`\n",
|
||||
"- `Vertex AI Hyperparameter Tuning`\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Training using a Python package.\n",
|
||||
"- Report accuracy when hyperparameter tuning.\n",
|
||||
"- Save the model artifacts to Cloud Storage using GCSFuse."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the [horses or humans dataset](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets). The trained model predicts if an image is of a horse or a human.\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages to execute this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "wyy5Lbnzg5fi"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade google-cloud-aiplatform -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e9255e3b156f"
|
||||
},
|
||||
"source": [
|
||||
"### Colab Only: Uncomment the following cell to restart the kernel"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "0c0b2427998a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
|
||||
"# import IPython\n",
|
||||
"\n",
|
||||
"# app = IPython.Application.instance()\n",
|
||||
"# app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "435b8e413535"
|
||||
},
|
||||
"source": [
|
||||
"### Before you begin\n",
|
||||
"\n",
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, try the following:\n",
|
||||
"- Run `gcloud config list`\n",
|
||||
"- Run `gcloud projects list`\n",
|
||||
"- See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "be175254a715"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# set the project id\n",
|
||||
"! gcloud config set project $PROJECT_ID"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2e6b8b324ce1"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable used by Vertex AI. \n",
|
||||
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6c43a8673066"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
|
||||
"\n",
|
||||
"**1. Vertex AI Workbench** \n",
|
||||
"- Do nothing as you are already authenticated.\n",
|
||||
"\n",
|
||||
"**2. Local JupyterLab Instance,** uncomment and run."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "fbc9cd30cc4b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! gcloud auth login"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cd0da2c26879"
|
||||
},
|
||||
"source": [
|
||||
"**3. Colab,** uncomment and run:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "a336a05c6149"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# from google.colab import auth\n",
|
||||
"# auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f6b2ccc891ed"
|
||||
},
|
||||
"source": [
|
||||
"**4. Service account or other**\n",
|
||||
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e5755d1a554f"
|
||||
},
|
||||
"source": [
|
||||
"### Create a Cloud Storage bucket\n",
|
||||
"\n",
|
||||
"Create a storage bucket to store intermediate artifacts such as datasets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "d2de92accb67"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = f\"gs://your-bucket-name-unique-{PROJECT_ID}\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "-EcIXiGsCePi"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "NIq7R4HZCfIc"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XoEqT2Y4DJmf"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "pRUOFELefqf1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from google.cloud.aiplatform import hyperparameter_tuning as hpt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "reBCSTKOg47l"
|
||||
},
|
||||
"source": [
|
||||
"### Write Dockerfile\n",
|
||||
"\n",
|
||||
"The first step in containerizing your code is to create a Dockerfile. In the Dockerfile, you'll include all the commands needed to run the image such as installing the necessary libraries and setting up the entry point for the training code.\n",
|
||||
"\n",
|
||||
"This Dockerfile uses the Deep Learning Container TensorFlow Enterprise 2.5 GPU Docker image. The Deep Learning Containers on Google Cloud come with many common ML and data science frameworks pre-installed. After downloading that image, this Dockerfile installs the [CloudML Hypertune](https://github.com/GoogleCloudPlatform/cloudml-hypertune) library and sets up the entrypoint for the training code.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e231837fe138"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile Dockerfile\n",
|
||||
"\n",
|
||||
"FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5\n",
|
||||
"WORKDIR /\n",
|
||||
"\n",
|
||||
"# Installs hypertune library\n",
|
||||
"RUN pip install cloudml-hypertune\n",
|
||||
"\n",
|
||||
"# Copies the trainer code to the docker image.\n",
|
||||
"COPY trainer /trainer\n",
|
||||
"\n",
|
||||
"# Sets up the entry point to invoke the trainer.\n",
|
||||
"ENTRYPOINT [\"python\", \"-m\", \"trainer.task\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "4c2ea367c79d"
|
||||
},
|
||||
"source": [
|
||||
"### Create training application code\n",
|
||||
"\n",
|
||||
"Next, you create a trainer directory with a `task.py` script that contains the code for your training application."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "MjJTYC86hPOZ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create trainer directory\n",
|
||||
"\n",
|
||||
"! mkdir trainer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ea83c1253a74"
|
||||
},
|
||||
"source": [
|
||||
"In the next cell, you write the contents of the training script, `task.py`. This file downloads the _horses or humans_ dataset from TensorFlow datasets and trains a `tf.keras` functional model using `MirroredStrategy` from the `tf.distribute` module.\n",
|
||||
"\n",
|
||||
"There are a few components that are specific to using the hyperparameter tuning service:\n",
|
||||
"\n",
|
||||
"* The script imports the `hypertune` library. Note that the Dockerfile included instructions to pip install the hypertune library.\n",
|
||||
"* The function `get_args()` defines a command-line argument for each hyperparameter you want to tune. In this example, the hyperparameters that will be tuned are the learning rate, the momentum value in the optimizer, and the number of units in the last hidden layer of the model. The value passed in those arguments is then used to set the corresponding hyperparameter in the code.\n",
|
||||
"* At the end of the `main()` function, the hypertune library is used to define the metric to optimize. In this example, the metric that will be optimized is the the validation accuracy. This metric is passed to an instance of `HyperTune`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "9b52fd75d90f"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile trainer/task.py\n",
|
||||
"\n",
|
||||
"import argparse\n",
|
||||
"import hypertune\n",
|
||||
"import tensorflow as tf\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"def get_args():\n",
|
||||
" \"\"\"Parses args. Must include all hyperparameters you want to tune.\"\"\"\n",
|
||||
"\n",
|
||||
" parser = argparse.ArgumentParser()\n",
|
||||
" parser.add_argument(\n",
|
||||
" '--learning_rate', required=True, type=float, help='learning rate')\n",
|
||||
" parser.add_argument(\n",
|
||||
" '--momentum', required=True, type=float, help='SGD momentum value')\n",
|
||||
" parser.add_argument(\n",
|
||||
" '--units',\n",
|
||||
" required=True,\n",
|
||||
" type=int,\n",
|
||||
" help='number of units in last hidden layer')\n",
|
||||
" parser.add_argument(\n",
|
||||
" '--epochs',\n",
|
||||
" required=False,\n",
|
||||
" type=int,\n",
|
||||
" default=10,\n",
|
||||
" help='number of training epochs')\n",
|
||||
" args = parser.parse_args()\n",
|
||||
" return args\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def preprocess_data(image, label):\n",
|
||||
" \"\"\"Resizes and scales images.\"\"\"\n",
|
||||
"\n",
|
||||
" image = tf.image.resize(image, (150, 150))\n",
|
||||
" return tf.cast(image, tf.float32) / 255., label\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def create_dataset(batch_size):\n",
|
||||
" \"\"\"Loads Horses Or Humans dataset and preprocesses data.\"\"\"\n",
|
||||
"\n",
|
||||
" data, info = tfds.load(\n",
|
||||
" name='horses_or_humans', as_supervised=True, with_info=True)\n",
|
||||
"\n",
|
||||
" # Create train dataset\n",
|
||||
" train_data = data['train'].map(preprocess_data)\n",
|
||||
" train_data = train_data.shuffle(1000)\n",
|
||||
" train_data = train_data.batch(batch_size)\n",
|
||||
"\n",
|
||||
" # Create validation dataset\n",
|
||||
" validation_data = data['test'].map(preprocess_data)\n",
|
||||
" validation_data = validation_data.batch(64)\n",
|
||||
"\n",
|
||||
" return train_data, validation_data\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def create_model(units, learning_rate, momentum):\n",
|
||||
" \"\"\"Defines and compiles model.\"\"\"\n",
|
||||
"\n",
|
||||
" inputs = tf.keras.Input(shape=(150, 150, 3))\n",
|
||||
" x = tf.keras.layers.Conv2D(16, (3, 3), activation='relu')(inputs)\n",
|
||||
" x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
|
||||
" x = tf.keras.layers.Conv2D(32, (3, 3), activation='relu')(x)\n",
|
||||
" x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
|
||||
" x = tf.keras.layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
|
||||
" x = tf.keras.layers.MaxPooling2D((2, 2))(x)\n",
|
||||
" x = tf.keras.layers.Flatten()(x)\n",
|
||||
" x = tf.keras.layers.Dense(units, activation='relu')(x)\n",
|
||||
" outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n",
|
||||
" model = tf.keras.Model(inputs, outputs)\n",
|
||||
" model.compile(\n",
|
||||
" loss='binary_crossentropy',\n",
|
||||
" optimizer=tf.keras.optimizers.SGD(\n",
|
||||
" learning_rate=learning_rate, momentum=momentum),\n",
|
||||
" metrics=['accuracy'])\n",
|
||||
" return model\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def main():\n",
|
||||
" args = get_args()\n",
|
||||
"\n",
|
||||
" # Create Strategy\n",
|
||||
" strategy = tf.distribute.MirroredStrategy()\n",
|
||||
"\n",
|
||||
" # Scale batch size\n",
|
||||
" GLOBAL_BATCH_SIZE = 64 * strategy.num_replicas_in_sync \n",
|
||||
" train_data, validation_data = create_dataset(GLOBAL_BATCH_SIZE)\n",
|
||||
"\n",
|
||||
" # Wrap model variables within scope\n",
|
||||
" with strategy.scope():\n",
|
||||
" model = create_model(args.units, args.learning_rate, args.momentum)\n",
|
||||
"\n",
|
||||
" # Train model\n",
|
||||
" history = model.fit(\n",
|
||||
" train_data, epochs=args.epochs, validation_data=validation_data)\n",
|
||||
"\n",
|
||||
" # Define Metric\n",
|
||||
" hp_metric = history.history['val_accuracy'][-1]\n",
|
||||
"\n",
|
||||
" hpt = hypertune.HyperTune()\n",
|
||||
" hpt.report_hyperparameter_tuning_metric(\n",
|
||||
" hyperparameter_metric_tag='accuracy',\n",
|
||||
" metric_value=hp_metric,\n",
|
||||
" global_step=args.epochs)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"if __name__ == '__main__':\n",
|
||||
" main()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2dc0a526f77e"
|
||||
},
|
||||
"source": [
|
||||
"### Build the Container\n",
|
||||
"\n",
|
||||
"In the next cells, you build the container and push it to Google Container Registry."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "a42d0b918ab4"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set the IMAGE_URI\n",
|
||||
"IMAGE_URI = f\"gcr.io/{PROJECT_ID}/horse-human:hypertune\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "360a5271fbdb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Build the docker image\n",
|
||||
"! docker build -f Dockerfile -t $IMAGE_URI ./"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "526475da6370"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Push it to Google Container Registry:\n",
|
||||
"! docker push $IMAGE_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "aaff6f5be7f6"
|
||||
},
|
||||
"source": [
|
||||
"### Create and run hyperparameter tuning job on Vertex AI\n",
|
||||
"\n",
|
||||
"Once your container is pushed to Google Container Registry, you use the Vertex SDK to create and run the hyperparameter tuning job.\n",
|
||||
"\n",
|
||||
"You define the following specifications:\n",
|
||||
"* `worker_pool_specs`: Dictionary specifying the machine type and Docker image. This example defines a single node cluster with one `n1-standard-4` machine with two `NVIDIA_TESLA_T4` GPUs.\n",
|
||||
"* `parameter_spec`: Dictionary specifying the parameters to optimize. The dictionary key is the string assigned to the command line argument for each hyperparameter in your training application code, and the dictionary value is the parameter specification. The parameter specification includes the type, min/max values, and scale for the hyperparameter.\n",
|
||||
"* `metric_spec`: Dictionary specifying the metric to optimize. The dictionary key is the `hyperparameter_metric_tag` that you set in your training application code, and the value is the optimization goal."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "aec22792ee84"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"worker_pool_specs = [\n",
|
||||
" {\n",
|
||||
" \"machine_spec\": {\n",
|
||||
" \"machine_type\": \"n1-standard-4\",\n",
|
||||
" \"accelerator_type\": \"NVIDIA_TESLA_T4\",\n",
|
||||
" \"accelerator_count\": 2,\n",
|
||||
" },\n",
|
||||
" \"replica_count\": 1,\n",
|
||||
" \"container_spec\": {\"image_uri\": IMAGE_URI},\n",
|
||||
" }\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"metric_spec = {\"accuracy\": \"maximize\"}\n",
|
||||
"\n",
|
||||
"parameter_spec = {\n",
|
||||
" \"learning_rate\": hpt.DoubleParameterSpec(min=0.001, max=1, scale=\"log\"),\n",
|
||||
" \"momentum\": hpt.DoubleParameterSpec(min=0, max=1, scale=\"linear\"),\n",
|
||||
" \"units\": hpt.DiscreteParameterSpec(values=[64, 128, 512], scale=None),\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ffd01019a764"
|
||||
},
|
||||
"source": [
|
||||
"Create a `CustomJob`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "f2eed1471a24"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create a CustomJob\n",
|
||||
"\n",
|
||||
"JOB_NAME = \"horses-humans-hyperparam-job\"\n",
|
||||
"\n",
|
||||
"my_custom_job = aiplatform.CustomJob(\n",
|
||||
" display_name=JOB_NAME,\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" worker_pool_specs=worker_pool_specs,\n",
|
||||
" staging_bucket=BUCKET_URI,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0e5ee7ee5ae4"
|
||||
},
|
||||
"source": [
|
||||
"Then, create and run a `HyperparameterTuningJob`.\n",
|
||||
"\n",
|
||||
"There are a few arguments to note:\n",
|
||||
"\n",
|
||||
"* `max_trial_count`: Sets an upper bound on the number of trials the service will run. The recommended practice is to start with a smaller number of trials and get a sense of how impactful your chosen hyperparameters are before scaling up.\n",
|
||||
"\n",
|
||||
"* `parallel_trial_count`: If you use parallel trials, the service provisions multiple training processing clusters. The worker pool spec that you specify when creating the job is used for each individual training cluster. Increasing the number of parallel trials reduces the amount of time the hyperparameter tuning job takes to run; however, it can reduce the effectiveness of the job overall. This is because the default tuning strategy uses results of previous trials to inform the assignment of values in subsequent trials.\n",
|
||||
" \n",
|
||||
"* `search_algorithm`: The available search algorithms are grid, random, or default (None). The default option applies Bayesian optimization to search the space of possible hyperparameter values and is the recommended algorithm."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "bb6ca1b103ef"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create and run HyperparameterTuningJob\n",
|
||||
"\n",
|
||||
"hp_job = aiplatform.HyperparameterTuningJob(\n",
|
||||
" display_name=JOB_NAME,\n",
|
||||
" custom_job=my_custom_job,\n",
|
||||
" metric_spec=metric_spec,\n",
|
||||
" parameter_spec=parameter_spec,\n",
|
||||
" max_trial_count=15,\n",
|
||||
" parallel_trial_count=3,\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" search_algorithm=None,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"hp_job.run()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "396d86efe829"
|
||||
},
|
||||
"source": [
|
||||
"Click on the generated link in the output to see your run in the Cloud Console. When the job completes, you will see the results of the tuning trials."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "bee87f15ff24"
|
||||
},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TpV-iwP9qw9c"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sx_vKniMq9ZX"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"hp_job.delete()\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "distributed_hyperparameter_tuning.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
Reference in New Issue
Block a user