From a9af97f1b91048ddb448b79fd7d09bbb1ce3b95a Mon Sep 17 00:00:00 2001 From: zbl94 <106645026+zbl94@users.noreply.github.com> Date: Wed, 3 Aug 2022 13:37:14 -0700 Subject: [PATCH] Added notebook demonstrating Tensorboard Custom Training with custom container. (#611) * Added notebook demonstrating Tensorboard Custom Training with custom container. * Added notebook demonstrating Tensorboard Custom Training with custom container. * update codeowners file * call Vertex API instead of gapic API * resolve comments for custom container * resolve comments and format * resolve comments * using --quiet for delete doctor repository * address more comments Co-authored-by: gericdong Co-authored-by: Andrew Ferlitsch --- ...ustom_training_with_custom_container.ipynb | 999 ++++++++++++++++++ 1 file changed, 999 insertions(+) create mode 100644 notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb diff --git a/notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb b/notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb new file mode 100644 index 000000000..c090cb441 --- /dev/null +++ b/notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb @@ -0,0 +1,999 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Gpc9ZhH_W4AP" + }, + "outputs": [], + "source": [ + "# Copyright 2022 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ed2pOXQMb8fY" + }, + "source": [ + "# Vertex AI TensorBoard Custom Training with Custom Container\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"GoogleOpen in Cloud Notebook\n", + " \n", + " \n", + " \n", + " \"Colab Open in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GXqVc0ADMq__" + }, + "source": [ + "## Overview" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WagqZOn_cdCb" + }, + "source": [ + "### What is Vertex AI TensorBoard\n", + "\n", + "[Open source TensorBoard](https://www.tensorflow.org/tensorboard/get_started)\n", + "(TB) is a Google open source project for machine learning experiment\n", + "visualization. Vertex AI TensorBoard is an enterprise-ready managed\n", + "version of TensorBoard.\n", + "\n", + "Vertex AI TensorBoard provides various detailed visualizations, that\n", + "includes:\n", + "\n", + "* Tracking and visualizing metrics such as loss and accuracy over time\n", + "* Visualizing model computational graphs (ops and layers)\n", + "* Viewing histograms of weights, biases, or other tensors as they change over time\n", + "* Projecting embeddings to a lower dimensional space\n", + "* Displaying image, text, and audio samples\n", + "\n", + "In addition to the powerful visualizations from\n", + "TensorBoard, Vertex AI TensorBoard provides:\n", + "\n", + "* A persistent, shareable link to your experiment's dashboard\n", + "\n", + "* A searchable list of all experiments in a project\n", + "\n", + "* Tight integrations with Vertex AI services for model training\n", + "\n", + "* Enterprise-grade security, privacy, and compliance\n", + "\n", + "With Vertex AI TensorBoard, you can track, visualize, and compare\n", + "ML experiments and share them with your team.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wkpL6qqIM1pO" + }, + "source": [ + "### Dataset\n", + "\n", + "Dataset used in this tutorial will be the [flower dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) provided by TensorFlow. No other datasets are required." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DFHvu3pYM1gY" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you learn how to create a custom training job using custom containers, and monitor your training process on Vertex AI TensorBoard in near real time.\n", + "\n", + "The steps performed include:\n", + "\n", + "* Create docker repository & config.\n", + "* Create a custom container image with your customized training code.\n", + "* Setup service account and Google Cloud Storage buckets.\n", + "* Create & launch your custom training job with your custom container." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lCzg9s63M1Oq" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "* Google Artifact Registry\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),[Cloud Storage pricing](https://cloud.google.com/storage/pricing), and [Google Artifact Registry pricing](https://cloud.google.com/artifact-registry/pricing). \n", + "\n", + "Use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rAPBk_OCNN_h" + }, + "source": [ + "### Set up your local development environment\n", + "\n", + "**If you are using Colab or Vertex AI Workbench**, your environment already meets all the requirements to run this notebook. You can skip this step.\n", + "\n", + "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n", + "\n", + "- The Cloud Storage SDK\n", + "- Git\n", + "- Python 3\n", + "- virtualenv\n", + "- Jupyter notebook running in a virtual environment with Python 3\n", + "\n", + "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n", + "\n", + "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n", + "\n", + "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n", + "\n", + "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n", + "\n", + "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n", + "\n", + "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", + "\n", + "6. Open this notebook in the Jupyter Notebook Dashboard." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0Edien3sDfJm" + }, + "source": [ + "### Install additional packages\n", + "\n", + "Install the following packages required to execute this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "n21c5UHogVEY" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "# The Vertex AI Workbench Notebook product has specific requirements\n", + "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n", + "\n", + "# Vertex AI Notebook requires dependencies to be installed with '--user'\n", + "USER_FLAG = \"\"\n", + "if IS_WORKBENCH_NOTEBOOK:\n", + " USER_FLAG = \"--user\"\n", + "\n", + "! pip3 install google-cloud-aiplatform {USER_FLAG} -q\n", + "\n", + "# Automatically restart kernel after installs\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r_dA3M6UJELw" + }, + "source": [ + "## Before you begin" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1Dunp1YrhPYo" + }, + "source": [ + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the Vertex AI, Cloud Storage, Cloud Build, and Artifact Registry APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,storage-component.googleapis.com,cloudbuild.googleapis.com,artifactregistry.googleapis.com)\n", + "\n", + "4. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", + "\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "Note: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lMqIOZfqhXyD" + }, + "source": [ + "### Set your project ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "siN0RvWyZYv9" + }, + "source": [ + "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lU-1yD4jZc-6" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"\"\n", + "\n", + "import os\n", + "\n", + "# Get your Google Cloud project ID from gcloud\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID: \", PROJECT_ID)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XNVqyIvYaPO-" + }, + "source": [ + "Otherwise, set your project ID here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "s0F0U7WZhfAZ" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None:\n", + " PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "K-KuU54IaVz5" + }, + "source": [ + "### Timestamp" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "doEJxrvsaWyt" + }, + "source": [ + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rheJHuCIadAL" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ee3vBgvdhgTb" + }, + "source": [ + "### Set your region" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "flcwBahRhi8b" + }, + "outputs": [], + "source": [ + "REGION = \"[your-region]\" # @param {type: \"string\"}\n", + "if REGION == \"[your-region]\":\n", + " REGION = \"us-central1\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KuNRbXkIijp6" + }, + "source": [ + "### Login to your Google Cloud account" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "P9vQxUzfirCV" + }, + "outputs": [], + "source": [ + "# The Google Cloud Notebook product has specific requirements\n", + "import os\n", + "import sys\n", + "\n", + "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n", + "\n", + "IS_COLAB = \"google.colab\" in sys.modules\n", + "# If on Google Cloud Notebooks, then don't execute this code\n", + "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n", + " if IS_COLAB:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OG3dstAuVtgz" + }, + "source": [ + "### Create Cloud Storage bucket" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TKVkz5uzV07p" + }, + "source": [ + "A Cloud Storage buckets will be used store your training code output (including TensorBoard logs). The bucket must be regional that is, not multi-region or dual-region, and the following resources must be in same region:\n", + "\n", + "* the Cloud Storage bucket\n", + "* the Vertex AI training job\n", + "* the Vertex AI TensorBoard instance" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_BzfwD8EV7Fw" + }, + "outputs": [], + "source": [ + "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n", + "\n", + "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", + " BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iVe3Y5UGWDoC" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket. The created bucket will be deleted in the cleaning up section in the end. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "M0krsPd9WJYl" + }, + "outputs": [], + "source": [ + "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oXo9TNzQWQBS" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BD-n41FzWY0G" + }, + "outputs": [], + "source": [ + "! gsutil ls -al {BUCKET_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "q73OgpxyDtSk" + }, + "source": [ + "### Import aiplatform" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "yO4wki2xD0Ys" + }, + "outputs": [], + "source": [ + "import google.cloud.aiplatform as aiplatform" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "79WoncsWbLTg" + }, + "source": [ + "### Initialize Vertex AI SDK for Python\n", + "Initialize the Vertex AI SDK for Python for your project and corresponding bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "y4qLmB4WbLwo" + }, + "outputs": [], + "source": [ + "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0j1NWIQEJI5i" + }, + "source": [ + "## Create Docker repository" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hNmHMIyjBzxx" + }, + "source": [ + "Create a Docker repository named `DOCKER_REPOSITORY` in your `REGION`.\n", + "This docker repository will be deleted in the clearning up section in the end." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "iz9CUUbdBTvF" + }, + "outputs": [], + "source": [ + "DOCKER_REPOSITORY = \"[your-docker-repository-name]\" # @param {type:\"string\"}\n", + "\n", + "if (\n", + " DOCKER_REPOSITORY == \"\"\n", + " or DOCKER_REPOSITORY is None\n", + " or DOCKER_REPOSITORY == \"[your-docker-repository-name]\"\n", + "):\n", + " DOCKER_REPOSITORY = \"tb-docker-repo-\" + PROJECT_ID + \"-\" + TIMESTAMP\n", + "\n", + "print(\"Docker repository to create:\", DOCKER_REPOSITORY)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "R9eQDigtC8XA" + }, + "outputs": [], + "source": [ + "! gcloud artifacts repositories create $DOCKER_REPOSITORY --project={PROJECT_ID} \\\n", + "--repository-format=docker \\\n", + "--location={REGION} --description=\"Repository for TensorBoard Custom Training Job\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V0pBSC0rDlvq" + }, + "source": [ + "Verify your Docker repository is created successfully." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fgHYvZlwDuAN" + }, + "outputs": [], + "source": [ + "! gcloud artifacts repositories list --project={PROJECT_ID}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "laaA77LsIt0c" + }, + "source": [ + "## Create a custom container image and push to Artifact Registry\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AVcNnHfEKBdm" + }, + "outputs": [], + "source": [ + "# Create a folder for the image.\n", + "! mkdir tb-custom-container\n", + "%cd tb-custom-container" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jUcVG77dKmPn" + }, + "source": [ + "Write your own training code in task.py file. You can use the following code as an example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7k1JDOGNKL3d" + }, + "outputs": [], + "source": [ + "%%writefile task.py\n", + "\n", + "import logging\n", + "import os\n", + "\n", + "import tensorflow as tf\n", + "import tensorflow_datasets as tfds\n", + "\n", + "IMG_WIDTH = 128\n", + "\n", + "def normalize_img(image):\n", + " \"\"\"Normalizes image.\n", + "\n", + " * Resizes image to IMG_WIDTH x IMG_WIDTH pixels\n", + " * Casts values from `uint8` to `float32`\n", + " * Scales values from [0, 255] to [0, 1]\n", + "\n", + " Returns:\n", + " A tensor with shape (IMG_WIDTH, IMG_WIDTH, 3). (3 color channels)\n", + " \"\"\"\n", + " image = tf.image.resize_with_pad(image, IMG_WIDTH, IMG_WIDTH)\n", + " return image / 255.\n", + "\n", + "\n", + "def normalize_img_and_label(image, label):\n", + " \"\"\"Normalizes image and label.\n", + "\n", + " * Performs normalize_img on image\n", + " * Passes through label unchanged\n", + "\n", + " Returns:\n", + " Tuple (image, label) where\n", + " * image is a tensor with shape (IMG_WIDTH, IMG_WIDTH, 3). (3 color\n", + " channels)\n", + " * label is an unchanged integer [0, 4] representing flower type\n", + " \"\"\"\n", + " return normalize_img(image), label\n", + "\n", + "logging.info('Loading and preprocessing data ...')\n", + "dataset = tfds.load('tf_flowers:3.*.*',\n", + " split='train',\n", + " try_gcs=True,\n", + " shuffle_files=True,\n", + " as_supervised=True)\n", + "dataset = dataset.map(normalize_img_and_label,\n", + " num_parallel_calls=tf.data.experimental.AUTOTUNE)\n", + "dataset = dataset.cache()\n", + "dataset = dataset.shuffle(1000)\n", + "dataset = dataset.batch(128)\n", + "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n", + "\n", + "logging.info('Creating and training model ...')\n", + "\n", + "model = tf.keras.Sequential([\n", + " tf.keras.layers.Conv2D(16,\n", + " 3,\n", + " padding='same',\n", + " activation='relu',\n", + " input_shape=(IMG_WIDTH, IMG_WIDTH, 3)),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Flatten(),\n", + " tf.keras.layers.Dense(512, activation=\"relu\"),\n", + " tf.keras.layers.Dense(5) # 5 classes\n", + "])\n", + "\n", + "logging.info('Compiling model ...')\n", + "model.compile(\n", + " optimizer='adam',\n", + " loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", + " metrics=['accuracy'])\n", + "\n", + "log_dir = \"logs\"\n", + "if 'AIP_TENSORBOARD_LOG_DIR' in os.environ:\n", + " log_dir=os.environ['AIP_TENSORBOARD_LOG_DIR']\n", + "\n", + "tensorboard_callback = tf.keras.callbacks.TensorBoard(\n", + " log_dir=log_dir,\n", + " histogram_freq=1)\n", + "\n", + "logging.info('Training model ...')\n", + "model.fit(dataset, epochs=13, callbacks=[tensorboard_callback])\n", + "\n", + "logging.info('Model training done')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hvNddrgPP9bs" + }, + "source": [ + "Create your own `Dockerfile` to specify all instructions needed to build your container. You can use the following `Dockerfile` as an example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fxi1HZbKQUKY" + }, + "outputs": [], + "source": [ + "%%writefile Dockerfile\n", + "\n", + "# Specifies base image and tag\n", + "FROM us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-8:latest\n", + "WORKDIR /root\n", + "\n", + "\n", + "# Installs additional packages as you need.\n", + "\n", + "# Copies the trainer code to the docker image.\n", + "COPY task.py /root/task.py\n", + "\n", + "# Sets up the entry point to invoke the trainer.\n", + "ENTRYPOINT [\"python\", \"task.py\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DK2E1xz8Q7Q-" + }, + "source": [ + "Build your container image using `gcloud builds` from your training code and `Dockerfile`. Note that this step may take a few minutes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BvyoFJW_LJeG" + }, + "outputs": [], + "source": [ + "IMAGE_NAME = \"tensorboard-custom-container\"\n", + "IMAGE_TAG = \"v1\"\n", + "IMAGE_URI = \"us-central1-docker.pkg.dev/{}/{}/{}:{}\".format(\n", + " PROJECT_ID, DOCKER_REPOSITORY, IMAGE_NAME, IMAGE_TAG\n", + ")\n", + "\n", + "! gcloud builds submit --project {PROJECT_ID} --region={REGION} --tag {IMAGE_URI} --timeout=20m" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hwXxa4Qgnh4Y" + }, + "source": [ + "## Setup service account and permissions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7qXFUiHLoFRw" + }, + "source": [ + "A service account will be used to create custom training job. If you do not want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID. You can create a service account by following the [instruction](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mpKjfsXumuNV" + }, + "outputs": [], + "source": [ + "SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RdWLe8S_tlee" + }, + "outputs": [], + "source": [ + "if (\n", + " SERVICE_ACCOUNT == \"\"\n", + " or SERVICE_ACCOUNT is None\n", + " or SERVICE_ACCOUNT == \"[your-service-account]\"\n", + "):\n", + " # Get your service account from gcloud\n", + " if not IS_COLAB:\n", + " shell_output = ! gcloud auth list 2>/dev/null\n", + " SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n", + "\n", + " else: # IS_COLAB:\n", + " shell_output = ! gcloud projects describe $PROJECT_ID\n", + " project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n", + " SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n", + "\n", + " print(\"Service Account:\", SERVICE_ACCOUNT)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UlDhuciOt5vo" + }, + "outputs": [], + "source": [ + "# Grant Cloud Storage permission.\n", + "! gcloud projects add-iam-policy-binding {PROJECT_ID} \\\n", + " --member=serviceAccount:{SERVICE_ACCOUNT} \\\n", + " --role=roles/storage.admin" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lTKVB71soRyr" + }, + "outputs": [], + "source": [ + "# Grant AI Platform permission.\n", + "! gcloud projects add-iam-policy-binding {PROJECT_ID} \\\n", + " --member=serviceAccount:{SERVICE_ACCOUNT} \\\n", + " --role=roles/aiplatform.user" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IaQjIPvuKLwW" + }, + "source": [ + "## Create a custom training job with your container" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "svUGBOow_Obj" + }, + "source": [ + "Create a TensorBoard instnace to be used by the custom training job." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OAe1xJeS_X3F" + }, + "outputs": [], + "source": [ + "TENSORBOARD_NAME = \"[your-tensorboard-name]\" # @param {type:\"string\"}\n", + "\n", + "if (\n", + " TENSORBOARD_NAME == \"\"\n", + " or TENSORBOARD_NAME is None\n", + " or TENSORBOARD_NAME == \"[your-tensorboard-name]\"\n", + "):\n", + " TENSORBOARD_NAME = PROJECT_ID + \"-tb-\" + TIMESTAMP\n", + "\n", + "tensorboard = aiplatform.Tensorboard.create(\n", + " display_name=TENSORBOARD_NAME, project=PROJECT_ID, location=REGION\n", + ")\n", + "TENSORBOARD_RESOURCE_NAME = tensorboard.gca_resource.name\n", + "print(\"TensorBoard resource name:\", TENSORBOARD_RESOURCE_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mudxBDal_a_k" + }, + "source": [ + "Run the following example request to create your own custom training job using the container you just built and uploaded to Artifact Registry, and stream the training results to TensorBoard." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wugHEC8czuBe" + }, + "outputs": [], + "source": [ + "JOB_NAME = \"tensorboard-example-job-{}\".format(TIMESTAMP)\n", + "BASE_OUTPUT_DIR = \"{}/{}\".format(BUCKET_URI, JOB_NAME)\n", + "\n", + "job = aiplatform.CustomContainerTrainingJob(\n", + " display_name=JOB_NAME,\n", + " container_uri=IMAGE_URI,\n", + " project=PROJECT_ID,\n", + " location=REGION,\n", + " staging_bucket=BUCKET_URI,\n", + ")\n", + "\n", + "job.run(\n", + " machine_type=\"n1-standard-8\",\n", + " replica_count=1,\n", + " service_account=SERVICE_ACCOUNT,\n", + " tensorboard=TENSORBOARD_RESOURCE_NAME,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VfMsn_RnEtnj" + }, + "source": [ + "In Google Cloud console, you can monitor your training job at Vertex AI > Training > Custom Jobs. In each custom training job, near real time updated TensorBoard is available at `OPEN TENSORBOARD` button." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bmbjz-nUW7KE" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, **if you created the individual resources in the notebook** you can delete them as follow:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-LSKjrRqW-Bk" + }, + "outputs": [], + "source": [ + "# Delete GCS bucket.\n", + "! gsutil -m rm -r {BUCKET_URI}\n", + "\n", + "# Delete docker repository.\n", + "! gcloud artifacts repositories delete $DOCKER_REPOSITORY --project {PROJECT_ID} --location {REGION} --quiet\n", + "\n", + "# Delete TensorBoard instance.\n", + "! gcloud ai tensorboards delete {TENSORBOARD_RESOURCE_NAME}\n", + "\n", + "# Delete custom job.\n", + "job.delete()" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "tensorboard_custom_training_with_custom_container.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}