mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
Added notebook demonstrating Tensorboard Custom Training with custom container. (#611)
* Added notebook demonstrating Tensorboard Custom Training with custom container. * Added notebook demonstrating Tensorboard Custom Training with custom container. * update codeowners file * call Vertex API instead of gapic API * resolve comments for custom container * resolve comments and format * resolve comments * using --quiet for delete doctor repository * address more comments Co-authored-by: gericdong <itseric@google.com> Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
zbl94
co-authored by
gericdong
Andrew Ferlitsch
gericdong
Andrew Ferlitsch
parent
d6431fab4c
commit
a9af97f1b9
+999
@@ -0,0 +1,999 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Gpc9ZhH_W4AP"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ed2pOXQMb8fY"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI TensorBoard Custom Training with Custom Container\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmaster%2Fnotebooks%2Fofficial%2Ftensorboard%2Ftensorboard_custom_training_with_custom_container.ipynb\">\n",
|
||||
" <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Cloud Notebook\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/tensorboard/tensorboard_custom_training_with_custom_container.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "GXqVc0ADMq__"
|
||||
},
|
||||
"source": [
|
||||
"## Overview"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WagqZOn_cdCb"
|
||||
},
|
||||
"source": [
|
||||
"### What is Vertex AI TensorBoard\n",
|
||||
"\n",
|
||||
"[Open source TensorBoard](https://www.tensorflow.org/tensorboard/get_started)\n",
|
||||
"(TB) is a Google open source project for machine learning experiment\n",
|
||||
"visualization. Vertex AI TensorBoard is an enterprise-ready managed\n",
|
||||
"version of TensorBoard.\n",
|
||||
"\n",
|
||||
"Vertex AI TensorBoard provides various detailed visualizations, that\n",
|
||||
"includes:\n",
|
||||
"\n",
|
||||
"* Tracking and visualizing metrics such as loss and accuracy over time\n",
|
||||
"* Visualizing model computational graphs (ops and layers)\n",
|
||||
"* Viewing histograms of weights, biases, or other tensors as they change over time\n",
|
||||
"* Projecting embeddings to a lower dimensional space\n",
|
||||
"* Displaying image, text, and audio samples\n",
|
||||
"\n",
|
||||
"In addition to the powerful visualizations from\n",
|
||||
"TensorBoard, Vertex AI TensorBoard provides:\n",
|
||||
"\n",
|
||||
"* A persistent, shareable link to your experiment's dashboard\n",
|
||||
"\n",
|
||||
"* A searchable list of all experiments in a project\n",
|
||||
"\n",
|
||||
"* Tight integrations with Vertex AI services for model training\n",
|
||||
"\n",
|
||||
"* Enterprise-grade security, privacy, and compliance\n",
|
||||
"\n",
|
||||
"With Vertex AI TensorBoard, you can track, visualize, and compare\n",
|
||||
"ML experiments and share them with your team.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "wkpL6qqIM1pO"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"Dataset used in this tutorial will be the [flower dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) provided by TensorFlow. No other datasets are required."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "DFHvu3pYM1gY"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to create a custom training job using custom containers, and monitor your training process on Vertex AI TensorBoard in near real time.\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"* Create docker repository & config.\n",
|
||||
"* Create a custom container image with your customized training code.\n",
|
||||
"* Setup service account and Google Cloud Storage buckets.\n",
|
||||
"* Create & launch your custom training job with your custom container."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "lCzg9s63M1Oq"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"* Google Artifact Registry\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),[Cloud Storage pricing](https://cloud.google.com/storage/pricing), and [Google Artifact Registry pricing](https://cloud.google.com/artifact-registry/pricing). \n",
|
||||
"\n",
|
||||
"Use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "rAPBk_OCNN_h"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench**, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
|
||||
"\n",
|
||||
"Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
|
||||
"\n",
|
||||
"- The Cloud Storage SDK\n",
|
||||
"- Git\n",
|
||||
"- Python 3\n",
|
||||
"- virtualenv\n",
|
||||
"- Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
|
||||
"\n",
|
||||
"2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
|
||||
"\n",
|
||||
"3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
|
||||
"\n",
|
||||
"4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"6. Open this notebook in the Jupyter Notebook Dashboard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0Edien3sDfJm"
|
||||
},
|
||||
"source": [
|
||||
"### Install additional packages\n",
|
||||
"\n",
|
||||
"Install the following packages required to execute this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "n21c5UHogVEY"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
|
||||
"\n",
|
||||
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_WORKBENCH_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"! pip3 install google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"\n",
|
||||
"# Automatically restart kernel after installs\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "r_dA3M6UJELw"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1Dunp1YrhPYo"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI, Cloud Storage, Cloud Build, and Artifact Registry APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,storage-component.googleapis.com,cloudbuild.googleapis.com,artifactregistry.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"Note: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "lMqIOZfqhXyD"
|
||||
},
|
||||
"source": [
|
||||
"### Set your project ID"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "siN0RvWyZYv9"
|
||||
},
|
||||
"source": [
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "lU-1yD4jZc-6"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"\"\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Get your Google Cloud project ID from gcloud\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID: \", PROJECT_ID)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XNVqyIvYaPO-"
|
||||
},
|
||||
"source": [
|
||||
"Otherwise, set your project ID here."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "s0F0U7WZhfAZ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
|
||||
" PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "K-KuU54IaVz5"
|
||||
},
|
||||
"source": [
|
||||
"### Timestamp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "doEJxrvsaWyt"
|
||||
},
|
||||
"source": [
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "rheJHuCIadAL"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Ee3vBgvdhgTb"
|
||||
},
|
||||
"source": [
|
||||
"### Set your region"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "flcwBahRhi8b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "KuNRbXkIijp6"
|
||||
},
|
||||
"source": [
|
||||
"### Login to your Google Cloud account"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "P9vQxUzfirCV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"IS_COLAB = \"google.colab\" in sys.modules\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if IS_COLAB:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "OG3dstAuVtgz"
|
||||
},
|
||||
"source": [
|
||||
"### Create Cloud Storage bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TKVkz5uzV07p"
|
||||
},
|
||||
"source": [
|
||||
"A Cloud Storage buckets will be used store your training code output (including TensorBoard logs). The bucket must be regional that is, not multi-region or dual-region, and the following resources must be in same region:\n",
|
||||
"\n",
|
||||
"* the Cloud Storage bucket\n",
|
||||
"* the Vertex AI training job\n",
|
||||
"* the Vertex AI TensorBoard instance"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "_BzfwD8EV7Fw"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "iVe3Y5UGWDoC"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket. The created bucket will be deleted in the cleaning up section in the end. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "M0krsPd9WJYl"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "oXo9TNzQWQBS"
|
||||
},
|
||||
"source": [
|
||||
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "BD-n41FzWY0G"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al {BUCKET_URI}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "q73OgpxyDtSk"
|
||||
},
|
||||
"source": [
|
||||
"### Import aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "yO4wki2xD0Ys"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "79WoncsWbLTg"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize Vertex AI SDK for Python\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "y4qLmB4WbLwo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0j1NWIQEJI5i"
|
||||
},
|
||||
"source": [
|
||||
"## Create Docker repository"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hNmHMIyjBzxx"
|
||||
},
|
||||
"source": [
|
||||
"Create a Docker repository named `DOCKER_REPOSITORY` in your `REGION`.\n",
|
||||
"This docker repository will be deleted in the clearning up section in the end."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "iz9CUUbdBTvF"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"DOCKER_REPOSITORY = \"[your-docker-repository-name]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if (\n",
|
||||
" DOCKER_REPOSITORY == \"\"\n",
|
||||
" or DOCKER_REPOSITORY is None\n",
|
||||
" or DOCKER_REPOSITORY == \"[your-docker-repository-name]\"\n",
|
||||
"):\n",
|
||||
" DOCKER_REPOSITORY = \"tb-docker-repo-\" + PROJECT_ID + \"-\" + TIMESTAMP\n",
|
||||
"\n",
|
||||
"print(\"Docker repository to create:\", DOCKER_REPOSITORY)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "R9eQDigtC8XA"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gcloud artifacts repositories create $DOCKER_REPOSITORY --project={PROJECT_ID} \\\n",
|
||||
"--repository-format=docker \\\n",
|
||||
"--location={REGION} --description=\"Repository for TensorBoard Custom Training Job\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "V0pBSC0rDlvq"
|
||||
},
|
||||
"source": [
|
||||
"Verify your Docker repository is created successfully."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "fgHYvZlwDuAN"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gcloud artifacts repositories list --project={PROJECT_ID}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "laaA77LsIt0c"
|
||||
},
|
||||
"source": [
|
||||
"## Create a custom container image and push to Artifact Registry\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AVcNnHfEKBdm"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create a folder for the image.\n",
|
||||
"! mkdir tb-custom-container\n",
|
||||
"%cd tb-custom-container"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "jUcVG77dKmPn"
|
||||
},
|
||||
"source": [
|
||||
"Write your own training code in task.py file. You can use the following code as an example."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7k1JDOGNKL3d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile task.py\n",
|
||||
"\n",
|
||||
"import logging\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import tensorflow as tf\n",
|
||||
"import tensorflow_datasets as tfds\n",
|
||||
"\n",
|
||||
"IMG_WIDTH = 128\n",
|
||||
"\n",
|
||||
"def normalize_img(image):\n",
|
||||
" \"\"\"Normalizes image.\n",
|
||||
"\n",
|
||||
" * Resizes image to IMG_WIDTH x IMG_WIDTH pixels\n",
|
||||
" * Casts values from `uint8` to `float32`\n",
|
||||
" * Scales values from [0, 255] to [0, 1]\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" A tensor with shape (IMG_WIDTH, IMG_WIDTH, 3). (3 color channels)\n",
|
||||
" \"\"\"\n",
|
||||
" image = tf.image.resize_with_pad(image, IMG_WIDTH, IMG_WIDTH)\n",
|
||||
" return image / 255.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def normalize_img_and_label(image, label):\n",
|
||||
" \"\"\"Normalizes image and label.\n",
|
||||
"\n",
|
||||
" * Performs normalize_img on image\n",
|
||||
" * Passes through label unchanged\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" Tuple (image, label) where\n",
|
||||
" * image is a tensor with shape (IMG_WIDTH, IMG_WIDTH, 3). (3 color\n",
|
||||
" channels)\n",
|
||||
" * label is an unchanged integer [0, 4] representing flower type\n",
|
||||
" \"\"\"\n",
|
||||
" return normalize_img(image), label\n",
|
||||
"\n",
|
||||
"logging.info('Loading and preprocessing data ...')\n",
|
||||
"dataset = tfds.load('tf_flowers:3.*.*',\n",
|
||||
" split='train',\n",
|
||||
" try_gcs=True,\n",
|
||||
" shuffle_files=True,\n",
|
||||
" as_supervised=True)\n",
|
||||
"dataset = dataset.map(normalize_img_and_label,\n",
|
||||
" num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
|
||||
"dataset = dataset.cache()\n",
|
||||
"dataset = dataset.shuffle(1000)\n",
|
||||
"dataset = dataset.batch(128)\n",
|
||||
"dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
|
||||
"\n",
|
||||
"logging.info('Creating and training model ...')\n",
|
||||
"\n",
|
||||
"model = tf.keras.Sequential([\n",
|
||||
" tf.keras.layers.Conv2D(16,\n",
|
||||
" 3,\n",
|
||||
" padding='same',\n",
|
||||
" activation='relu',\n",
|
||||
" input_shape=(IMG_WIDTH, IMG_WIDTH, 3)),\n",
|
||||
" tf.keras.layers.MaxPooling2D(),\n",
|
||||
" tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),\n",
|
||||
" tf.keras.layers.MaxPooling2D(),\n",
|
||||
" tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),\n",
|
||||
" tf.keras.layers.MaxPooling2D(),\n",
|
||||
" tf.keras.layers.Flatten(),\n",
|
||||
" tf.keras.layers.Dense(512, activation=\"relu\"),\n",
|
||||
" tf.keras.layers.Dense(5) # 5 classes\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"logging.info('Compiling model ...')\n",
|
||||
"model.compile(\n",
|
||||
" optimizer='adam',\n",
|
||||
" loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
|
||||
" metrics=['accuracy'])\n",
|
||||
"\n",
|
||||
"log_dir = \"logs\"\n",
|
||||
"if 'AIP_TENSORBOARD_LOG_DIR' in os.environ:\n",
|
||||
" log_dir=os.environ['AIP_TENSORBOARD_LOG_DIR']\n",
|
||||
"\n",
|
||||
"tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
|
||||
" log_dir=log_dir,\n",
|
||||
" histogram_freq=1)\n",
|
||||
"\n",
|
||||
"logging.info('Training model ...')\n",
|
||||
"model.fit(dataset, epochs=13, callbacks=[tensorboard_callback])\n",
|
||||
"\n",
|
||||
"logging.info('Model training done')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hvNddrgPP9bs"
|
||||
},
|
||||
"source": [
|
||||
"Create your own `Dockerfile` to specify all instructions needed to build your container. You can use the following `Dockerfile` as an example."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "fxi1HZbKQUKY"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile Dockerfile\n",
|
||||
"\n",
|
||||
"# Specifies base image and tag\n",
|
||||
"FROM us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-8:latest\n",
|
||||
"WORKDIR /root\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Installs additional packages as you need.\n",
|
||||
"\n",
|
||||
"# Copies the trainer code to the docker image.\n",
|
||||
"COPY task.py /root/task.py\n",
|
||||
"\n",
|
||||
"# Sets up the entry point to invoke the trainer.\n",
|
||||
"ENTRYPOINT [\"python\", \"task.py\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "DK2E1xz8Q7Q-"
|
||||
},
|
||||
"source": [
|
||||
"Build your container image using `gcloud builds` from your training code and `Dockerfile`. Note that this step may take a few minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "BvyoFJW_LJeG"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"IMAGE_NAME = \"tensorboard-custom-container\"\n",
|
||||
"IMAGE_TAG = \"v1\"\n",
|
||||
"IMAGE_URI = \"us-central1-docker.pkg.dev/{}/{}/{}:{}\".format(\n",
|
||||
" PROJECT_ID, DOCKER_REPOSITORY, IMAGE_NAME, IMAGE_TAG\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"! gcloud builds submit --project {PROJECT_ID} --region={REGION} --tag {IMAGE_URI} --timeout=20m"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hwXxa4Qgnh4Y"
|
||||
},
|
||||
"source": [
|
||||
"## Setup service account and permissions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "7qXFUiHLoFRw"
|
||||
},
|
||||
"source": [
|
||||
"A service account will be used to create custom training job. If you do not want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID. You can create a service account by following the [instruction](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "mpKjfsXumuNV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "RdWLe8S_tlee"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if (\n",
|
||||
" SERVICE_ACCOUNT == \"\"\n",
|
||||
" or SERVICE_ACCOUNT is None\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
"):\n",
|
||||
" # Get your service account from gcloud\n",
|
||||
" if not IS_COLAB:\n",
|
||||
" shell_output = ! gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
"\n",
|
||||
" else: # IS_COLAB:\n",
|
||||
" shell_output = ! gcloud projects describe $PROJECT_ID\n",
|
||||
" project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
|
||||
" SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
|
||||
"\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "UlDhuciOt5vo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Grant Cloud Storage permission.\n",
|
||||
"! gcloud projects add-iam-policy-binding {PROJECT_ID} \\\n",
|
||||
" --member=serviceAccount:{SERVICE_ACCOUNT} \\\n",
|
||||
" --role=roles/storage.admin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "lTKVB71soRyr"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Grant AI Platform permission.\n",
|
||||
"! gcloud projects add-iam-policy-binding {PROJECT_ID} \\\n",
|
||||
" --member=serviceAccount:{SERVICE_ACCOUNT} \\\n",
|
||||
" --role=roles/aiplatform.user"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "IaQjIPvuKLwW"
|
||||
},
|
||||
"source": [
|
||||
"## Create a custom training job with your container"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "svUGBOow_Obj"
|
||||
},
|
||||
"source": [
|
||||
"Create a TensorBoard instnace to be used by the custom training job."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "OAe1xJeS_X3F"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TENSORBOARD_NAME = \"[your-tensorboard-name]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if (\n",
|
||||
" TENSORBOARD_NAME == \"\"\n",
|
||||
" or TENSORBOARD_NAME is None\n",
|
||||
" or TENSORBOARD_NAME == \"[your-tensorboard-name]\"\n",
|
||||
"):\n",
|
||||
" TENSORBOARD_NAME = PROJECT_ID + \"-tb-\" + TIMESTAMP\n",
|
||||
"\n",
|
||||
"tensorboard = aiplatform.Tensorboard.create(\n",
|
||||
" display_name=TENSORBOARD_NAME, project=PROJECT_ID, location=REGION\n",
|
||||
")\n",
|
||||
"TENSORBOARD_RESOURCE_NAME = tensorboard.gca_resource.name\n",
|
||||
"print(\"TensorBoard resource name:\", TENSORBOARD_RESOURCE_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "mudxBDal_a_k"
|
||||
},
|
||||
"source": [
|
||||
"Run the following example request to create your own custom training job using the container you just built and uploaded to Artifact Registry, and stream the training results to TensorBoard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "wugHEC8czuBe"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"JOB_NAME = \"tensorboard-example-job-{}\".format(TIMESTAMP)\n",
|
||||
"BASE_OUTPUT_DIR = \"{}/{}\".format(BUCKET_URI, JOB_NAME)\n",
|
||||
"\n",
|
||||
"job = aiplatform.CustomContainerTrainingJob(\n",
|
||||
" display_name=JOB_NAME,\n",
|
||||
" container_uri=IMAGE_URI,\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" location=REGION,\n",
|
||||
" staging_bucket=BUCKET_URI,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job.run(\n",
|
||||
" machine_type=\"n1-standard-8\",\n",
|
||||
" replica_count=1,\n",
|
||||
" service_account=SERVICE_ACCOUNT,\n",
|
||||
" tensorboard=TENSORBOARD_RESOURCE_NAME,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "VfMsn_RnEtnj"
|
||||
},
|
||||
"source": [
|
||||
"In Google Cloud console, you can monitor your training job at Vertex AI > Training > Custom Jobs. In each custom training job, near real time updated TensorBoard is available at `OPEN TENSORBOARD` button."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Bmbjz-nUW7KE"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, **if you created the individual resources in the notebook** you can delete them as follow:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-LSKjrRqW-Bk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete GCS bucket.\n",
|
||||
"! gsutil -m rm -r {BUCKET_URI}\n",
|
||||
"\n",
|
||||
"# Delete docker repository.\n",
|
||||
"! gcloud artifacts repositories delete $DOCKER_REPOSITORY --project {PROJECT_ID} --location {REGION} --quiet\n",
|
||||
"\n",
|
||||
"# Delete TensorBoard instance.\n",
|
||||
"! gcloud ai tensorboards delete {TENSORBOARD_RESOURCE_NAME}\n",
|
||||
"\n",
|
||||
"# Delete custom job.\n",
|
||||
"job.delete()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "tensorboard_custom_training_with_custom_container.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
Reference in New Issue
Block a user