xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore, refactor, fix, feat(egen): template structure+K80GPU+contract words+clean up (#3085)

Browse Source 
* chore, refactor, fix, feat: template structure, removes redundant code, contracts content, replaces K80 with T4, cleans up local files

* chore,chore, fix: addresses review comments+ sets replica=1,accelerator_count=4

* fix, chore: reduces the GPU count to 1, rewords the title

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
This commit is contained in:
Krishna Chaithanya Movva
2024-06-14 21:12:17 +00:00
committed by GitHub
co-authored by krishr2d2
parent d7c334353c
commit 7cd354436e
1 changed files with 231 additions and 350 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb
+231 -350
View File
@@ -29,27 +29,29 @@
"id": "c414a395a19b"
},
"source": [
"# PyTorch image classification multi-node NCCL distributed data parallel training on cpu using Vertex training with custom container\n",
"# PyTorch image classification using multi-node NCCL distributed data parallel training on CPU and Vertex AI\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Ftraining%2Fmulti_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td> \n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
@@ -77,18 +79,16 @@
"\n",
"In this tutorial, you learn how to create a distributed PyTorch training job using Vertex AI SDK for Python and custom containers. You set up a custom container, a Vertex AI TensorBoard instance and run a custom training job. \n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"This tutorial uses the following Vertex AI services:\n",
"\n",
"- `Vertex AI SDK`\n",
"- `Vertex AI TensorBoard`\n",
"- `CustomContainerTrainingJob`\n",
"- `Artifact Registry`\n",
"- Vertex AI TensorBoard\n",
"- Vertex AI Training\n",
"\n",
"The steps performed include:\n",
"\n",
"- Building Custom Container using Artifact Registry and Docker\n",
"- Create a Vertex AI tensorboard instance to store your Vertex AI experiment\n",
"- Run a Vertex AI SDK CustomContainerTrainingJob"
"- Building a custom container using Artifact Registry and Docker.\n",
"- Creating a Vertex AI tensorboard instance to store your Vertex AI experiment.\n",
"- Run a Vertex AI training job using Vertex AI SDK for Python."
]
},
{
@@ -99,7 +99,7 @@
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the <a href=\"http://yann.lecun.com/exdb/mnist/\">MNIST database</a>. The MNIST database of handwritten digits has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.\n"
"The dataset used for this tutorial is the <a href=\"http://yann.lecun.com/exdb/mnist/\">MNIST database</a>. The MNIST database of handwritten digits has a training set of 60,000 examples, and a test set of 10,000 examples. This dataset is a subset of a larger dataset constructed from binary images of handwritten digits. The digits have been size-normalized and centered in a fixed-size image.\n"
]
},
{
@@ -122,15 +122,22 @@
" to generate a cost estimate based on your projected usage.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f0316df526f8"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "03d216c7f7b1"
},
"source": [
"## Installation\n",
"\n",
"Install the following packages required to execute this notebook. "
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
@@ -141,7 +148,13 @@
},
"outputs": [],
"source": [
"! pip3 install --quiet --upgrade google-cloud-aiplatform"
"# Install the Vertex AI SDK for Python\n",
"! pip3 install --quiet --upgrade google-cloud-aiplatform\n",
"\n",
"# Install other required libraries\n",
"! pip3 install --quiet torch \\\n",
" torchvision \\\n",
" tensorboard"
]
},
{
@@ -150,7 +163,9 @@
"id": "35942e320683"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel."
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
@@ -161,32 +176,53 @@
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0e3cab0cc491"
"id": "ee775571c2b5"
},
"source": [
"## Before you begin\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "92e68cfc3a90"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"### Set up your Google Cloud project\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "46604f70e831"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
" from google.colab import auth\n",
"\n",
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
"\n",
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
" auth.authenticate_user()"
]
},
{
@@ -195,12 +231,9 @@
"id": "80f70e8aa911"
},
"source": [
"#### Set your project ID\n",
"### Set Google Cloud project information\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
" Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -212,123 +245,7 @@
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "88dd74c4c84e"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5c615e53149f"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "070f83c35863"
},
"source": [
"#### UUID\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e87d5856317d"
},
"outputs": [],
"source": [
"import random\n",
"import string\n",
"\n",
"\n",
"# Generate a uuid of a specifed length(default=8)\n",
"def generate_uuid(length: int = 8) -> str:\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"To authenticate your Google Cloud account, follow the instructions for your Jupyter environment:\n",
"\n",
"**1. Vertex AI Workbench**\n",
"<br>You are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "254614fa0c46"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ef21552ccea8"
},
"source": [
"**3. Colab**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "603adbbf0532"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
]
},
{
@@ -359,7 +276,7 @@
"id": "create_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
@@ -370,7 +287,31 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "306bf9ab7f7f"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and enable the [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4dc6b3ba241c"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI, location=LOCATION)"
]
},
{
@@ -379,9 +320,9 @@
"id": "05d881f62170"
},
"source": [
"#### Service Account \n",
"### Service Account \n",
"\n",
"You use a service account to run Vetex AI CustomContainerTrainingJob. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
"You use a service account to run Vetex AI CustomContainerTrainingJob. If you don't want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
]
},
{
@@ -405,7 +346,7 @@
"source": [
"import sys\n",
"\n",
"IS_COLAB = 'google.colab' in sys.modules\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if (\n",
" SERVICE_ACCOUNT == \"\"\n",
" or SERVICE_ACCOUNT is None\n",
@@ -427,25 +368,30 @@
{
"cell_type": "markdown",
"metadata": {
"id": "d48860504181"
"id": "2d242773d707"
},
"source": [
"### Import libraries and define constants"
"### Enable Artifact Registry API\n",
"You must enable the [Artifact Registry API](https://cloud.google.com/artifact-registry/docs/reference/rest) service for your project.\n",
"\n",
"<a href=\"https://cloud.google.com/artifact-registry/docs/enable-service\">Learn more about Enabling service</a>."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "40b9227cb6a1"
"id": "00ceac726715"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"! gcloud services enable artifactregistry.googleapis.com\n",
"\n",
"content_name = \"pt-img-cls-multi-node-ddp-cust-cont\""
"if os.getenv(\"IS_TESTING\"):\n",
" ! sudo apt-get update --yes && sudo apt-get --only-upgrade --yes install google-cloud-sdk-cloud-run-proxy google-cloud-sdk-harbourbridge google-cloud-sdk-cbt google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-kpt google-cloud-sdk-local-extract google-cloud-sdk-minikube google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-python google-cloud-sdk-spanner-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-nomos google-cloud-sdk-package-go-module google-cloud-sdk-firestore-emulator kubectl google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-kubectl-oidc google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-grpc google-cloud-sdk-pubsub-emulator google-cloud-sdk-datalab google-cloud-sdk-skaffold google-cloud-sdk google-cloud-sdk-terraform-tools google-cloud-sdk-config-connector\n",
" ! gcloud components update --quiet"
]
},
{
@@ -454,11 +400,9 @@
"id": "12c03eca5908"
},
"source": [
"# Create Custom Training Python Package\n",
"## Create training application\n",
"\n",
"Before you can perform local training, you must create source code file, requirements file, docker file.\n",
"\n",
"You will create a directory and write all of our files into that folder."
"Before you can perform local training, create a source code file, a requirements file, and a docker file for your training application."
]
},
{
@@ -469,17 +413,9 @@
},
"outputs": [],
"source": [
"PYTHON_PACKAGE_APPLICATION_DIR = \"trainer\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5bc3955a567b"
},
"outputs": [],
"source": [
"PYTHON_PACKAGE_APPLICATION_DIR = \"trainer\"\n",
"\n",
"# Create the app folder\n",
"!mkdir -p $PYTHON_PACKAGE_APPLICATION_DIR"
]
},
@@ -489,7 +425,9 @@
"id": "436cc7d0197e"
},
"source": [
"### Write the Training Script"
"### Create training script\n",
"\n",
"Create the training file that runs the PyTorch distributed training."
]
},
{
@@ -850,7 +788,7 @@
"id": "253359c5153a"
},
"source": [
"### Write requirements file"
"### Create requirements file"
]
},
{
@@ -863,7 +801,6 @@
"source": [
"%%writefile {PYTHON_PACKAGE_APPLICATION_DIR}/requirements.txt\n",
"\n",
"\n",
"torch\n",
"torchvision\n",
"tensorboard\n"
@@ -875,7 +812,7 @@
"id": "34bea792269a"
},
"source": [
"### Write the docker file"
"### Create Dockerfile"
]
},
{
@@ -913,21 +850,9 @@
"id": "57bf6f8b4361"
},
"source": [
"## Local Training\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e5d8a3443da0"
},
"outputs": [],
"source": [
"! ls trainer\n",
"! cat trainer/requirements.txt\n",
"! pip install -r trainer/requirements.txt\n",
"! cat trainer/task.py"
"## Train locally\n",
"\n",
"Run your training script locally to test your application."
]
},
{
@@ -949,72 +874,17 @@
},
"outputs": [],
"source": [
"# List the outputs\n",
"! ls ./tmp"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3614fcdfcd62"
},
"source": [
"Clean up temporary files"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "48d56ec621cc"
},
"outputs": [],
"source": [
"! rm -rf ./tmp"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8f3ea1210749"
},
"source": [
"## Vertex AI Training using a custom container"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "93002a20a2a6"
},
"source": [
"### Build Custom Container\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2d242773d707"
},
"source": [
"#### Enable Artifact Registry API\n",
"You must enable the Artifact Registry API service for your project.\n",
"\n",
"<a href=\"https://cloud.google.com/artifact-registry/docs/enable-service\">Learn more about Enabling service</a>."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9d72f89cabd5"
},
"outputs": [],
"source": [
"! gcloud services enable artifactregistry.googleapis.com\n",
"\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! sudo apt-get update --yes && sudo apt-get --only-upgrade --yes install google-cloud-sdk-cloud-run-proxy google-cloud-sdk-harbourbridge google-cloud-sdk-cbt google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-kpt google-cloud-sdk-local-extract google-cloud-sdk-minikube google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-python google-cloud-sdk-spanner-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-nomos google-cloud-sdk-package-go-module google-cloud-sdk-firestore-emulator kubectl google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-kubectl-oidc google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-grpc google-cloud-sdk-pubsub-emulator google-cloud-sdk-datalab google-cloud-sdk-skaffold google-cloud-sdk google-cloud-sdk-terraform-tools google-cloud-sdk-config-connector\n",
" ! gcloud components update --quiet"
"## Create a custom image for training"
]
},
{
@@ -1024,11 +894,12 @@
},
"source": [
"### Create a private Docker repository\n",
"Your first step is to create your own Docker repository in Google Artifact Registry.\n",
"\n",
"1 - Run the gcloud artifacts repositories create command to create a new Docker repository with your region with the description \"docker repository\".\n",
"Your first step is to create your own Docker repository in the Artifact Registry.\n",
"\n",
"2 - Run the gcloud artifacts repositories list command to verify that your repository was created."
"1. Run the `gcloud artifacts repositories create` command to create a new Docker repository with your region with the description \"docker repository\".\n",
"\n",
"2. Run the `gcloud artifacts repositories list` command to verify that your repository was created."
]
},
{
@@ -1039,10 +910,13 @@
},
"outputs": [],
"source": [
"PRIVATE_REPO = \"my-docker-repo\"\n",
"# Set the name for your private repo\n",
"PRIVATE_REPO = \"my-docker-repo-unique\"\n",
"\n",
"! gcloud artifacts repositories create {PRIVATE_REPO} --repository-format=docker --location={REGION} --description=\"Docker repository\"\n",
"# Create the repository\n",
"! gcloud artifacts repositories create {PRIVATE_REPO} --repository-format=docker --location={LOCATION} --description=\"Docker repository\"\n",
"\n",
"# List the repositories and verify\n",
"! gcloud artifacts repositories list"
]
},
@@ -1055,7 +929,7 @@
"outputs": [],
"source": [
"DEPLOY_IMAGE = (\n",
" f\"{REGION}-docker.pkg.dev/\" + PROJECT_ID + f\"/{PRIVATE_REPO}\" + \"/tf_serving\"\n",
" f\"{LOCATION}-docker.pkg.dev/\" + PROJECT_ID + f\"/{PRIVATE_REPO}\" + \"/tf_serving\"\n",
")"
]
},
@@ -1076,7 +950,11 @@
"id": "535932b8c42d"
},
"source": [
"## Executes in Workbench\n"
"## Executes in a notebook environment\n",
"\n",
"If you're running this tutorial in a notebook environment, for example Vertex AI workbench, proceed with the below steps. \n",
"\n",
"If you're running on Colab, skip to the \"[Executes in Colab](#Executes-in-Colab)\" section."
]
},
{
@@ -1098,7 +976,7 @@
"outputs": [],
"source": [
"if not IS_COLAB:\n",
" ! gcloud auth configure-docker {REGION}-docker.pkg.dev --quiet"
" ! gcloud auth configure-docker {LOCATION}-docker.pkg.dev --quiet"
]
},
{
@@ -1107,12 +985,13 @@
"id": "a822a118c210"
},
"source": [
"### Container (Docker) image for serving\n",
"Set the TensorFlow Serving Docker container image for serving prediction.\n",
"### Build and push the image for serving\n",
"Set the TensorFlow Serving image for serving predictions.\n",
"\n",
"1. Pull the corresponding CPU or GPU Docker image for TF Serving from Docker Hub.\n",
"2. Create a tag for registering the image with Artifact Registry\n",
"3. Register the image with Artifact Registry.\n",
"The following code:\n",
"1. Pulls the corresponding CPU or GPU Docker image for TF Serving from Docker Hub.\n",
"2. Creates a tag for registering the image with Artifact Registry\n",
"3. Registers the image with Artifact Registry.\n",
"\n",
"<a href=\"https://www.tensorflow.org/tfx/serving/docker\">Learn more about TensorFlow Serving</a>."
]
@@ -1139,7 +1018,7 @@
"source": [
"## Executes in Colab\n",
"\n",
"Build and push a Docker image with Cloud Build"
"Build and push a Docker image with Cloud Build."
]
},
{
@@ -1151,31 +1030,7 @@
"outputs": [],
"source": [
"if IS_COLAB:\n",
" ! cd trainer && gcloud builds submit --timeout=1800s --region={REGION} --tag $DEPLOY_IMAGE"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "10c8cc6b3334"
},
"source": [
"### Initialize Vertex AI SDK"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "42e981cefe41"
},
"outputs": [],
"source": [
"aiplatform.init(\n",
" project=PROJECT_ID,\n",
" staging_bucket=BUCKET_URI,\n",
" location=REGION,\n",
")"
" ! cd trainer && gcloud builds submit --timeout=1800s --region={LOCATION} --tag $DEPLOY_IMAGE"
]
},
{
@@ -1184,9 +1039,16 @@
"id": "73c92c9298e9"
},
"source": [
"### Create a Vertex AI Tensorboard instance\n",
"## Create a Vertex AI Tensorboard instance\n",
"\n",
"NOTE: <a href=\"https://cloud.google.com/vertex-ai/pricing#tensorboard\">Vertex AI TensorBoard </a> charges a monthly fee of $300 per unique active user. Active users are measured through the Vertex AI TensorBoard UI. You also pay for Google Cloud resources you use with Vertex AI TensorBoard, such as TensorBoard logs stored in Cloud Storage.</a>Please check above link for latest prices."
"Set a display name for your Tensorboard instance below and run the create command.\n",
"\n",
"**Note:** To use an existing Vertex AI Tensorboard instance, use the following method.\n",
"\n",
"```\n",
"tensorboard_name = \"Your Tensorboard resource name or Tensorboard instance ID\"\n",
"tensorboard = aiplatform.Tensorboard(tensorboard_name=tensorboard_name)\n",
"```"
]
},
{
@@ -1197,7 +1059,8 @@
},
"outputs": [],
"source": [
"content_name = content_name + \"-cpu\" + \"_\" + UUID"
"# Set display name for Tensorboard instance\n",
"TENSORBOARD_DISPLAY_NAME = \"pytorch-image-classify-multi-node-ddp\""
]
},
{
@@ -1208,32 +1071,21 @@
},
"outputs": [],
"source": [
"# Create the Tensorboard instance\n",
"tensorboard = aiplatform.Tensorboard.create(\n",
" display_name=content_name,\n",
" display_name=TENSORBOARD_DISPLAY_NAME,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a1f0a4f54037"
},
"source": [
"#### Option: Use a previously created Vertex AI Tensorboard instance\n",
"\n",
"```\n",
"tensorboard_name = \"Your Tensorboard Resource Name or Tensorboard ID\"\n",
"tensorboard = aiplatform.Tensorboard(tensorboard_name=tensorboard_name)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a4cac84e04ac"
},
"source": [
"### Run a Vertex AI SDK CustomContainerTrainingJob"
"## Create a Vertex AI custom container training job\n",
"\n",
"Learn more about [configuring compute resources for custom training](https://cloud.google.com/vertex-ai/docs/training/configure-compute)."
]
},
{
@@ -1244,15 +1096,20 @@
},
"outputs": [],
"source": [
"display_name = content_name\n",
"gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
"\n",
"replica_count = 1\n",
"machine_type = \"n1-standard-4\"\n",
"accelerator_count = 4\n",
"accelerator_type = \"NVIDIA_TESLA_K80\"\n",
"\n",
"args = [\n",
"# Set display name for your training job\n",
"JOB_DISPLAY_NAME = \"pytorch-image-classify-multi-node-ddp-training\"\n",
"# Set a Cloud Storage output path\n",
"GCS_OUTPUT_URI_PREFIX = f\"{BUCKET_URI}/{JOB_DISPLAY_NAME}\"\n",
"# Set the number of replicas in the workerpool\n",
"REPLICA_COUNT = 1\n",
"# Set the machine type for running the training job\n",
"MACHINE_TYPE = \"n1-standard-4\"\n",
"# Set the number of accelerators needed for training\n",
"ACCELERATOR_COUNT = 1\n",
"# Set the accelerator type\n",
"ACCELERATOR_TYPE = \"NVIDIA_TESLA_T4\"\n",
"# Set the args to be passed to the training application\n",
"ARGS = [\n",
" \"--backend\",\n",
" \"nccl\",\n",
" \"--no-cuda\",\n",
@@ -1271,12 +1128,22 @@
},
"outputs": [],
"source": [
"# Create the custom container training job\n",
"custom_container_training_job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=display_name,\n",
" display_name=JOB_DISPLAY_NAME,\n",
" container_uri=DEPLOY_IMAGE,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "24e0350a3c68"
},
"source": [
"## Run the training job"
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -1286,12 +1153,12 @@
"outputs": [],
"source": [
"custom_container_training_job.run(\n",
" args=args,\n",
" base_output_dir=gcs_output_uri_prefix,\n",
" replica_count=replica_count,\n",
" machine_type=machine_type,\n",
" accelerator_count=accelerator_count,\n",
" accelerator_type=accelerator_type,\n",
" args=ARGS,\n",
" base_output_dir=GCS_OUTPUT_URI_PREFIX,\n",
" replica_count=REPLICA_COUNT,\n",
" machine_type=MACHINE_TYPE,\n",
" accelerator_count=ACCELERATOR_COUNT,\n",
" accelerator_type=ACCELERATOR_TYPE,\n",
" tensorboard=tensorboard.resource_name,\n",
" service_account=SERVICE_ACCOUNT,\n",
")"
@@ -1306,7 +1173,7 @@
"outputs": [],
"source": [
"print(f\"Custom Training Job Name: {custom_container_training_job.resource_name}\")\n",
"print(f\"GCS Output URI Prefix: {gcs_output_uri_prefix}\")"
"print(f\"GCS Output URI Prefix: {GCS_OUTPUT_URI_PREFIX}\")"
]
},
{
@@ -1315,7 +1182,7 @@
"id": "78398f52807b"
},
"source": [
"### View training output artifact"
"## View training output artifacts"
]
},
{
@@ -1326,7 +1193,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gsutil ls $GCS_OUTPUT_URI_PREFIX"
]
},
{
@@ -1335,15 +1202,17 @@
"id": "5e99a6a05b10"
},
"source": [
"# Cleaning up\n",
"## Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
"- Vertex AI Training job\n",
"- Vertex AI Tensorboard\n",
"- Cloud Storage Bucket"
"- Cloud Storage bucket\n",
"- Artifact Registry repository"
]
},
{
@@ -1354,17 +1223,29 @@
},
"outputs": [],
"source": [
"# Set this to true only if you'd like to delete your bucket\n",
"delete_bucket = False\n",
"delete_tensorboard = False\n",
"# Delete the training job\n",
"custom_container_training_job.delete()\n",
"\n",
"! gsutil rm -rf $gcs_output_uri_prefix\n",
"# Delete the Tensorboard instance\n",
"delete_tensorboard = False # Set True for deletion\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
"if delete_tensorboard:\n",
" tensorboard.delete()\n",
"\n",
"# Delete the Cloud Storage bucket\n",
"delete_bucket = False # Set True for deletion\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI\n",
"\n",
"if delete_tensorboard or os.getenv(\"IS_TESTING\"):\n",
" tensorboard.delete()"
"# Delete the repository in Artifact Registry\n",
"! gcloud artifacts repositories delete {PRIVATE_REPO} --location {LOCATION} --quiet\n",
"\n",
"# Remove the outputs from the local execution\n",
"! rm -rf ./tmp\n",
"\n",
"# Remove the training app folder\n",
"! rm -rf ./trainer"
]
}
],