xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refresh of PyTorch Image Classification Multi-Node Distributed Data Parallel Training on GPU using Vertex Training with Custom Container (#509)

Browse Source 
* notebook refresh from vertex ai sdk project

* linter test

* notebook refresh from vertex ai sdk project with trainer folder

* linter test

* add pyarrow

* modified notebook

* ran linter test

Co-authored-by: Andrew Ferlitsch <aferlitsch@gmail.com>
Co-authored-by: sudarshan-SpringML <sudarshan.c@springml.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
manuelamunategui
2022-08-03 13:32:44 -07:00
committed by GitHub
co-authored by Andrew Ferlitsch sudarshan-SpringML Andrew Ferlitsch
parent 53f25201cc
commit 5d747ffa0d
1 changed files with 925 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/pytorch_image_classification_distributed_data_parallel_training_with_vertex_sdk/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb
+925
View File
@@ -0,0 +1,925 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a6b56b1c7b76"
},
"outputs": [],
"source": [
"# Copyright 2022 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "20a5ea0081d0"
},
"source": [
"# PyTorch Image Classification Multi-Node Distributed Data Parallel Training on GPU using Vertex Training with Custom Container"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8752d4a255fb"
},
"source": [
" \n",
"<table align=\"left\">\n",
"<td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pytorch_image_classification_distributed_data_parallel_training_with_vertex_sdk/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" \n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pytorch_image_classification_distributed_data_parallel_training_with_vertex_sdk/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td> \n",
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pytorch_image_classification_distributed_data_parallel_training_with_vertex_sdk/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
"Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ab06064bf20c"
},
"source": [
"## Overview\n",
"\n",
"\n",
"This tutorial demonstrates how to create a multi-node, distributed image classification using PyTorch on Vertex SDK with GPU. This can help your training job scale to handle large amounts of data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2ee27912fc7e"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the <a href=\"http://yann.lecun.com/exdb/mnist/\">MNIST database</a>. The MNIST database of handwritten digits has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "968b7ddf103f"
},
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to create a distributed PyTorch training job using Vertex AI SDK for Python and custom containers. You will set up GCP to use a custom container, a Vertex Tensorboard Instance and run a custom training job. \n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI SDK`\n",
"- `Vertex AI TensorBoard`\n",
"- `CustomContainerTrainingJob`\n",
"- `Artifact Registry`\n",
"\n",
"The steps performed include:\n",
"\n",
"- Setting up your GCP project : Setting up the PROJECT_ID, REGION & SERVICE_ACCOUNT\n",
"- Creating a cloud storage bucket\n",
"- Building Custom Container using Artifact Registry and Docker\n",
"- Create a Vertex AI Tensorboard Instance to store your Vertex AI experiment\n",
"- Run a Vertex AI SDK CustomContainerTrainingJob"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7db7019c32bf"
},
"source": [
"## Costs\n",
" \n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"Vertex AI\n",
"\n",
"Cloud Storage\n",
"\n",
"Vertex AI TensorBoard (Note that Vertex AI TensorBoard charges a monthly fee of $300 per unique active user. Active users are measured through the Vertex AI TensorBoard UI. You also pay for Google Cloud resources you use with Vertex AI TensorBoard, such as TensorBoard logs stored in Cloud Storage. <a href='https://cloud.google.com/vertex-ai/pricing#tensorboard'>Check the link for latest prices.</a>)\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/),\n",
" to generate a cost estimate based on your projected usage.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a25595b8a10e"
},
"source": [
"### Install additional packages\n",
"\n",
"Install the latest version of Vertex AI SDK for Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "6d015287b38d"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# The Google Cloud Notebook product has specific requirements\n",
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
"\n",
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
" USER_FLAG = \"--user\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "4acbeefa0e09"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: google-cloud-aiplatform in /home/jupyter/.local/lib/python3.7/site-packages (1.16.0)\n",
"Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.34.4)\n",
"Requirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.8.1)\n",
"Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /home/jupyter/.local/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.44.0)\n",
"Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.5.1)\n",
"Requirement already satisfied: proto-plus<2.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.20.6)\n",
"Requirement already satisfied: protobuf<4.0.0dev,>=3.19.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (3.19.4)\n",
"Requirement already satisfied: packaging<22.0.0dev,>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (21.3)\n",
"Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /home/jupyter/.local/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.35.0)\n",
"Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.56.3)\n",
"Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.28.1)\n",
"Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.47.0)\n",
"Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.47.0)\n",
"Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.1)\n",
"Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
"Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.3)\n",
"Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /opt/conda/lib/python3.7/site-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.12.4)\n",
"Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.16.0)\n",
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<22.0.0dev,>=14.3->google-cloud-aiplatform) (3.0.9)\n",
"Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (4.8)\n",
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.2.7)\n",
"Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (59.8.0)\n",
"Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/jupyter/.local/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (4.2.4)\n",
"Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.1.2)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.26.9)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (3.3)\n",
"Requirement already satisfied: charset-normalizer<3,>=2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.1.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2022.6.15)\n",
"Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.15.0)\n",
"Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.4.8)\n",
"Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.21)\n",
"Requirement already satisfied: pyarrow in /opt/conda/lib/python3.7/site-packages (8.0.0)\n",
"Requirement already satisfied: numpy>=1.16.6 in /opt/conda/lib/python3.7/site-packages (from pyarrow) (1.19.5)\n"
]
}
],
"source": [
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform\n",
"! pip3 install {USER_FLAG} --upgrade pyarrow"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "35942e320683"
},
"source": [
"### Restart the kernel\n",
"\n",
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "a7e5937088c8"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "70b5da791577"
},
"source": [
"# Setup\n",
"\n",
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`.! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "03d216c7f7b1"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "5bf9979b96ff"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Project ID: vertex-ai-dev\n"
]
}
],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "09021c90b34c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Updated property [core/project].\n"
]
}
],
"source": [
"! gcloud config set project $PROJECT_ID"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "88dd74c4c84e"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
"- Asia Pacific: `asia-east1`\n",
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "5c615e53149f"
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
"\n",
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "870777863e09"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Google Cloud Notebooks**, your environment is already\n",
"authenticated. Skip this step."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "378e70541ba9"
},
"source": [
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"2. Click **Create service account**.\n",
"\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "f81b13f6ed38"
},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"\n",
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"# The Google Cloud Notebook product has specific requirements\n",
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
"\n",
"# If on Google Cloud Notebooks, then don't execute this code\n",
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d98e540b8e3f"
},
"source": [
"#### Service Account\n",
"If you don't know your service account, try to get your service account using gcloud command by executing the second cell below."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "a9db65cc849b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Service Account: 931647533046-compute@developer.gserviceaccount.com\n"
]
}
],
"source": [
"SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}\n",
"\n",
"if (\n",
" SERVICE_ACCOUNT == \"\"\n",
" or SERVICE_ACCOUNT is None\n",
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
"):\n",
" # Get your GCP project id from gcloud\n",
" shell_output = !gcloud auth list 2>/dev/null\n",
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
" print(\"Service Account:\", SERVICE_ACCOUNT)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "454cb4f7a9e9"
},
"source": [
"#### Timestamp\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "a8c17026c384"
},
"outputs": [],
"source": [
"from datetime import datetime\n",
"\n",
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "37aa2089e5a5"
},
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"\n",
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "c5ac73516218"
},
"outputs": [],
"source": [
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "6f9da502010b"
},
"outputs": [],
"source": [
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "15bb44ff961c"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "fde15c57652b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating gs://vertex-ai-devaip-20220803062855/...\n"
]
}
],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1bba6423a764"
},
"source": [
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "0b5ae674177e"
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d48860504181"
},
"source": [
"### Import libraries and define constants"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "40b9227cb6a1"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"content_name = \"pt-img-cls-multi-node-ddp-cust-cont\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5307fe28b633"
},
"source": [
"## Vertex AI Training using a custom container"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "46cb58c7fbf9"
},
"source": [
"### Built Custom Container"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "97e66e9f9bab"
},
"outputs": [],
"source": [
"PRIVATE_REPO = \"my-docker-repo\"\n",
"CUSTOM_CONTAINER_IMAGE_URI = (\n",
" f\"{REGION}-docker.pkg.dev/\" + PROJECT_ID + f\"/{PRIVATE_REPO}\" + \"/tf_serving\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ae9b29c4773f"
},
"source": [
"### Initialize Vertex AI SDK"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "6964be27b98e"
},
"outputs": [],
"source": [
"aiplatform.init(\n",
" project=PROJECT_ID,\n",
" staging_bucket=BUCKET_URI,\n",
" location=REGION,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "594a91f438f2"
},
"source": [
"### Create a Vertex AI TensorBoard Instance\n",
"\n",
"NOTE: <a href=\"https://cloud.google.com/vertex-ai/pricing#tensorboard\">Vertex AI TensorBoard </a> charges a monthly fee of $300 per unique active user. Active users are measured through the Vertex AI TensorBoard UI. You also pay for Google Cloud resources you use with Vertex AI TensorBoard, such as TensorBoard logs stored in Cloud Storage.</a>Please check above link for latest prices."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"id": "c2bd82dbcd9b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating Tensorboard\n",
"Create Tensorboard backing LRO: projects/931647533046/locations/us-central1/tensorboards/1159025993164259328/operations/3994130263121592320\n",
"Tensorboard created. Resource name: projects/931647533046/locations/us-central1/tensorboards/1159025993164259328\n",
"To use this Tensorboard in another session:\n",
"tb = aiplatform.Tensorboard('projects/931647533046/locations/us-central1/tensorboards/1159025993164259328')\n"
]
}
],
"source": [
"content_name = content_name + \"-gpu\"\n",
"\n",
"tensorboard = aiplatform.Tensorboard.create(\n",
" display_name=content_name,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ebc593c6472e"
},
"source": [
"#### Option: Use a previously created Vertex AI TensorBoard instance\n",
"\n",
"```\n",
"tensorboard_name = \"Your Tensorboard Resource Name or Tensorboard ID\"\n",
"tensorboard = aiplatform.Tensorboard(tensorboard_name=tensorboard_name)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0769e8e34c2f"
},
"source": [
"### Run a Vertex AI SDK CustomContainerTrainingJob"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "023f33ece826"
},
"outputs": [],
"source": [
"display_name = content_name\n",
"gcs_output_uri_prefix = f\"{BUCKET_URI}/{display_name}\"\n",
"\n",
"replica_count = 1\n",
"machine_type = \"n1-standard-4\"\n",
"accelerator_count = 4\n",
"accelerator_type = \"NVIDIA_TESLA_K80\"\n",
"\n",
"args = [\n",
" \"--backend\",\n",
" \"nccl\",\n",
" \"--batch-size\",\n",
" \"128\",\n",
" \"--epochs\",\n",
" \"25\",\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"id": "d4b599e726ef"
},
"outputs": [],
"source": [
"custom_container_training_job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=display_name,\n",
" container_uri=CUSTOM_CONTAINER_IMAGE_URI,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a788686d460b"
},
"source": [
"You can use the Vertex AI Tensorboard `custom_container_training_job` link from the output to view the Accuracy and Loss of model"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"id": "81321e3bdf7f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Output directory:\n",
"gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu \n",
"View Training:\n",
"https://console.cloud.google.com/ai/platform/locations/us-central1/training/7221734483110658048?project=931647533046\n",
"View backing custom job:\n",
"https://console.cloud.google.com/ai/platform/locations/us-central1/training/7880456297357770752?project=931647533046\n",
"View tensorboard:\n",
"https://us-central1.tensorboard.googleusercontent.com/experiment/projects+931647533046+locations+us-central1+tensorboards+1159025993164259328+experiments+7880456297357770752\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 current state:\n",
"PipelineState.PIPELINE_STATE_RUNNING\n",
"CustomContainerTrainingJob run completed. Resource name: projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048\n",
"Training did not produce a Managed Model returning None. Training Pipeline projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048 is not configured to upload a Model. Create the Training Pipeline with model_serving_container_image_uri and model_display_name passed in. Ensure that your training script saves to model to os.environ['AIP_MODEL_DIR'].\n"
]
}
],
"source": [
"custom_container_training_job.run(\n",
" args=args,\n",
" base_output_dir=gcs_output_uri_prefix,\n",
" replica_count=replica_count,\n",
" machine_type=machine_type,\n",
" accelerator_count=accelerator_count,\n",
" accelerator_type=accelerator_type,\n",
" tensorboard=tensorboard.resource_name,\n",
" service_account=SERVICE_ACCOUNT,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "5100712c2c4c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Custom Training Job Name: projects/931647533046/locations/us-central1/trainingPipelines/7221734483110658048\n",
"GCS Output URI Prefix: gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu\n"
]
}
],
"source": [
"print(f\"Custom Training Job Name: {custom_container_training_job.resource_name}\")\n",
"print(f\"GCS Output URI Prefix: {gcs_output_uri_prefix}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9b77676e5a6"
},
"source": [
"### View output artifact from training"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "0e171ce95ace"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/\n",
"gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/checkpoints/\n",
"gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/logs/\n"
]
}
],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cf1b74a12b87"
},
"source": [
"# Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
"- Artifacts\n",
"- Cloud Storage Bucket"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"id": "a0b15089c341"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removing gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/#1659508605105683...\n",
"Removing gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/checkpoints/#1659508606162102...\n",
"Removing gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/logs/#1659508605385570...\n",
"Removing gs://vertex-ai-devaip-20220803062855/pt-img-cls-multi-node-ddp-cust-cont-gpu/logs/events.out.tfevents.1659508605.bfb4bda22390.1.0#1659508943933410...\n",
"/ [4 objects] \n",
"Operation completed over 4 objects. \n",
"Removing gs://vertex-ai-devaip-20220803062855/...\n"
]
}
],
"source": [
"delete_bucket = False\n",
"delete_tensorboard = False\n",
"\n",
"! gsutil rm -rf $gcs_output_uri_prefix\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI\n",
"\n",
"if delete_tensorboard or os.getenv(\"IS_TESTING\"):\n",
" tensorboard.delete()"
]
}
],
"metadata": {
"colab": {
"name": "multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}