mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
fix: improved notebook comments and readme (#1131)
* fix: improved notebook comments * fix: improved notebook comments and formatting
This commit is contained in:
gericdong
@@ -1,10 +1,11 @@
|
||||
|
||||
[PyTorch distributed training with Vertex AI Reduction Server](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/reduction_server/pytorch_distributed_training_reduction_server.ipynb)
|
||||
|
||||
Learn how to create a Python source distribution with the training code and dependencies to use with a pre-built containers on Vertex AI.
|
||||
Learn how to create a PyTorch distributed training job that uses PyTorch distributed training framework and tools, and run the training job on the Vertex AI Training service with Reduction Server.
|
||||
|
||||
The steps performed include:
|
||||
|
||||
* Create a custom training job that uses Reduction Server
|
||||
* Submit and monitor the job
|
||||
|
||||
* Create a PyTorch distributed training application
|
||||
* Package the training application with pre-built containers
|
||||
* Create a custom job on Vertex AI with Reduction Server
|
||||
* Submit and monitor the job
|
||||
|
||||
+13
-11
@@ -61,7 +61,7 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"When you train a large ML model using multiple nodes, communicating gradients between nodes can contribute significant latency. Reduction Server is an all-reduce algorithm that can increase throughput and reduce latency for distributed training. This notebook demonstrates how to run a PyTorch distributed training job with Reduction Server on Vertex AI. The training job is created to fine-tune pretrained model `bert-large-cased` from the Hugging Face Transformers library on the `imdb` dataset for sentiment classification."
|
||||
"When you run a distributed training job across multiple nodes using GPUs, communicating gradients between nodes can contribute significant latency. Reduction Server is an all-reduce algorithm that can increase throughput and reduce latency for distributed training. This notebook demonstrates how to run a PyTorch distributed training job with Reduction Server on Vertex AI. The training job is created to fine-tune pretrained model `bert-large-cased` from the Hugging Face Transformers library on the `imdb` dataset for sentiment classification."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -72,7 +72,7 @@
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this notebook, you learn how to create a Python source distribution with the training code and dependencies to use with a pre-built containers on Vertex AI. You learn how to configure, submit, and monitor a Vertex AI Training job that uses PyTorch distributed training framework and tools with Reduction Server to optimize network bandwith and latency of the gradient reduction operation in distributed training.\n",
|
||||
"In this notebook, you learn how to create a PyTorch distributed training job that uses PyTorch distributed training framework and tools, and run the training job on the Vertex AI Training service with Reduction Server.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services and resources:\n",
|
||||
"\n",
|
||||
@@ -82,8 +82,10 @@
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"* Create a custom training job that uses Reduction Server\n",
|
||||
"* Submit and monitor the job\n"
|
||||
"* Create a PyTorch distributed training application\n",
|
||||
"* Package the training application with pre-built containers\n",
|
||||
"* Create a custom job on Vertex AI with Reduction Server\n",
|
||||
"* Submit and monitor the job \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -357,7 +359,7 @@
|
||||
"import string\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Generate a uuid of a specifed length(default=8)\n",
|
||||
"# Generate a uuid of a specified length(default=8)\n",
|
||||
"def generate_uuid(length: int = 8) -> str:\n",
|
||||
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
|
||||
"\n",
|
||||
@@ -606,11 +608,11 @@
|
||||
"source": [
|
||||
"### Packaging the training application\n",
|
||||
"\n",
|
||||
"Before running the training job on Vertex AI, the training application code and any dependencies must be packaged and uploaded to Cloud Storage bucket or Container Registry or Artifact Registry that your Google Cloud project can access. This sections shows how to package and stage your application in the cloud.\n",
|
||||
"Before running the training job on Vertex AI, the training application code and any dependencies must be packaged and uploaded to Cloud Storage bucket or Container Registry or Artifact Registry that your Google Cloud project can access. This section shows how to package and stage your application in the cloud.\n",
|
||||
"\n",
|
||||
"There are two ways to package your application and dependencies and train on Vertex AI:\n",
|
||||
"\n",
|
||||
"1. [Create a Python source distribution](https://cloud.google.com/vertex-ai/docs/training/create-python-pre-built-container) with the training code and dependencies to use with a [pre-built containers](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) on Vertex AI\n",
|
||||
"1. [Create a Python source distribution](https://cloud.google.com/vertex-ai/docs/training/create-python-pre-built-container) with the training code and dependencies to use with a [pre-built container](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) on Vertex AI\n",
|
||||
"2. Use [custom containers](https://cloud.google.com/ai-platform/training/docs/custom-containers-training) to package dependencies using Docker containers\n",
|
||||
"\n",
|
||||
"**This notebook shows the Python source distribution option to run a custom training job on Vertex AI.**"
|
||||
@@ -685,7 +687,7 @@
|
||||
"id": "4NqTnsxaAdRp"
|
||||
},
|
||||
"source": [
|
||||
"#### Create file struture of the training application"
|
||||
"#### Create file structure of the training application"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -879,7 +881,7 @@
|
||||
" # For multiprocessing distributed training, rank needs to be the\n",
|
||||
" # global rank among all the processes\n",
|
||||
" rank = rank * ngpus_per_node + local_rank\n",
|
||||
" print (f\"Distributed and Multiprocesing. Setting rank for each worker. rank={rank}\")\n",
|
||||
" print (f\"Distributed and Multi-processing. Setting rank for each worker. rank={rank}\")\n",
|
||||
"\n",
|
||||
" dist.init_process_group(\n",
|
||||
" backend=\"nccl\", \n",
|
||||
@@ -969,7 +971,7 @@
|
||||
"id": "gv0PB6WhjKsA"
|
||||
},
|
||||
"source": [
|
||||
"### Run custome training job with Reduction Server on Vertex AI\n",
|
||||
"### Run custom training job with Reduction Server on Vertex AI\n",
|
||||
"\n",
|
||||
"Configure a custom job with the pre-built container image for PyTorch and training code packaged as Python source distribution."
|
||||
]
|
||||
@@ -1078,7 +1080,7 @@
|
||||
"source": [
|
||||
"#### Submit the training job\n",
|
||||
"\n",
|
||||
"After the taining experiment and worker pool configuration parameters have been defined, use the Vertex AI SDK for Python to submit and monitor a training job.\n",
|
||||
"After the training cluster configuration parameters have been defined, use the Vertex AI SDK for Python to submit and monitor a training job.\n",
|
||||
"\n",
|
||||
"*NOTE: When using Vertex AI SDK for Python for submitting a training job, it creates a Training Pipeline which launches the custom job on Vertex AI Training service.*"
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user