mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
[community-content] PyTorch on Google Cloud Vertex AI - Fixes based on feedback (#290)
* PyTorch on Vertex - Updated to match GCPC v0.2.2 API * PyTorch on Vertex - Fixes based on review comments * PyTorch on Vertex - fixes based on review * PyTorch on Vertex - linter fixes * PyTorch on Vertex - fixes based on feedback * PyTorch on Vertex - fixes based on feedback Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Rajesh Thallam
co-authored by
Karl Weinmeister
Andrew Ferlitsch
Karl Weinmeister
Andrew Ferlitsch
parent
031a9190c3
commit
4ec6e2df04
+6
-3
@@ -2,10 +2,13 @@
|
||||
FROM pytorch/torchserve:latest-cpu
|
||||
|
||||
# install dependencies
|
||||
RUN python3 -m pip install --upgrade pip
|
||||
RUN pip3 install transformers
|
||||
|
||||
USER model-server
|
||||
|
||||
# copy model artifacts, custom handler and other dependencies
|
||||
COPY ./custom_text_handler.py /home/model-server/
|
||||
COPY ./custom_handler.py /home/model-server/
|
||||
COPY ./index_to_name.json /home/model-server/
|
||||
COPY ./model/finetuned-bert-classifier/ /home/model-server/
|
||||
|
||||
@@ -21,7 +24,7 @@ EXPOSE 7080
|
||||
EXPOSE 7081
|
||||
|
||||
# create model archive file packaging model artifacts and dependencies
|
||||
RUN torch-model-archiver -f --model-name=finetuned-bert-classifier --version=1.0 --serialized-file=/home/model-server/pytorch_model.bin --handler=/home/model-server/custom_text_handler.py --extra-files "/home/model-server/config.json,/home/model-server/tokenizer.json,/home/model-server/training_args.bin,/home/model-server/tokenizer_config.json,/home/model-server/special_tokens_map.json,/home/model-server/vocab.txt,/home/model-server/index_to_name.json" --export-path=/home/model-server/model-store
|
||||
RUN torch-model-archiver -f --model-name=finetuned-bert-classifier --version=1.0 --serialized-file=/home/model-server/pytorch_model.bin --handler=/home/model-server/custom_handler.py --extra-files "/home/model-server/config.json,/home/model-server/tokenizer.json,/home/model-server/training_args.bin,/home/model-server/tokenizer_config.json,/home/model-server/special_tokens_map.json,/home/model-server/vocab.txt,/home/model-server/index_to_name.json" --export-path=/home/model-server/model-store
|
||||
|
||||
# run Torchserve HTTP serve to respond to prediction requests
|
||||
CMD ["torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "finetuned-bert-classifier=finetuned-bert-classifier.mar", "--model-store", "/home/model-server/model-store"]
|
||||
CMD ["torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "finetuned-bert-classifier=finetuned-bert-classifier.mar", "--model-store", "/home/model-server/model-store"]
|
||||
|
||||
+62
-59
@@ -63,12 +63,12 @@
|
||||
"- [Training](#Training)\n",
|
||||
" - [Run Training Locally in the Notebook](#Training-locally-in-the-notebook)\n",
|
||||
" - [Run Training Job on Vertex AI](#Training-on-Vertex-AI)\n",
|
||||
" - [Training with pre-built container](#Run-Custom-Job-on-Vertex-Training-with-a-pre-built-container)\n",
|
||||
" - [Training with custom container](#Run-Custom-Job-on-Vertex-Training-with-custom-container)\n",
|
||||
" - [Training with pre-built container](#Run-Custom-Job-on-Vertex-AI-Training-with-a-pre-built-container)\n",
|
||||
" - [Training with custom container](#Run-Custom-Job-on-Vertex-AI-Training-with-custom-container)\n",
|
||||
"- [Tuning](#Hyperparameter-Tuning) \n",
|
||||
" - [Run Hyperparameter Tuning job on Vertex AI](#Run-Hyperparameter-Tuning-Job-on-Vertex-AI)\n",
|
||||
"- [Deploying](#Deploying)\n",
|
||||
" - [Deploying model on Vertex Predictions with custom container](#Deploying-model-on-Vertex-Predictions-with-custom-container)\n",
|
||||
" - [Deploying model on Vertex AI Predictions with custom container](#Deploying-model-on-Vertex AI-Predictions-with-custom-container)\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
@@ -202,9 +202,9 @@
|
||||
"id": "e0c1dcadc2c8"
|
||||
},
|
||||
"source": [
|
||||
"We will be using [Vertex SDK for Python](https://cloud.google.com/vertex-ai/docs/start/client-libraries#python) to interact with Vertex AI services. The high-level `aiplatform` library is designed to simplify common data science workflows by using wrapper classes and opinionated defaults. \n",
|
||||
"We will be using [Vertex AI SDK for Python](https://cloud.google.com/vertex-ai/docs/start/client-libraries#python) to interact with Vertex AI services. The high-level `aiplatform` library is designed to simplify common data science workflows by using wrapper classes and opinionated defaults. \n",
|
||||
"\n",
|
||||
"#### Install Vertex SDK for Python"
|
||||
"#### Install Vertex AI SDK for Python"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1199,7 +1199,7 @@
|
||||
"source": [
|
||||
"### Run predictions locally with sample examples\n",
|
||||
"\n",
|
||||
"Using the trained model, we can predict the sentiment label for an input text after applying the preprocessing function that was used during the training. We will run the predictions locally in the notebook and later show how you can deploy the model to an endpoint using [TorchServe](https://pytorch.org/serve/) on Vertex Predictions."
|
||||
"Using the trained model, we can predict the sentiment label for an input text after applying the preprocessing function that was used during the training. We will run the predictions locally in the notebook and later show how you can deploy the model to an endpoint using [TorchServe](https://pytorch.org/serve/) on Vertex AI Predictions."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1382,7 +1382,7 @@
|
||||
"id": "f7466d414a0e"
|
||||
},
|
||||
"source": [
|
||||
"### Run Custom Job on Vertex Training with a pre-built container"
|
||||
"### Run Custom Job on Vertex AI Training with a pre-built container"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1395,7 +1395,7 @@
|
||||
"\n",
|
||||
"In this notebook, we are using Hugging Face Datasets and fine tuning a transformer model from Hugging Face Transformers Library for sentiment analysis task using PyTorch. We will use [pre-built container for PyTorch](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers#pytorch) and package the training application code by adding standard Python dependencies - `transformers`, `datasets` and `tqdm` - in the `setup.py` file. \n",
|
||||
"\n",
|
||||
""
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1569,7 +1569,7 @@
|
||||
"source": [
|
||||
"#### **Run custom training job on Vertex AI**\n",
|
||||
"\n",
|
||||
"We use [Vertex SDK for Python](https://cloud.google.com/vertex-ai/docs/start/client-libraries#client_libraries) to create and submit training job to the Vertex training service."
|
||||
"We use [Vertex AI SDK for Python](https://cloud.google.com/vertex-ai/docs/start/client-libraries#client_libraries) to create and submit training job to the Vertex AI training service."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1578,7 +1578,7 @@
|
||||
"id": "5d2957ef04fd"
|
||||
},
|
||||
"source": [
|
||||
"##### **Initialize the Vertex SDK for Python**"
|
||||
"##### **Initialize the Vertex AI SDK for Python**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1598,7 +1598,7 @@
|
||||
"id": "6b0fed34b728"
|
||||
},
|
||||
"source": [
|
||||
"##### **Configure and submit Custom Job to Vertex Training service**"
|
||||
"##### **Configure and submit Custom Job to Vertex AI Training service**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1609,7 +1609,7 @@
|
||||
"source": [
|
||||
"Configure a [Custom Job](https://cloud.google.com/vertex-ai/docs/training/create-custom-job) with the [pre-built container](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) image for PyTorch and training code packaged as Python source distribution. \n",
|
||||
"\n",
|
||||
"**NOTE:** When using Vertex SDK for Python for submitting a training job, it creates a [Training Pipeline](https://cloud.google.com/vertex-ai/docs/training/create-training-pipeline) which launches the Custom Job on Vertex Training service."
|
||||
"**NOTE:** When using Vertex AI SDK for Python for submitting a training job, it creates a [Training Pipeline](https://cloud.google.com/vertex-ai/docs/training/create-training-pipeline) which launches the Custom Job on Vertex AI Training service."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1686,7 +1686,7 @@
|
||||
"\n",
|
||||
"You can monitor the custom job launched from Cloud Console following the link [here](https://console.cloud.google.com/vertex-ai/training/training-pipelines/) or use gcloud CLI command [`gcloud beta ai custom-jobs stream-logs`](https://cloud.google.com/sdk/gcloud/reference/beta/ai/custom-jobs/stream-logs)\n",
|
||||
"\n",
|
||||
""
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1798,7 +1798,7 @@
|
||||
"id": "c170d386492b"
|
||||
},
|
||||
"source": [
|
||||
"### Run Custom Job on Vertex Training with custom container"
|
||||
"### Run Custom Job on Vertex AI Training with custom container"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1807,7 +1807,7 @@
|
||||
"id": "035227b6e581"
|
||||
},
|
||||
"source": [
|
||||
"To create a [training job with custom container](https://cloud.google.com/vertex-ai/docs/training/create-custom-container?hl=hr), you define a `Dockerfile` to install or add the dependencies required for the training job. Then, you build and test your Docker image locally to verify, push the image to Container Registry and submit a Custom Job to Vertex Training service.\n",
|
||||
"To create a [training job with custom container](https://cloud.google.com/vertex-ai/docs/training/create-custom-container?hl=hr), you define a `Dockerfile` to install or add the dependencies required for the training job. Then, you build and test your Docker image locally to verify, push the image to Container Registry and submit a Custom Job to Vertex AI Training service.\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
@@ -1834,7 +1834,7 @@
|
||||
"%%writefile ./custom_container/Dockerfile\n",
|
||||
"\n",
|
||||
"# Use pytorch GPU base image\n",
|
||||
"FROM gcr.io/cloud-aiplatform/training/pytorch-gpu.1-7\n",
|
||||
"FROM us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-10:latest\n",
|
||||
"\n",
|
||||
"# set working directory\n",
|
||||
"WORKDIR /app\n",
|
||||
@@ -1968,7 +1968,7 @@
|
||||
"id": "a23e5e34bea9"
|
||||
},
|
||||
"source": [
|
||||
"##### **Initialize the Vertex SDK for Python**"
|
||||
"##### **Initialize the Vertex AI SDK for Python**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1988,11 +1988,11 @@
|
||||
"id": "abf1fa4085cb"
|
||||
},
|
||||
"source": [
|
||||
"##### **Configure and submit Custom Job to Vertex Training service**\n",
|
||||
"##### **Configure and submit Custom Job to Vertex AI Training service**\n",
|
||||
"\n",
|
||||
"Configure a [Custom Job](https://cloud.google.com/vertex-ai/docs/training/create-custom-job) with the [custom container](https://cloud.google.com/vertex-ai/docs/training/create-custom-container) image with training code and other dependencies\n",
|
||||
"\n",
|
||||
"**NOTE:** When using Vertex SDK for Python for submitting a training job, it creates a [Training Pipeline](https://cloud.google.com/vertex-ai/docs/training/create-training-pipeline) which launches the Custom Job to train on Vertex Training."
|
||||
"**NOTE:** When using Vertex AI SDK for Python for submitting a training job, it creates a [Training Pipeline](https://cloud.google.com/vertex-ai/docs/training/create-training-pipeline) which launches the Custom Job to train on Vertex AI Training."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2044,7 +2044,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# submit the custom job to Vertex training service\n",
|
||||
"# submit the custom job to Vertex AI training service\n",
|
||||
"model = job.run(\n",
|
||||
" replica_count=1,\n",
|
||||
" machine_type=\"n1-standard-8\",\n",
|
||||
@@ -2065,7 +2065,7 @@
|
||||
"\n",
|
||||
"You can monitor the custom job launched from Cloud Console following the link [here](https://console.cloud.google.com/vertex-ai/training/training-pipelines/) or use gcloud CLI command [`gcloud beta ai custom-jobs stream-logs`](https://cloud.google.com/sdk/gcloud/reference/beta/ai/custom-jobs/stream-logs)\n",
|
||||
"\n",
|
||||
""
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2148,11 +2148,11 @@
|
||||
"id": "ba6122f929e3"
|
||||
},
|
||||
"source": [
|
||||
"The training application code for fine-tuning a transformer model for sentiment analysis task uses hyperparameters such as learning rate and weight decay. These hyperparameters control the behavior of the training algorithm and can have a significant effect on the performance of the resulting model. This part of the notebook show how you can automate tuning these hyperparameters with Vertex Training service.\n",
|
||||
"The training application code for fine-tuning a transformer model for sentiment analysis task uses hyperparameters such as learning rate and weight decay. These hyperparameters control the behavior of the training algorithm and can have a significant effect on the performance of the resulting model. This part of the notebook show how you can automate tuning these hyperparameters with Vertex AI Training service.\n",
|
||||
"\n",
|
||||
"We submit a [Hyperparameter Tuning job](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview) to Vertex Training service by packaging the training application code and dependencies in a Docker container and push the container to Google Container Registry, similar to running a Custom Job on Vertex AI with Custom Container.\n",
|
||||
"We submit a [Hyperparameter Tuning job](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview) to Vertex AI Training service by packaging the training application code and dependencies in a Docker container and push the container to Google Container Registry, similar to running a Custom Job on Vertex AI with Custom Container.\n",
|
||||
"\n",
|
||||
""
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2163,7 +2163,7 @@
|
||||
"source": [
|
||||
"### How hyperparameter tuning works in Vertex AI?\n",
|
||||
"\n",
|
||||
"Following are the high level steps involved in running a Hyperparameter Tuning job on Vertex Training service:\n",
|
||||
"Following are the high level steps involved in running a Hyperparameter Tuning job on Vertex AI Training service:\n",
|
||||
"\n",
|
||||
"- You define the hyperparameters to tune the model along with the metric (or goal) to optimize\n",
|
||||
"- Vertex AI runs multiple trials of your training application with the hyperparameters and limits you specified - maximum number of trials to run and number of parallel trials. \n",
|
||||
@@ -2297,7 +2297,7 @@
|
||||
"source": [
|
||||
"### Run Hyperparameter Tuning Job on Vertex AI\n",
|
||||
"\n",
|
||||
"Before submitting the hyperparameter tuning job to Vertex AI, push the custom container image with training application to Google Cloud Container Registry and then submit the job to Vertex AI. We will be using the same image used for running Custom Job on Vertex Training service."
|
||||
"Before submitting the hyperparameter tuning job to Vertex AI, push the custom container image with training application to Google Cloud Container Registry and then submit the job to Vertex AI. We will be using the same image used for running Custom Job on Vertex AI Training service."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2326,7 +2326,7 @@
|
||||
"id": "f60fab07d67c"
|
||||
},
|
||||
"source": [
|
||||
"##### **Initialize the Vertex SDK for Python**"
|
||||
"##### **Initialize the Vertex AI SDK for Python**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2346,7 +2346,7 @@
|
||||
"id": "6652aa63ddff"
|
||||
},
|
||||
"source": [
|
||||
"##### **Configure and submit Hyperparameter Tuning Job to Vertex Training service**\n",
|
||||
"##### **Configure and submit Hyperparameter Tuning Job to Vertex AI Training service**\n",
|
||||
"\n",
|
||||
"Configure a [Hyperparameter Tuning Job](https://cloud.google.com/vertex-ai/docs/training/using-hyperparameter-tuning) with the [custom container](https://cloud.google.com/vertex-ai/docs/training/create-custom-container) image with training code and other dependencies.\n",
|
||||
"\n",
|
||||
@@ -2374,7 +2374,7 @@
|
||||
"id": "9d46db3a8b23"
|
||||
},
|
||||
"source": [
|
||||
"Define the training arguments with `hp-tune` argument set to `y` so that training application code can report metrics to Vertex"
|
||||
"Define the training arguments with `hp-tune` argument set to `y` so that training application code can report metrics to Vertex AI"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2548,7 +2548,7 @@
|
||||
"\n",
|
||||
"You can monitor the hyperparameter tuning job launched from Cloud Console following the link [here](https://console.cloud.google.com/vertex-ai/training/hyperparameter-tuning-jobs/) or use gcloud CLI command [`gcloud beta ai custom-jobs stream-logs`](https://cloud.google.com/sdk/gcloud/reference/beta/ai/custom-jobs/stream-logs)\n",
|
||||
"\n",
|
||||
""
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2557,7 +2557,7 @@
|
||||
"id": "ba934b434f03"
|
||||
},
|
||||
"source": [
|
||||
"After the job is finished, you can view and format the results of the hyperparameter tuning Trials (run by Vertex Training service) as a Pandas dataframe"
|
||||
"After the job is finished, you can view and format the results of the hyperparameter tuning Trials (run by Vertex AI Training service) as a Pandas dataframe"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2612,7 +2612,7 @@
|
||||
"id": "5dbccb2b7d32"
|
||||
},
|
||||
"source": [
|
||||
"Now from the results of Trials, you can pick the best performing Trial to deploy to Vertex Predictions"
|
||||
"Now from the results of Trials, you can pick the best performing Trial to deploy to Vertex AI Predictions"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2701,8 +2701,8 @@
|
||||
"JOB_NAME=${JOB_PREFIX}-pytorch-hptune-$(date +%Y%m%d%H%M%S)\n",
|
||||
"echo \"Launching hyperparameter tuning job with display name as \"$JOB_NAME\n",
|
||||
"\n",
|
||||
"# BUCKET_NAME: Change to your bucket name\n",
|
||||
"BUCKET_NAME=$1 # <-- CHANGE TO YOUR BUCKET NAME\n",
|
||||
"# BUCKET_NAME is a required parameter to run the cell.\n",
|
||||
"BUCKET_NAME=$1\n",
|
||||
"\n",
|
||||
"# APP_NAME: get application name\n",
|
||||
"APP_NAME=$2\n",
|
||||
@@ -2711,7 +2711,7 @@
|
||||
"JOB_DIR=${BUCKET_NAME}/${JOB_PREFIX}/model/${JOB_NAME}\n",
|
||||
"\n",
|
||||
"# custom container image URI\n",
|
||||
"CUSTOM_TRAIN_IMAGE_URI=f'gcr.io/'${PROJECT_ID}'/pytorch_gpu_train_'${APP_NAME}\n",
|
||||
"CUSTOM_TRAIN_IMAGE_URI='gcr.io/'${PROJECT_ID}'/pytorch_gpu_train_'${APP_NAME}\n",
|
||||
"\n",
|
||||
"# ========================================================\n",
|
||||
"# create hyperparameter tuning configuration file\n",
|
||||
@@ -2772,20 +2772,20 @@
|
||||
"source": [
|
||||
"## Deploying\n",
|
||||
"\n",
|
||||
"Deploying a PyTorch model on [Vertex Predictions](https://cloud.google.com/vertex-ai/docs/predictions/getting-predictions) requires to use a custom container that serves online predictions. You will deploy a container running [PyTorch's TorchServe](https://pytorch.org/serve/) tool in order to serve predictions from a fine-tuned transformer model from Hugging Face Transformers for sentiment analysis task. You can then use Vertex Predictions to classify sentiment of input texts. \n",
|
||||
"Deploying a PyTorch model on [Vertex AI Predictions](https://cloud.google.com/vertex-ai/docs/predictions/getting-predictions) requires to use a custom container that serves online predictions. You will deploy a container running [PyTorch's TorchServe](https://pytorch.org/serve/) tool in order to serve predictions from a fine-tuned transformer model from Hugging Face Transformers for sentiment analysis task. You can then use Vertex AI Predictions to classify sentiment of input texts. \n",
|
||||
"\n",
|
||||
"### Deploying model on Vertex Predictions with custom container\n",
|
||||
"### Deploying model on Vertex AI Predictions with custom container\n",
|
||||
"\n",
|
||||
"To use a custom container to serve predictions from a PyTorch model, you must provide Vertex AI with a Docker container image that runs an HTTP server, such as TorchServe in this case. Please refer to [documentation](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements) that describes the container image requirements to be compatible with Vertex Predictions.\n",
|
||||
"To use a custom container to serve predictions from a PyTorch model, you must provide Vertex AI with a Docker container image that runs an HTTP server, such as TorchServe in this case. Please refer to [documentation](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements) that describes the container image requirements to be compatible with Vertex AI Predictions.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Essentially, to deploy a PyTorch model on Vertex Predictions following are the steps:\n",
|
||||
"Essentially, to deploy a PyTorch model on Vertex AI Predictions following are the steps:\n",
|
||||
"\n",
|
||||
"1. Package the trained model artifacts including [default](https://pytorch.org/serve/#default-handlers) or [custom](https://pytorch.org/serve/custom_service.html) handlers by creating an archive file using [Torch model archiver](https://github.com/pytorch/serve/tree/master/model-archiver)\n",
|
||||
"2. Build a [custom container](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements) compatible with Vertex Predictions to serve the model using Torchserve\n",
|
||||
"3. Upload the model with custom container image to serve predictions as a Vertex Model resource\n",
|
||||
"4. Create a Vertex Endpoint and [deploy the model](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api) resource"
|
||||
"2. Build a [custom container](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements) compatible with Vertex AI Predictions to serve the model using Torchserve\n",
|
||||
"3. Upload the model with custom container image to serve predictions as a Vertex AI Model resource\n",
|
||||
"4. Create a Vertex AI Endpoint and [deploy the model](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api) resource"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3048,10 +3048,13 @@
|
||||
"FROM pytorch/torchserve:latest-cpu\n",
|
||||
"\n",
|
||||
"# install dependencies\n",
|
||||
"RUN python3 -m pip install --upgrade pip\n",
|
||||
"RUN pip3 install transformers\n",
|
||||
"\n",
|
||||
"USER model-server\n",
|
||||
"\n",
|
||||
"# copy model artifacts, custom handler and other dependencies\n",
|
||||
"COPY ./custom_text_handler.py /home/model-server/\n",
|
||||
"COPY ./custom_handler.py /home/model-server/\n",
|
||||
"COPY ./index_to_name.json /home/model-server/\n",
|
||||
"COPY ./model/$APP_NAME/ /home/model-server/\n",
|
||||
"\n",
|
||||
@@ -3071,7 +3074,7 @@
|
||||
" --model-name=$APP_NAME \\\n",
|
||||
" --version=1.0 \\\n",
|
||||
" --serialized-file=/home/model-server/pytorch_model.bin \\\n",
|
||||
" --handler=/home/model-server/custom_text_handler.py \\\n",
|
||||
" --handler=/home/model-server/custom_handler.py \\\n",
|
||||
" --extra-files \"/home/model-server/config.json,/home/model-server/tokenizer.json,/home/model-server/training_args.bin,/home/model-server/tokenizer_config.json,/home/model-server/special_tokens_map.json,/home/model-server/vocab.txt,/home/model-server/index_to_name.json\" \\\n",
|
||||
" --export-path=/home/model-server/model-store\n",
|
||||
"\n",
|
||||
@@ -3130,7 +3133,7 @@
|
||||
"source": [
|
||||
"#### **Run the container locally** ***[Optional]***\n",
|
||||
"\n",
|
||||
"Before push the container image to Container Registry to use it with Vertex Predictions, you can run it as a container in your local environment to verify that the server works as expected"
|
||||
"Before push the container image to Container Registry to use it with Vertex AI Predictions, you can run it as a container in your local environment to verify that the server works as expected"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3268,9 +3271,9 @@
|
||||
"id": "69477b3a00c0"
|
||||
},
|
||||
"source": [
|
||||
"#### **Deploying the serving container to Vertex Predictions**\n",
|
||||
"#### **Deploying the serving container to Vertex AI Predictions**\n",
|
||||
"\n",
|
||||
"We create a model resource on Vertex AI and deploy the model to a Vertex Endpoints. You must deploy a model to an endpoint before using the model. The deployed model runs the custom container image to serve predictions. "
|
||||
"We create a model resource on Vertex AI and deploy the model to a Vertex AI Endpoints. You must deploy a model to an endpoint before using the model. The deployed model runs the custom container image to serve predictions. "
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3301,7 +3304,7 @@
|
||||
"id": "a3da91e19af4"
|
||||
},
|
||||
"source": [
|
||||
"##### **Initialize the Vertex SDK for Python**"
|
||||
"##### **Initialize the Vertex AI SDK for Python**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3438,7 +3441,7 @@
|
||||
"id": "bc4673478269"
|
||||
},
|
||||
"source": [
|
||||
"#### **Invoking the Endpoint with deployed Model using Vertex SDK to make predictions**"
|
||||
"#### **Invoking the Endpoint with deployed Model using Vertex AI SDK to make predictions**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3488,7 +3491,7 @@
|
||||
"source": [
|
||||
"##### **Formatting input for online prediction**\n",
|
||||
"\n",
|
||||
"For online prediction requests, the prediction input instances must be formatted as JSON with base64 encoding as shown here:\n",
|
||||
"This notebook uses [Torchserve's KServe based inference API](https://pytorch.org/serve/inference_api.html#kserve-inference-api) which is also [Vertex AI Predictions compatible format](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements#prediction). For online prediction requests, format the prediction input instances as JSON with base64 encoding as shown here:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"[\n",
|
||||
@@ -3561,9 +3564,9 @@
|
||||
},
|
||||
"source": [
|
||||
"##### ***[Optional]*** **Make prediction requests using gcloud CLI**\n",
|
||||
"You can also call the Vertex Endpoint to make predictions using [`gcloud beta ai endpoints predict`](https://cloud.google.com/sdk/gcloud/reference/beta/ai/endpoints/predict). \n",
|
||||
"You can also call the Vertex AI Endpoint to make predictions using [`gcloud beta ai endpoints predict`](https://cloud.google.com/sdk/gcloud/reference/beta/ai/endpoints/predict). \n",
|
||||
"\n",
|
||||
"The following cell shows how to make a prediction request to Vertex Endpoints using `gcloud` CLI: "
|
||||
"The following cell shows how to make a prediction request to Vertex AI Endpoints using `gcloud` CLI: "
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3654,12 +3657,12 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_custom_job = True\n",
|
||||
"delete_hp_tuning_job = True\n",
|
||||
"delete_custom_job = False\n",
|
||||
"delete_hp_tuning_job = False\n",
|
||||
"delete_endpoint = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_bucket = True\n",
|
||||
"delete_image = True"
|
||||
"delete_model = False\n",
|
||||
"delete_bucket = False\n",
|
||||
"delete_image = False"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3687,7 +3690,7 @@
|
||||
"\n",
|
||||
"client_options = {\"api_endpoint\": API_ENDPOINT}\n",
|
||||
"\n",
|
||||
"# Initialize Vertex SDK\n",
|
||||
"# Initialize Vertex AI SDK\n",
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user