mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
added MLPerf benchmark reference and updated Criteo sample to use GRPC for stock containers (#630)
* added MLPerf benchmark reference and updated Criteo sample to use GRPC for stock containers * addressed feedback for BERT sample and did similar changes to Criteo sample Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Aleksey Vlasenko
co-authored by
Andrew Ferlitsch
Andrew Ferlitsch
parent
0250879f65
commit
4e7061b2db
+52
-31
@@ -67,9 +67,9 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"In this sample we demonstrate how to fine-tune a BERT base classification model for sentiment analysis.\n",
|
||||
"In this sample you learn how to fine-tune a BERT base classification model for sentiment analysis.\n",
|
||||
"\n",
|
||||
"Then we export a trained model to Vertex AI Prediction service using an open source based TensorFlow 2.7 container and the optimized TensorFlow runtime container. We benchmark those models so we can compare their predictions.\n",
|
||||
"Then you export a trained model to Vertex AI Prediction service using an open source based TensorFlow 2.7 container and the optimized TensorFlow runtime container, run performance evaluation for those models and compare their predictions.\n",
|
||||
"\n",
|
||||
"For additional information about Vertex AI Prediction optimized TensorFlow runtime containers, see https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime.\n",
|
||||
"\n",
|
||||
@@ -116,7 +116,7 @@
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
|
||||
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
|
||||
"all the requirements to run this notebook. You can skip this step."
|
||||
]
|
||||
},
|
||||
@@ -179,10 +179,10 @@
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"# Vertex AI Workbench Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
@@ -196,12 +196,12 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-text==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-serving-api==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tf-models-official==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage"
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-text==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-serving-api==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tf-models-official==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -363,7 +363,7 @@
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
@@ -412,10 +412,10 @@
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"# If on Vertex AI Workbench Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
@@ -571,7 +571,7 @@
|
||||
"id": "9cbb5302a3c4"
|
||||
},
|
||||
"source": [
|
||||
"## Downloading dataset"
|
||||
"## Download the dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -612,7 +612,7 @@
|
||||
"id": "8b7051896d19"
|
||||
},
|
||||
"source": [
|
||||
"## Preprocess dataset"
|
||||
"## Preprocess the dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -703,7 +703,7 @@
|
||||
"id": "d56746554dd6"
|
||||
},
|
||||
"source": [
|
||||
"As a base for our model we take uncased BERT-Base model from TensorFlow Hub:\n",
|
||||
"As a base for our model you take uncased BERT-Base model from TensorFlow Hub:\n",
|
||||
"https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3"
|
||||
]
|
||||
},
|
||||
@@ -757,7 +757,7 @@
|
||||
},
|
||||
"source": [
|
||||
"Define a classification model by feeding the output BERT encoder model into dropout and dense layers.\n",
|
||||
"For details, see the https://www.tensorflow.org/text/tutorials/classify_text_with_bert tutorial."
|
||||
"Learn more about [Classify text with BERT]( https://www.tensorflow.org/text/tutorials/classify_text_with_bert)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -917,7 +917,7 @@
|
||||
"id": "11cc837b06f9"
|
||||
},
|
||||
"source": [
|
||||
"Check the model signature.\n",
|
||||
"Check the model signature to see which fields prediction request should have.\n",
|
||||
"\n",
|
||||
"Note that you might see a stacktrace message about a missing 'CaseFoldUTF8' op. This is a known issue with `saved_model_cli` that you can ignore."
|
||||
]
|
||||
@@ -948,7 +948,7 @@
|
||||
"id": "953eb5e1e86e"
|
||||
},
|
||||
"source": [
|
||||
"Now we can generate requests to send to our model for inference. Requests are generated in the JSON Lines format, one request per line."
|
||||
"Now you can generate requests to send to our model for inference. Requests are generated in the JSON Lines format, one request per line."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1047,7 +1047,7 @@
|
||||
"id": "5bbdded5d2ab"
|
||||
},
|
||||
"source": [
|
||||
"The TensorFlow runtime has components that are lazily initialized. Lazy initialization might result in high latency for the first requests that are sent to a model after it's loaded. This latency can be several orders of magnitude higher than that of a single inference request. \n",
|
||||
"The TensorFlow runtime has components that are lazily initialized. Lazy initialization might result in high latency for the first requests that are sent to a model after it's loaded. This latency can be several orders of magnitude higher than that of a single inference request.\n",
|
||||
"\n",
|
||||
"For more information about SavedModel warmup, see https://www.tensorflow.org/tfx/serving/saved_model_warmup.\n",
|
||||
"\n",
|
||||
@@ -1209,9 +1209,9 @@
|
||||
"id": "83f62a939359"
|
||||
},
|
||||
"source": [
|
||||
"The AI Platform Python client library works as a client/server model. \n",
|
||||
"The AI Platform Python client library works as a client/server model.\n",
|
||||
"\n",
|
||||
"We are going to use following clients in this sample:\n",
|
||||
"You are going to use following clients in this sample:\n",
|
||||
"- Model Service for managing models.\n",
|
||||
"- Endpoint Service for deployment.\n",
|
||||
"- Prediction Service for serving."
|
||||
@@ -1254,7 +1254,7 @@
|
||||
"\n",
|
||||
"`artifact_uri` argument should point to a GCS path where `saved_model.pb` file is located for your model.\n",
|
||||
"\n",
|
||||
"`image_uri` specifies which docker image to use. Here we upload the same model using TF2.7 GPU and Vertex AI Prediction optimized TensorFlow runtime images."
|
||||
"`image_uri` specifies which docker image to use. Here you upload the same model using TF2.7 GPU and Vertex AI Prediction optimized TensorFlow runtime images."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1386,7 +1386,7 @@
|
||||
"id": "5ff288387d03"
|
||||
},
|
||||
"source": [
|
||||
"See [endpoint_service.create_endpoint](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_create_endpoint) API description for details."
|
||||
"Learn more about [endpoint_service.create_endpoint](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_create_endpoint)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1467,7 +1467,7 @@
|
||||
"id": "ebb2d55b45e0"
|
||||
},
|
||||
"source": [
|
||||
"For details about the `enpoint_service.deploy_model` API, see [enpoint_service.deploy_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_deploy_model)."
|
||||
"Learn more about [enpoint_service.deploy_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_deploy_model)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1510,7 +1510,7 @@
|
||||
"source": [
|
||||
"tf_opt_gpu_deployed_model_dict = {\n",
|
||||
" \"model\": tf_opt_gpu_model,\n",
|
||||
" \"display_name\": \"Criteo Kaggle optimized TensorFlow runtime GPU model\",\n",
|
||||
" \"display_name\": \"BERT Base optimized TensorFlow runtime GPU model\",\n",
|
||||
" \"dedicated_resources\": {\n",
|
||||
" \"min_replica_count\": 1,\n",
|
||||
" \"max_replica_count\": 1,\n",
|
||||
@@ -1540,7 +1540,7 @@
|
||||
"source": [
|
||||
"tf_opt_lossy_gpu_deployed_model_dict = {\n",
|
||||
" \"model\": tf_opt_lossy_gpu_model,\n",
|
||||
" \"display_name\": \"Criteo Kaggle optimized TensorFlow runtime GPU model with lossy optimizations\",\n",
|
||||
" \"display_name\": \"BERT Base optimized TensorFlow runtime GPU model with lossy optimizations\",\n",
|
||||
" \"dedicated_resources\": {\n",
|
||||
" \"min_replica_count\": 1,\n",
|
||||
" \"max_replica_count\": 1,\n",
|
||||
@@ -1598,7 +1598,7 @@
|
||||
"id": "ca1dfbcdf81e"
|
||||
},
|
||||
"source": [
|
||||
"Alternatively you can send POST REST requests without using the SDK. For more details, see https://cloud.google.com/vertex-ai/docs/predictions/online-predictions-custom-models#online_predict_custom_trained-drest.\n",
|
||||
"Alternatively you can send POST REST requests without using the SDK. Learn more about https://cloud.google.com/vertex-ai/docs/predictions/online-predictions-custom-models#online_predict_custom_trained-drest.\n",
|
||||
"This method is slightly faster."
|
||||
]
|
||||
},
|
||||
@@ -1904,6 +1904,24 @@
|
||||
"You can see that the Vertex AI Prediction optimized TensorFlow runtime has signficantly higher throughput and lower latency compared to TensorFlow 2.7."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "06f2711e6221"
|
||||
},
|
||||
"source": [
|
||||
"## (Optional) Compare performance of deployed models using MLPerf Inference loadgen"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "adcd7c46de26"
|
||||
},
|
||||
"source": [
|
||||
"MLPerf Inference is a benchmark suite for measuring how fast systems can run models in a variety of deployment scenarios. MLPerf is now an industry standard way of measuring model performance. You can follow instructions at https://github.com/tensorflow/tpu/tree/master/models/experimental/inference/load_test to run MLPerf Inferenence benchmark for deployed models."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1921,7 +1939,7 @@
|
||||
"source": [
|
||||
"In this sample the Vertex Prediction optimized TensorFlow runtime is used with the `allow_precision_affecting_optimizations` flag set to `true` to gain additional speedup. Now let's check how those optimizations effect prediction results.\n",
|
||||
"\n",
|
||||
"We compare the results of predictions for 32,000 requests for a model running on the optimized TensorFlow runtime with lossy optimizations and on TF2.7."
|
||||
"Compare the results of predictions for 32,000 requests for a model running on the optimized TensorFlow runtime with lossy optimizations and on TF2.7."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2083,7 +2101,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!gsutil rm -r $BUCKET_URI"
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" !gsutil rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+72
-33
@@ -67,14 +67,14 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"In this sample we demonstrate how to train a tabular model using TensorFlow Keras or Estimator API using Criteo Kaggle dataset.\n",
|
||||
"Next, we export a trained model to the Vertex AI Prediction service using open source based TensorFlow 2.7 container and the optimized TensorFlow runtime container. We benchmark those models side by side and compare predictions.\n",
|
||||
"In this sample you learn how to train a tabular model using TensorFlow Keras or Estimator API using Criteo Kaggle dataset.\n",
|
||||
"Next, you export a trained model to the Vertex AI Prediction service using open source based TensorFlow 2.7 container and the optimized TensorFlow runtime container, run performance evaluation for those models side by side and compare predictions.\n",
|
||||
"\n",
|
||||
"For additional information about Vertex AI Prediction optimized TensorFlow runtime containers, please refer to https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime.\n",
|
||||
"\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"We use Criteo Kaggle dataset, which takes about 4GB in this sample.\n",
|
||||
"In this sample you use Criteo Kaggle dataset, which takes about 4GB.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Objective\n",
|
||||
@@ -113,7 +113,7 @@
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Google Cloud Notebooks**, your environment meets the requirements to run this notebook. You can skip this step."
|
||||
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment meets the requirements to run this notebook. You can skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -122,7 +122,7 @@
|
||||
"id": "gCuSR8GkAgzl"
|
||||
},
|
||||
"source": [
|
||||
"**If you are not using Colab or Google Cloud Notebooks**, you must have the following in your environment to meet this notebook's requirements.\n",
|
||||
"**If you are not using Colab or Vertex AI Workbench Notebooks**, you must have the following in your environment to meet this notebook's requirements.\n",
|
||||
"\n",
|
||||
"* The Google Cloud SDK\n",
|
||||
"* Git\n",
|
||||
@@ -172,10 +172,10 @@
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"# Vertex AI Workbench Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
@@ -189,10 +189,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-serving-api==2.7.0\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage"
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade tensorflow-serving-api==2.7.0 -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform -q\n",
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -345,7 +345,7 @@
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
@@ -395,10 +395,10 @@
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"# If on Vertex AI Workbench Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
@@ -556,7 +556,7 @@
|
||||
"id": "9ee07345bcc6"
|
||||
},
|
||||
"source": [
|
||||
"## Downloading Dataset"
|
||||
"## Download the dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -620,7 +620,7 @@
|
||||
"id": "ng-Nrc8UdbP2"
|
||||
},
|
||||
"source": [
|
||||
"## Reading and transforming dataset\n",
|
||||
"## Read and transform the dataset\n",
|
||||
"\n",
|
||||
"Before the model can be trained, the variables must be pre-processed.\n",
|
||||
"\n",
|
||||
@@ -954,7 +954,7 @@
|
||||
"id": "AVibdmrufYRA"
|
||||
},
|
||||
"source": [
|
||||
"Check the model signature."
|
||||
"Check the model signature to see which fields prediction request should have."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -986,7 +986,7 @@
|
||||
"Another option to train a model is to use the TensorFlow Estimator API. For more information, see\n",
|
||||
"https://github.com/tensorflow/docs/blob/r2.4/site/en/tutorials/estimator/premade.ipynb\n",
|
||||
"\n",
|
||||
"The following code is provided only for illustration purposes. We use the Keras model for deployment."
|
||||
"The following code is provided only for illustration purposes. You use the Keras model for deployment."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1232,7 +1232,7 @@
|
||||
"id": "9c75a550809d"
|
||||
},
|
||||
"source": [
|
||||
"The TensorFlow runtime has components that are lazily initialized. Lazy initialization might result in high latency for the first requests that are sent to a model after it's loaded. This latency can be several orders of magnitude higher than that of a single inference request. \n",
|
||||
"The TensorFlow runtime has components that are lazily initialized. Lazy initialization might result in high latency for the first requests that are sent to a model after it's loaded. This latency can be several orders of magnitude higher than that of a single inference request.\n",
|
||||
"\n",
|
||||
"For more information about SavedModel warmup, see https://www.tensorflow.org/tfx/serving/saved_model_warmup.\n",
|
||||
"\n",
|
||||
@@ -1435,7 +1435,7 @@
|
||||
"id": "9d259d62f8a4"
|
||||
},
|
||||
"source": [
|
||||
"The throughput and latency of the Criteo model we trained is sensitive to network performance."
|
||||
"The throughput and latency of the Criteo model you trained is sensitive to network performance."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1488,7 +1488,7 @@
|
||||
"id": "743b3af8f747"
|
||||
},
|
||||
"source": [
|
||||
"For simplicity, we setup VPC peering to the default network. You can create a different network for your project.\n",
|
||||
"For simplicity, you setup VPC peering to the default network. You can create a different network for your project.\n",
|
||||
"\n",
|
||||
"If you setup VPC peering with any other network, make sure that the network already exists and that your VM is running on that network."
|
||||
]
|
||||
@@ -1598,12 +1598,16 @@
|
||||
"id": "lYFkS2H9dDfd"
|
||||
},
|
||||
"source": [
|
||||
"See [model_service.upload_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.model_service.ModelServiceClient#google_cloud_aiplatform_v1_services_model_service_ModelServiceClient_upload_model) documentation for details.\n",
|
||||
"Learn more about [model_service.upload_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.model_service.ModelServiceClient#google_cloud_aiplatform_v1_services_model_service_ModelServiceClient_upload_model).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"`artifact_uri` argument should point to a GCS path where `saved_model.pb` file is located for your model.\n",
|
||||
"\n",
|
||||
"`image_uri` specifies which docker image to use. Here we upload the same model using TF2.7 GPU and Vertex AI Prediction optimized TensorFlow runtime images."
|
||||
"`image_uri` specifies which docker image to use. Here we upload the same model using TF2.7 GPU and Vertex AI Prediction optimized TensorFlow runtime images.\n",
|
||||
"\n",
|
||||
"In order to be able to send requests to your models over gRPC, you need to set `model_name` argument and update `predict_route` and `health_route` accordingly.\n",
|
||||
"\n",
|
||||
"Please note that gRPC support in Vertex AI Prediction is still experimental."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1619,6 +1623,15 @@
|
||||
" \"artifact_uri\": BUCKET_URI,\n",
|
||||
" \"container_spec\": {\n",
|
||||
" \"image_uri\": \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-7:latest\",\n",
|
||||
" \"args\": [\n",
|
||||
" \"--port=8500\",\n",
|
||||
" \"--rest_api_port=8080\",\n",
|
||||
" \"--model_name=default\",\n",
|
||||
" \"--model_base_path=$(AIP_STORAGE_URI)\",\n",
|
||||
" ],\n",
|
||||
" \"ports\": [{\"container_port\": 8080}],\n",
|
||||
" \"predict_route\": \"/v1/models/default:predict\",\n",
|
||||
" \"health_route\": \"/v1/models/default\",\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"tf27_cpu_model = (\n",
|
||||
@@ -1642,6 +1655,15 @@
|
||||
" \"artifact_uri\": BUCKET_URI,\n",
|
||||
" \"container_spec\": {\n",
|
||||
" \"image_uri\": \"us-docker.pkg.dev/vertex-ai/prediction/tf2-gpu.2-7:latest\",\n",
|
||||
" \"args\": [\n",
|
||||
" \"--port=8500\",\n",
|
||||
" \"--rest_api_port=8080\",\n",
|
||||
" \"--model_name=default\",\n",
|
||||
" \"--model_base_path=$(AIP_STORAGE_URI)\",\n",
|
||||
" ],\n",
|
||||
" \"ports\": [{\"container_port\": 8080}],\n",
|
||||
" \"predict_route\": \"/v1/models/default:predict\",\n",
|
||||
" \"health_route\": \"/v1/models/default\",\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"tf27_gpu_model = (\n",
|
||||
@@ -1664,11 +1686,7 @@
|
||||
"- *allow_precompilation* - turns on model pre-compilation for better performance. Note that model precompilation happens when the first request with the new batch size arrives, and the response for that request is sent after precompilation is complete. To mitigate this, specify a warmup file (see the section earlier in this colab). Model precompilation works for different kinds of models, and in most cases has a positive effect on performance. However, we recommend that you try it out for your model before you enable it in production.\n",
|
||||
"- *allow_precision_affecting_optimizations* - enables precision affecting optimizations. In some cases this makes the model run significantly faster at the cost of very minimal loss to model prediction power. You should assess the precision impact to your model when using this optimization.\n",
|
||||
"\n",
|
||||
"For the list of available optimized TensorFlow runtimer containers and options, see https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime.\n",
|
||||
"\n",
|
||||
"In order to be able to send requests to your models over gRPC, you need to set `model_name` argument and update `predict_route` and `health_route` accordingly.\n",
|
||||
"\n",
|
||||
"Please note that gRPC support in Vertex AI Prediction is still experimental and only available for models deployed using optimized TensorFlow runtime."
|
||||
"For the list of available optimized TensorFlow runtimer containers and options, see https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1768,7 +1786,7 @@
|
||||
"id": "e4279643c2cf"
|
||||
},
|
||||
"source": [
|
||||
"See [endpoint_service.create_endpoint](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_create_endpoint) API description for details."
|
||||
"Learn more about [endpoint_service.create_endpoint](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_create_endpoint)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1887,7 +1905,7 @@
|
||||
"id": "f8586711566b"
|
||||
},
|
||||
"source": [
|
||||
"For details about the `enpoint_service.deploy_model` API, see [enpoint_service.deploy_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_deploy_model)."
|
||||
"Learn more about [enpoint_service.deploy_model](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient#google_cloud_aiplatform_v1_services_endpoint_service_EndpointServiceClient_deploy_model)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2244,7 +2262,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tf27_cpu_results = benchmark_rest_private_endpoint(\n",
|
||||
"tf27_cpu_results = benchmark_grpc_private_endpoint(\n",
|
||||
" tf27_cpu_endpoint, [10, 20, 30, 40, 50, 55]\n",
|
||||
")\n",
|
||||
"tf27_cpu_results"
|
||||
@@ -2258,7 +2276,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tf27_gpu_results = benchmark_rest_private_endpoint(\n",
|
||||
"tf27_gpu_results = benchmark_grpc_private_endpoint(\n",
|
||||
" tf27_gpu_endpoint, [10, 20, 30, 40, 50, 60, 70, 75]\n",
|
||||
")\n",
|
||||
"tf27_gpu_results"
|
||||
@@ -2382,6 +2400,24 @@
|
||||
"You can see that the Vertex AI Prediction optimized TensorFlow runtime has signficantly higher throughput and lower latency compared to TensorFlow 2.7."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3f369366c0af"
|
||||
},
|
||||
"source": [
|
||||
"## (Optional) Compare performance of deployed models using MLPerf Inference loadgen"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "bfd210c97934"
|
||||
},
|
||||
"source": [
|
||||
"MLPerf Inference is a benchmark suite for measuring how fast systems can run models in a variety of deployment scenarios. MLPerf is now an industry standard way of measuring model performance. You can follow instructions at https://github.com/tensorflow/tpu/tree/master/models/experimental/inference/load_test to run MLPerf Inferenence benchmark for deployed models.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -2559,7 +2595,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!gsutil rm -r $BUCKET_URI"
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" !gsutil rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user