mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-27 07:31:58 +00:00
Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Andrew FerlitschandGitHub
|
f85c66b0a3 | ||
|
Andrew FerlitschandGitHub
|
bb17381b03 | ||
|
Andrew Ferlitsch
|
59f50df4f5 | ||
|
Andrew FerlitschandGitHub
|
3f06f48282 | ||
|
|
33abd1e427 | ||
|
Peter PingandGitHub
|
1d9bfe9934 | ||
|
|
fb9defa985 | ||
|
Soheila ZangenehandGitHub
|
824fb689e4 | ||
|
Andrew FerlitschandGitHub
|
c48dd8662b | ||
|
gericdongandGitHub
|
f251721d23 | ||
|
Andrew FerlitschandGitHub
|
e949eb128f | ||
|
|
df48e74f59 |
@@ -17,6 +17,7 @@
|
||||
/explainable_ai/SDK_Custom_Container_XAI.ipynb @brianchunkang
|
||||
/matching_engine/sdk_matching_engine_for_indexing.ipynb @ivanmkc
|
||||
/matching_engine/matching_engine_for_indexing.ipynb @yinghsienwu
|
||||
/matching_engine/stream_update_for_matching_engine.ipynb @peterping666
|
||||
/sdk/pytorch_lightning_custom_container_training.ipynb @brianchunkang
|
||||
/tensorboard @yfang1
|
||||
/feature_store @nayaknishant @morgandu
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -212,7 +212,7 @@
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component)\n",
|
||||
"\n",
|
||||
"4. [Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebooks.\n",
|
||||
"4. [Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Vertex AI Workbench Notebooks.\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -374,15 +374,8 @@
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "32e1cd21a5d5"
|
||||
},
|
||||
"source": [
|
||||
"authenticated. \n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
|
||||
Binary file not shown.
@@ -33,12 +33,12 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/community/ml_ops/get_started_with_automl_image_model_batch.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_automl_image_model_batch.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/community/ml_ops/get_started_with_automl_image_model_batch.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_automl_image_model_batch.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
@@ -245,7 +245,7 @@
|
||||
"\n",
|
||||
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -253,6 +253,17 @@
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -712,7 +723,7 @@
|
||||
"\n",
|
||||
"The `run` method when completed returns the `Model` resource.\n",
|
||||
"\n",
|
||||
"The execution of the training pipeline will take upto 20 minutes."
|
||||
"The execution of the training pipeline may take upto 20 minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -744,7 +755,7 @@
|
||||
"\n",
|
||||
"Batch prediction provides the ability to do offline batch processing of large amounts of prediction requests. Resources are only provisioned during the batch process and then deprovisioned when the batch request is completed. The results are stored in Cloud Storage, in contrast to online prediction where the results are returned as a HTTP response packet.\n",
|
||||
"\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server will convert to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server converts to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"\n",
|
||||
"### Input format for batch prediction jobs\n",
|
||||
"\n",
|
||||
@@ -777,7 +788,7 @@
|
||||
"\n",
|
||||
"**CSV**\n",
|
||||
"\n",
|
||||
"The csv header in the first line will always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"The csv header in the first line is always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"\n",
|
||||
" col1,col2,col3\n",
|
||||
" 1,3,\"cat1\"\n",
|
||||
@@ -955,7 +966,7 @@
|
||||
"- `prediction_format`: The format of the batch prediction response file: \"jsonl\", \"csv\", \"bigquery\", \"tf-record\", \"tf-record-gzip\" or \"file-list\"\n",
|
||||
"- `gcs_source`: A list of one or more batch request input files.\n",
|
||||
"- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
|
||||
"- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete."
|
||||
"- `sync`: If set to True, the call blocks while waiting for the asynchronous batch job to complete."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+24
-16
@@ -33,12 +33,12 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/community/ml_ops/get_started_with_automl_tabular_model_batch.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_automl_tabular_model_batch.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/community/ml_ops/get_started_with_automl_tabular_model_batch.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_automl_tabular_model_batch.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
@@ -106,7 +106,7 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the [Bank Marketing](https://pantheon.corp.google.com/storage/browser/_details/cloud-ml-tables-data/bank-marketing.csv) . This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
|
||||
"The dataset used for this tutorial is the [Bank Marketing](https://pantheon.corp.google.com/storage/browser/_details/cloud-ml-tables-data/bank-marketing.csv) . This dataset does not require any feature engineering. The version of the dataset you use in this tutorial is stored in a public Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -119,14 +119,11 @@
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"- BigQuery\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -250,7 +247,7 @@
|
||||
"\n",
|
||||
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -258,6 +255,17 @@
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -743,7 +751,7 @@
|
||||
"\n",
|
||||
"The `run` method when completed returns the `Model` resource.\n",
|
||||
"\n",
|
||||
"The execution of the training pipeline will take upto 8 hours."
|
||||
"The execution of the training pipeline may take upto 8 hours."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -776,7 +784,7 @@
|
||||
"\n",
|
||||
"Batch prediction provides the ability to do offline batch processing of large amounts of prediction requests. Resources are only provisioned during the batch process and then deprovisioned when the batch request is completed. The results are stored in Cloud Storage, in contrast to online prediction where the results are returned as a HTTP response packet.\n",
|
||||
"\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server will convert to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server converts to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"\n",
|
||||
"### Input format for batch prediction jobs\n",
|
||||
"\n",
|
||||
@@ -814,7 +822,7 @@
|
||||
"\n",
|
||||
"**CSV**\n",
|
||||
"\n",
|
||||
"The csv header in the first line will always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"The csv header in the first line is always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"\n",
|
||||
" col1,col2,col3\n",
|
||||
" 1,3,\"cat1\"\n",
|
||||
@@ -928,7 +936,7 @@
|
||||
"- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
|
||||
"- `instances_format`: The format for the input instances, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
|
||||
"- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
|
||||
"- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete."
|
||||
"- `sync`: If set to True, the call blocks while waiting for the asynchronous batch job to complete."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1109,7 +1117,7 @@
|
||||
"- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
|
||||
"- `instances_format`: The format for the input instances, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
|
||||
"- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
|
||||
"- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete."
|
||||
"- `sync`: If set to True, the call blocks while waiting for the asynchronous batch job to complete."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -244,7 +244,7 @@
|
||||
"\n",
|
||||
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -252,6 +252,17 @@
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"- identity - unique player identitity numbers\n",
|
||||
"- demographic features - information about the player, such as the geographic region in which a player is located\n",
|
||||
"- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level\n",
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n",
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player may churn, i.e. stop being an active player.\n",
|
||||
"\n",
|
||||
"**CSV batch input example**\n",
|
||||
"\n",
|
||||
@@ -574,7 +574,7 @@
|
||||
"\n",
|
||||
"Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n",
|
||||
"\n",
|
||||
"*Note*: TF releases before 2.3 for GPU support will fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3. This is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
|
||||
"*Note*: TF releases before 2.3 for GPU support fails to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3. This is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -638,7 +638,7 @@
|
||||
"\n",
|
||||
"Next, set the machine type to use for prediction.\n",
|
||||
"\n",
|
||||
"- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for prediction.\n",
|
||||
"- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VMs you use for for prediction.\n",
|
||||
" - `machine type`\n",
|
||||
" - `n1-standard`: 3.75GB of memory per vCPU.\n",
|
||||
" - `n1-highmem`: 6.5GB of memory per vCPU\n",
|
||||
@@ -712,7 +712,7 @@
|
||||
"\n",
|
||||
"Batch prediction provides the ability to do offline batch processing of large amounts of prediction requests. Resources are only provisioned during the batch process and then deprovisioned when the batch request is completed. The results are stored in Cloud Storage, in contrast to online prediction where the results are returned as a HTTP response packet.\n",
|
||||
"\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server will convert to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"The input format for your batch job is dependent on the format supported by your model server. Foremost, the web server in your model server must support a JSONL format, which the web server converts to a format support either directly by the model input intertace or a serving function interface. For batch prediction, this JSONL format is referred to as the `pivot` format.\n",
|
||||
"\n",
|
||||
"### Input format for batch prediction jobs\n",
|
||||
"\n",
|
||||
@@ -749,7 +749,7 @@
|
||||
"\n",
|
||||
"**CSV**\n",
|
||||
"\n",
|
||||
"The csv header in the first line will always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"The csv header in the first line is always be ignored. String fields are required to be double quoted explicitly, otherwise the row is discarded and parsing error messages are outputted to error files. Non-quoted values are always transferred as floats.\n",
|
||||
"\n",
|
||||
" col1,col2,col3\n",
|
||||
" 1,3,\"cat1\"\n",
|
||||
@@ -910,7 +910,7 @@
|
||||
"- `prediction_format`: The format of the batch prediction response file: \"jsonl\", \"csv\", \"bigquery\", \"tf-record\", \"tf-record-gzip\" or \"file-list\"\n",
|
||||
"- `job_display_name`: The human readable name for the prediction job.\n",
|
||||
" - `gcs_source`: A list of one or more Cloud Storage paths to your batch prediction requests.\n",
|
||||
"- `gcs_destination_prefix`: The Cloud Storage path that the service will write the predictions to.\n",
|
||||
"- `gcs_destination_prefix`: The Cloud Storage path that the service writes the predictions to.\n",
|
||||
"- `model_parameters`: Additional filtering parameters for serving prediction results.\n",
|
||||
"- `machine_type`: The type of machine to use for training.\n",
|
||||
"- `accelerator_type`: The hardware accelerator type.\n",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+3
-3
@@ -133,8 +133,6 @@
|
||||
"id": "8yVpQt-JHKPF"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench notebooks**, your environment already meets\n",
|
||||
@@ -247,6 +245,8 @@
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
@@ -1521,7 +1521,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "model_monitoring_automl.ipynb",
|
||||
"name": "get_started_with_model_monitoring_automl.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
+13
-13
@@ -35,20 +35,20 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fmodel_monitoring_custom.ipynb\">\n",
|
||||
" <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Workbench AI Notebook\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/model_monitoring_custom.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage7/model_monitoring_custom.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/model_monitoring_custom.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage7/model_monitoring_custom.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td> <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/ml_ops/stage7/model_monitoring_custom.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
@@ -110,7 +110,7 @@
|
||||
"- identity - unique player identitity numbers\n",
|
||||
"- demographic features - information about the player, such as the geographic region in which a player is located\n",
|
||||
"- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level\n",
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player."
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player may churn, i.e. stop being an active player."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -140,8 +140,6 @@
|
||||
"id": "8yVpQt-JHKPF"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench notebooks**, your environment already meets\n",
|
||||
@@ -254,6 +252,8 @@
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
@@ -264,7 +264,7 @@
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -780,7 +780,7 @@
|
||||
"\n",
|
||||
"When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
|
||||
"\n",
|
||||
"The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
|
||||
"The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service attempts to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
|
||||
"\n",
|
||||
"For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
|
||||
"\n",
|
||||
@@ -1643,7 +1643,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "model_monitoring_custom.ipynb",
|
||||
"name": "get_started_with_model_monitoring_custom.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
+6
-6
@@ -110,7 +110,7 @@
|
||||
"- identity - unique player identitity numbers\n",
|
||||
"- demographic features - information about the player, such as the geographic region in which a player is located\n",
|
||||
"- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level\n",
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player."
|
||||
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player may churn, i.e. stop being an active player."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -140,8 +140,6 @@
|
||||
"id": "8yVpQt-JHKPF"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench notebooks**, your environment already meets\n",
|
||||
@@ -254,6 +252,8 @@
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
@@ -264,7 +264,7 @@
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
@@ -944,7 +944,7 @@
|
||||
"\n",
|
||||
"When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
|
||||
"\n",
|
||||
"The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
|
||||
"The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service attempts to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
|
||||
"\n",
|
||||
"For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
|
||||
"\n",
|
||||
@@ -1859,7 +1859,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "model_monitoring_custom_tf_serving.ipynb",
|
||||
"name": "get_started_with_model_monitoring_custom_tf_serving.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
+3
-3
@@ -140,8 +140,6 @@
|
||||
"id": "8yVpQt-JHKPF"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench notebooks**, your environment already meets\n",
|
||||
@@ -254,6 +252,8 @@
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
@@ -1697,7 +1697,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "model_monitoring_setup.ipynb",
|
||||
"name": "get_started_with_model_monitoring_setup.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -56,7 +56,7 @@ def parse_notebook(path):
|
||||
# cell 1 is copyright
|
||||
nth = 0
|
||||
cell, nth = get_cell(path, cells, nth)
|
||||
if not cell['source'][0].startswith('# Copyright'):
|
||||
if not 'Copyright' in cell['source'][0]:
|
||||
report_error(path, 0, "missing copyright cell")
|
||||
|
||||
# check for notices
|
||||
|
||||
@@ -29,6 +29,9 @@
|
||||
/automl/automl_forecasting_bqml_arima_plus_comparison.ipynb @TheMichaelHu
|
||||
/automl/automl_tabular_on_vertex_pipelines.ipynb @helinwang
|
||||
/custom/custom_training_tensorboard_profiler.ipynb @itseric
|
||||
/workbench/spark/spark_sample_notebook.ipynb @bmiro
|
||||
/workbench/spark/spark_sample_notebook.ipynb @bradmiro
|
||||
/workbench/spark/spark_ml.ipynb @bradmiro
|
||||
/model-registry/bqml-vertexai-model-registry.ipynb @soheilazangeneh
|
||||
/workbench/exploratory_data_analysis/explore_data_in_bigquery_with_workbench.ipynb @alokpattani
|
||||
/model_evaluation/automl_tabular_classification_model_evaluation.ipynb @soheilazangeneh
|
||||
/model_evaluation/automl_tabular_regression_model_evaluation.ipynb @soheilazangeneh
|
||||
|
||||
@@ -263,7 +263,7 @@
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"3. [Enable the following APIs: Vertex AI API, Cloud Resource Manager API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,cloudresourcemanager.googleapis.com).\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
|
||||
@@ -32,18 +32,18 @@
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery-ml/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model-registry/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery-ml/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model-registry/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery-ml/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model-registry/bqml-vertexai-model-registry.ipynb\" target=\"_blank\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
@@ -677,7 +677,7 @@
|
||||
"source": [
|
||||
"### Find the model in the Vertex Model Registry\n",
|
||||
"\n",
|
||||
"You can use the `Vertex AI Model list()` method with a filter query to find the automatically registered model."
|
||||
"You can use the `Vertex AI Model()` method with `model_name` parameter to find the automatically registered model."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+1415
File diff suppressed because it is too large
Load Diff
+1473
File diff suppressed because it is too large
Load Diff
BIN
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
@@ -81,7 +81,6 @@
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Define and compile a `Vertex AI` pipeline.\n",
|
||||
"- Schedule a recurring pipeline run.\n",
|
||||
"- Specify which service account to use for a pipeline run."
|
||||
]
|
||||
},
|
||||
@@ -97,13 +96,9 @@
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"* Cloud Functions\n",
|
||||
"* Cloud Scheduler\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
|
||||
"[Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
|
||||
"[Cloud Functions pricing](ttps://cloud.google.com/functions/pricing), and\n",
|
||||
"[Clould Scheduler pricing]((https://cloud.google.com/scheduler/pricing)),\n",
|
||||
"and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
@@ -926,69 +921,6 @@
|
||||
"job.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "schedule_pipeline_run"
|
||||
},
|
||||
"source": [
|
||||
"## Recurring pipeline runs: create a scheduled pipeline job\n",
|
||||
"\n",
|
||||
"This section shows how to create a **scheduled pipeline job**. You do this using the pipeline you already defined.\n",
|
||||
"\n",
|
||||
"Under the hood, the scheduled jobs are supported by the Cloud Scheduler and a Cloud Functions function. Check first that the APIs for both of these services are enabled.\n",
|
||||
"You will need to first enable the [enable the Cloud Scheduler API](http://console.cloud.google.com/apis/library/cloudscheduler.googleapis.com) and the [Cloud Functions and Cloud Build APIs](https://console.cloud.google.com/flows/enableapi?apiid=cloudfunctions,cloudbuild.googleapis.com) if you have not already done so.\n",
|
||||
"Note:you need to [create an App Engine app for your project](https://cloud.google.com/scheduler/docs/quickstart) if one does not already exist.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"See the [Cloud Scheduler](https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules) documentation for more on the cron syntax.\n",
|
||||
"\n",
|
||||
"Create a scheduled pipeline job, passing as an argument the job specification file that you compiled above.\n",
|
||||
"\n",
|
||||
"*Note:* You can pass a `parameter_values` dict that specifies the pipeline input parameters you want to use."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Ty5hDoNX2Ou8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" from kfp.v2.google.client import AIPlatformClient # noqa: F811\n",
|
||||
"\n",
|
||||
" api_client = AIPlatformClient(project_id=PROJECT_ID, region=REGION)\n",
|
||||
"\n",
|
||||
" # adjust time zone and cron schedule as necessary\n",
|
||||
" response = api_client.create_schedule_from_job_spec(\n",
|
||||
" job_spec_path=\"intro_pipeline.json\",\n",
|
||||
" schedule=\"2 * * * *\",\n",
|
||||
" time_zone=\"America/Los_Angeles\", # change this as necessary\n",
|
||||
" parameter_values={\"text\": \"Hello world!\"},\n",
|
||||
" # pipeline_root=PIPELINE_ROOT # this argument is necessary if you did not specify PIPELINE_ROOT as part of the pipeline definition.\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "J8AP1viy2Ou8"
|
||||
},
|
||||
"source": [
|
||||
"Once the scheduled job is created, you can see it listed in the [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler/) panel in the Console.\n",
|
||||
"\n",
|
||||
"<a href=\"https://storage.googleapis.com/amy-jo/images/kf-pls/pipelines_scheduler.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/kf-pls/pipelines_scheduler.png\" width=\"95%\"/></a>\n",
|
||||
"\n",
|
||||
"You can test the setup from the Cloud Scheduler panel by clicking 'RUN NOW'.\n",
|
||||
"\n",
|
||||
"> **Note**: The implementation is using a Cloud Functions function, which you can see listed in the [Cloud Functions](https://console.cloud.google.com/functions/list) panel in the console as `templated_http_request-v1`.\n",
|
||||
"Don't delete this function, as it will prevent the Cloud Scheduler jobs from actually kicking off the pipeline run. If you do delete it, create a new scheduled job in order to recreate the function.\n",
|
||||
"\n",
|
||||
"When you're done experimenting, you probably want to **PAUSE** your scheduled job from the Cloud Scheduler panel, so that the recurrent jobs do not keep running."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1222,7 +1154,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_pipeline = True\n",
|
||||
"delete_bucket = True\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user