mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-27 15:42:05 +00:00
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Karl WeinmeisterandGitHub
|
d9a1b4fc3b | ||
|
Andrew FerlitschandGitHub
|
b1de8caab3 | ||
|
Andrew FerlitschandGitHub
|
76c3ca1968 | ||
|
Andrew FerlitschandGitHub
|
6d62d238dd | ||
|
Andrew FerlitschandGitHub
|
cb1959a83d | ||
|
Andrew FerlitschandGitHub
|
93665056ec |
@@ -1,3 +1,2 @@
|
||||
notebooks/official
|
||||
notebooks/notebook_template.ipynb
|
||||
notebooks/community/ml_ops
|
||||
|
||||
@@ -2,7 +2,7 @@ git+https://github.com/tensorflow/docs
|
||||
ipython
|
||||
jupyter
|
||||
nbconvert
|
||||
black==22.3.0
|
||||
black==22.1.0
|
||||
pyupgrade==2.31.1
|
||||
isort==5.10.1
|
||||
flake8==4.0.1
|
||||
|
||||
@@ -3,4 +3,3 @@
|
||||
/pytorch_text_classification_using_vertex_sdk_and_gcloud @RajeshThallam
|
||||
/pytorch_text_classification_using_vertex_sdk_and_gcloud @RajeshThallam @ultrons
|
||||
/sklearn_text_classification_from_script_using_vertex_sdk @maxhardt
|
||||
/pluto_on_workbench @wkharold
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
# Overview
|
||||
*Pluto* is a programming environment for Julia, designed to be interactive and helpful. It provides a familiar notebook interface but it is not a Jupyter notebook. The biggest difference is that Pluto notebooks are reactive, changing a variable or function in one cell causes the cells that depend on that variable or function to be reevaluated. Pluto also provides useful interaction mechanisms that allow users to dynamically interact with the notebooks computation state.
|
||||
|
||||
The JuliaCon 2020 presentation: [Interactive notebooks ~ Pluto.jl]() provides a good introduction to Pluto. The source is at [fonsp/Pluto.jl]()
|
||||
|
||||
# Install Pluto
|
||||
|
||||
## Create a Vertex AI JupyterLab Instance
|
||||
|
||||
1. From the [GCP console](https://console.cloud.google.com) "hamburger menu"
|
||||
|
||||
select Vertex AI > Workbench
|
||||
2. Click NEW NOTEBOOK
|
||||
|
||||
* Choose Python 3 if you won't be using a GPU
|
||||
* Choose Python 3 (CUDA Toolkit xx.y) if you do want use a GPU
|
||||
3. Give the notebook an appropriate name
|
||||
4. Edit Notebook properties if you have special requirements otherwise accept the defaults and click CREATE
|
||||
5. When the notebook instance is ready click OPEN JUPYTERLAB
|
||||
|
||||
## Configure JupyterLab
|
||||
|
||||
1. Open a terminal by clicking the Terminal icon.
|
||||
1. Install the plutoserver
|
||||
pip3 install git+https://github.com/fonsp/pluto-on-jupyterlab.git
|
||||
1. In a browser go to [julialang.org/downloads](https://julialang.org/downloads/)
|
||||
1. In the Current stable release right click on the `Generic Linux on x86 / 64-bit (glibc)` link
|
||||
Select copy link address
|
||||
1. Back in the terminal switch to root via
|
||||
sudo -i
|
||||
1. Download the release to /opt and install julia in /usr/local/bin
|
||||
```bash
|
||||
cd /opt
|
||||
wget <paste the release link address>
|
||||
tar xf <name of the downloaded tar file>
|
||||
ln -s /opt/<julia-x.y.z>/bin/julia /usr/local/bin
|
||||
^d
|
||||
```
|
||||
1. Add the Pluto package to Julia
|
||||
```bash
|
||||
julia
|
||||
julia> ]add Pluto
|
||||
julia> bksp
|
||||
julia> using Pluto
|
||||
julia> ^d
|
||||
```
|
||||
1. From the JupyterLab menu bar select File > Shut Down
|
||||
|
||||
# Start Pluto
|
||||
1. Click OPEN JUPYTERLAB in the Workbench
|
||||
1. In the Notebook section of the Launcher click Pluto.jl
|
||||
1. The welcome to Pluto.jl screen should appear
|
||||
+2
-2
@@ -706,8 +706,8 @@
|
||||
" else:\n",
|
||||
" data_spec = training_data_spec_transformation_fn(\n",
|
||||
" agent.policy.trajectory_spec)\n",
|
||||
" replay_buffer = trainer.get_replay_buffer(data_spec, environment.batch_size,\n",
|
||||
" steps_per_loop)\n",
|
||||
" replay_buffer = trainer.get_replay_buffer(data_spec, environment.batch_size,\n",
|
||||
" steps_per_loop)\n",
|
||||
"\n",
|
||||
" # `step_metric` records the number of individual rounds of bandit interaction;\n",
|
||||
" # that is, (number of trajectories) * batch_size.\n",
|
||||
|
||||
@@ -19,4 +19,3 @@
|
||||
/feature_store @nayaknishant @morgandu
|
||||
/vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb @entrpn
|
||||
/vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb @RajeshThallam
|
||||
/vertex_endpoints/optimized_tensorflow_runtime @vlasenkoalexey
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 153 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 138 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 141 KiB |
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 230 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 182 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 142 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 140 KiB |
+766
-892
File diff suppressed because it is too large
Load Diff
+886
-972
File diff suppressed because it is too large
Load Diff
@@ -12,7 +12,7 @@ The purpose of this set of notebooks and markdown files is to demonstrate Google
|
||||
2. [Experimentation](stage2)
|
||||
3. [Formalization](stage3)
|
||||
4. [Evaluation](stage4)
|
||||
5. [Deployment](stage5)
|
||||
5. Deployment
|
||||
6. [Serving](stage6)
|
||||
7. Monitoring
|
||||
8. Continuous Training
|
||||
|
||||
@@ -76,18 +76,6 @@ The steps performed include:
|
||||
- image data
|
||||
```
|
||||
|
||||
[Get Started with Data Labeling](get_started_data_labeling.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Create a Specialist Pool for data labelers.
|
||||
- Create a data labeling job.
|
||||
- Submit the data labeling job.
|
||||
- List data labeling jobs.
|
||||
- Cancel a data labeling job.
|
||||
```
|
||||
|
||||
### E2E Stage Example
|
||||
|
||||
[Stage 1: Data Management](mlops_data_management.ipynb)
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -38,15 +38,9 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -73,7 +67,7 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). In this version of the dataset you consider the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -110,7 +104,7 @@
|
||||
"source": [
|
||||
"### Recommendations\n",
|
||||
"\n",
|
||||
"When doing E2E MLOps on Google Cloud, following are the best practices when dealing with structured (tabular) data in BigQuery:\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following best practices with structured (tabular) data in BigQuery:\n",
|
||||
"\n",
|
||||
"- For AutoML training:\n",
|
||||
" - Create a managed dataset with Vertex AI `TabularDataset`.\n",
|
||||
@@ -130,7 +124,7 @@
|
||||
" - Within the generator (upstream)\n",
|
||||
" - Within the model (downstream)\n",
|
||||
" - XGBoost model training:\n",
|
||||
" - Use BigQuery ML built-in XGBoost training.\n",
|
||||
" - Use BigQuery ML builtin XGBoost training.\n",
|
||||
" - Alternatively, create a DMatrix generator from CSV files extracted from BigQuery table.\n",
|
||||
" - Pytorch model training:\n",
|
||||
" - Extract the BigQuery to a pandas dataframe.\n",
|
||||
@@ -138,19 +132,10 @@
|
||||
" - Create a DataLoader generator from the pandas dataframe.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"- Alternatively:\n",
|
||||
"- Alternately:\n",
|
||||
" - Extract the BigQuery table to CSV files.\n",
|
||||
" - Preprocess the CSV files.\n",
|
||||
" - Create a tf.data.Dataset generator from the CSV files.\n",
|
||||
" \n",
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"- BigQuery\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
" - Create a tf.data.Dataset generator from the CSV files."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -161,7 +146,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages to execute this notebook."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -172,22 +157,40 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"# Install the packages\n",
|
||||
"! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install -U xgboost $USER_FLAG\n",
|
||||
"! pip3 install -U tensorflow-io==0.18 $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "install_xgboost"
|
||||
},
|
||||
"source": [
|
||||
"Install the latest GA version of *XGBoost* library as well."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U xgboost $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -219,39 +222,6 @@
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a47846030fef"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "84cd83853240"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI, BigQuery, Compute Engine and Cloud Storage APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,bigquery,compute_component,storage_component).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -328,10 +298,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -358,66 +325,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "77c385f0db59"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex AI\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "535223fa4b84"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -428,7 +335,12 @@
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"When you create a dataset resource using the Vertex SDK, you can provide a Cloud Storage bucket that contains the data. Vertex AI creates the dataset resource from the data. In this tutorial, Vertex AI also creates a dataset resource from your data in the Cloud Storage bucket.\n",
|
||||
"When you submit a custom training job using the Vertex SDK, you upload a Python package\n",
|
||||
"containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
|
||||
"the code from this package. In this tutorial, Vertex AI also saves the\n",
|
||||
"trained model that results from your job in the same bucket. You can then\n",
|
||||
"create an `Endpoint` resource based on this output in order to serve\n",
|
||||
"online predictions.\n",
|
||||
"\n",
|
||||
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
|
||||
]
|
||||
@@ -441,7 +353,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -452,8 +364,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -473,7 +385,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -493,7 +405,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -502,6 +414,9 @@
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
@@ -513,12 +428,75 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform\n",
|
||||
"import pandas as pd\n",
|
||||
"import xgboost as xgb\n",
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_bq"
|
||||
},
|
||||
"source": [
|
||||
"#### Import BigQuery\n",
|
||||
"\n",
|
||||
"Import the BigQuery package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_bq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import bigquery"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_xgboost"
|
||||
},
|
||||
"source": [
|
||||
"#### Import XGBoost\n",
|
||||
"\n",
|
||||
"Import the XGBoost package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import xgboost as xgb"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_pandas"
|
||||
},
|
||||
"source": [
|
||||
"#### Import pandas\n",
|
||||
"\n",
|
||||
"Import the pandas package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_pandas"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -538,7 +516,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION)"
|
||||
"aip.init(project=PROJECT_ID, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -560,7 +538,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bqclient = bigquery.Client(project=PROJECT_ID)"
|
||||
"bqclient = bigquery.Client()"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -571,7 +549,7 @@
|
||||
"source": [
|
||||
"#### Location of BigQuery training data.\n",
|
||||
"\n",
|
||||
"Now, set the variable `IMPORT_FILE` to the location of the data table in BigQuery and `BQ_TABLE` with the table id."
|
||||
"Now set the variable `IMPORT_FILE` to the location of the data table in BigQuery."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -613,10 +591,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.TabularDataset.create(\n",
|
||||
"dataset = aip.TabularDataset.create(\n",
|
||||
" display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
|
||||
" bq_source=[IMPORT_FILE],\n",
|
||||
" labels={\"user_metadata\": BUCKET_URI[5:]},\n",
|
||||
" labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\"\n",
|
||||
@@ -632,7 +610,7 @@
|
||||
"source": [
|
||||
"### Copy the dataset to Cloud Storage\n",
|
||||
"\n",
|
||||
"Next, you make a copy of the BigQuery table as a CSV file, to Cloud Storage using the BigQuery extract command.\n",
|
||||
"Next, you make a copy of the BigQuery dataset, as a CSV file, to Cloud Storage using the BigQuery extract command.\n",
|
||||
"\n",
|
||||
"Learn more about [BigQuery command line interface](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference)."
|
||||
]
|
||||
@@ -648,9 +626,9 @@
|
||||
"comps = BQ_TABLE.split(\".\")\n",
|
||||
"BQ_PROJECT_DATASET_TABLE = comps[0] + \":\" + comps[1] + \".\" + comps[2]\n",
|
||||
"\n",
|
||||
"! bq --location=us extract --destination_format CSV $BQ_PROJECT_DATASET_TABLE $BUCKET_URI/mydata*.csv\n",
|
||||
"! bq --location=us extract --destination_format CSV $BQ_PROJECT_DATASET_TABLE $BUCKET_NAME/mydata*.csv\n",
|
||||
"\n",
|
||||
"IMPORT_FILES = ! gsutil ls $BUCKET_URI/mydata*.csv\n",
|
||||
"IMPORT_FILES = ! gsutil ls $BUCKET_NAME/mydata*.csv\n",
|
||||
"\n",
|
||||
"print(IMPORT_FILES)\n",
|
||||
"\n",
|
||||
@@ -686,12 +664,15 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gcs_source = IMPORT_FILES\n",
|
||||
"if \"IMPORT_FILES\" in globals():\n",
|
||||
" gcs_source = IMPORT_FILES\n",
|
||||
"else:\n",
|
||||
" gcs_source = [IMPORT_FILE]\n",
|
||||
"\n",
|
||||
"dataset = aiplatform.TabularDataset.create(\n",
|
||||
"dataset = aip.TabularDataset.create(\n",
|
||||
" display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
|
||||
" gcs_source=gcs_source,\n",
|
||||
" labels={\"user_metadata\": BUCKET_URI[5:]},\n",
|
||||
" labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -713,30 +694,6 @@
|
||||
"Learn more about [Creating BigQuery views](https://cloud.google.com/bigquery/docs/views)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7dc142433e50"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set dataset name and view name in BigQuery\n",
|
||||
"BQ_MY_DATASET = \"[your-dataset-name]\"\n",
|
||||
"BQ_MY_TABLE = \"[your-view-name]\"\n",
|
||||
"\n",
|
||||
"# Otherwise, use the default names\n",
|
||||
"if (\n",
|
||||
" BQ_MY_DATASET == \"\"\n",
|
||||
" or BQ_MY_DATASET is None\n",
|
||||
" or BQ_MY_DATASET == \"[your-dataset-name]\"\n",
|
||||
"):\n",
|
||||
" BQ_MY_DATASET = \"mlops_dataset_\" + TIMESTAMP\n",
|
||||
"\n",
|
||||
"if BQ_MY_TABLE == \"\" or BQ_MY_TABLE is None or BQ_MY_TABLE == \"[your-view-name]\":\n",
|
||||
" BQ_MY_TABLE = \"mlops_view_\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -745,7 +702,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create the resources\n",
|
||||
"BQ_MY_DATASET = 'mydataset'\n",
|
||||
"BQ_MY_TABLE = 'myview'\n",
|
||||
"! bq --location=US mk -d \\\n",
|
||||
"$PROJECT_ID:$BQ_MY_DATASET\n",
|
||||
"\n",
|
||||
@@ -786,8 +744,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Download the table.\n",
|
||||
"table = bigquery.TableReference.from_string(BQ_TABLE)\n",
|
||||
"# Download a table.\n",
|
||||
"table = bigquery.TableReference.from_string(\"bigquery-public-data.samples.gsod\")\n",
|
||||
"\n",
|
||||
"rows = bqclient.list_rows(\n",
|
||||
" table,\n",
|
||||
@@ -1073,6 +1031,22 @@
|
||||
"TABLE_ID = \"gsod\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def create_bigquery_dataset(dataset_id):\n",
|
||||
" dataset = bigquery.Dataset(\n",
|
||||
" bigquery.dataset.DatasetReference(PROJECT_ID, dataset_id)\n",
|
||||
" )\n",
|
||||
" dataset.location = \"us\"\n",
|
||||
"\n",
|
||||
" try:\n",
|
||||
" dataset = bqclient.create_dataset(dataset) # API request\n",
|
||||
" return True\n",
|
||||
" except Exception as err:\n",
|
||||
" print(err)\n",
|
||||
" if err.code != 409: # http_client.CONFLICT\n",
|
||||
" raise\n",
|
||||
" return False\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def load_data_into_bigquery(url, dataset_id, table_id):\n",
|
||||
" create_bigquery_dataset(dataset_id)\n",
|
||||
" dataset = bqclient.dataset(dataset_id)\n",
|
||||
@@ -1105,11 +1079,13 @@
|
||||
"source": [
|
||||
"### Read BigQuery table into XGboost DMatrix\n",
|
||||
"\n",
|
||||
"Currently, there is no direct data feeding connector between BigQuery and the open source XGBoost. The BigQuery ML service has a built-in XGBoost training module.\n",
|
||||
"Currently, there is no direct data feeding connector between BigQuery and the open source XGBoost.\n",
|
||||
"\n",
|
||||
"Alernatively, you extract the data either as a pandas dataframe or as CSV files. The extracted data is then given as an input to a `DMatrix` object when training the model.\n",
|
||||
"The BigQuery ML service has XGBoost training builtin.\n",
|
||||
"\n",
|
||||
"Learn more about [Getting started with built-in XGBoost](https://cloud.google.com/ai-platform/training/docs/algorithms/xgboost-start)."
|
||||
"Alernatively, you extract the data either as a pandas dataframe or as CSV files. The extracted data is then inputted to a `DMatrix` object when training the model.\n",
|
||||
"\n",
|
||||
"Learn more about [Getting started with builtin XGBoost](https://cloud.google.com/ai-platform/training/docs/algorithms/xgboost-start)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1120,7 +1096,7 @@
|
||||
"source": [
|
||||
"### Read pandas table into XGboost DMatrix\n",
|
||||
"\n",
|
||||
"Next, you load the pandas dataframe into a `DMatrix` object. XGBoost does not support non-numeric inputs. Any column that is categorical need to be one-hot encoded prior to loading the dataframe."
|
||||
"Next, you load the pandas dataframe into a `DMatrix` object. XGBoost does not support non-numeric inputs. Any column that is categorical will need to be one-hot encoded prior to loading the dataframe."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1146,7 +1122,7 @@
|
||||
"source": [
|
||||
"### Read CSV files into XGboost DMatrix\n",
|
||||
"\n",
|
||||
"Currently, there is no Cloud Storage support in XGBoost. If you use CSV files for input, you need to download them locally."
|
||||
"Currently, there is no Cloud Storage support in XGBoost. If you use CSV files for input, you will need to download them locally."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1168,41 +1144,87 @@
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"# Clean up\n",
|
||||
"# Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Vertex AI Dataset resource\n",
|
||||
"- Cloud Storage Bucket\n",
|
||||
"- BigQuery Dataset\n",
|
||||
"\n",
|
||||
"Set `delete_storage` to _True_ to delete the storage resources used in this notebook."
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "47ad926d84e8"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"# Delete the dataset using the Vertex dataset object\n",
|
||||
"dataset.delete()\n",
|
||||
"# Delete the temporary BigQuery dataset\n",
|
||||
"! bq rm -r -f $PROJECT_ID:$DATASET_ID\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"delete_storage = False\n",
|
||||
"if delete_storage or os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Delete the created GCS bucket\n",
|
||||
" ! gsutil rm -r $BUCKET_URI\n",
|
||||
" # Delete the created BigQuery datasets\n",
|
||||
" ! bq rm -r -f $PROJECT_ID:$BQ_MY_DATASET"
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -153,13 +153,12 @@ The steps performed include:
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Create a local BigQuery table in your project
|
||||
- Train a BQML model
|
||||
- Evaluate the BQML model
|
||||
- Export the BQML model as a cloud model
|
||||
- Upload the exported model as a `Vertex AI Model` resource
|
||||
- Hyperparameter tune a BQML model with `Vertex AI Vizier`
|
||||
- Automatically register a BQML model to `Vertex AI Model Registry`
|
||||
- Create a local BQ table in your project.
|
||||
- Train a BQML model.
|
||||
- Evaluate the BQML model.
|
||||
- Export the BQML model as a cloud model.
|
||||
- Upload the exported model as a Vertex AI Model resource.
|
||||
- Hyperparameter tune a BQML model with Vertex AI Vizier.
|
||||
```
|
||||
|
||||
[Get Started with Vertex Feature Store](get_started_vertex_feature_store.ipynb)
|
||||
@@ -186,48 +185,6 @@ The steps performed include:
|
||||
- Train an AutoML model with CMEK encryption.
|
||||
```
|
||||
|
||||
[Get Started with TensorFlow Hub models](get_started_with_tfhub_models.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a TensorFlow Hub prebuilt model.
|
||||
- Add the task component as a classifier for the CIFAR-10 dataset.
|
||||
- Fine tune locally the model with transfer learning training.
|
||||
- Construct a custom training script:
|
||||
- Get training data from TensorFlow Datasets
|
||||
- Get model architecture from TensorFlow Hub
|
||||
- Train then model
|
||||
- Save model artifacts and upload as Vertex AI Model resource.
|
||||
```
|
||||
|
||||
[Get Started with Vertex AI TabNet builtin algorithm](get_started_with_tabnet.ipynb)
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Get the training data.
|
||||
- Configure training parameters for the Vertex AI TabNet container.
|
||||
- Train the model using Vertex AI Training using CSV data.
|
||||
- Upload the model as a Vertex AI Model resource.
|
||||
- Deploy the Vertex AI Model resource to a Vertex AI Endpoint resource.
|
||||
- Make a prediction with the deployed model.
|
||||
- Hyperparameter tuning the Vertex AI TabNet model.
|
||||
- Train the model using Vertex AI Training using BigQuery table.
|
||||
```
|
||||
[Get Started with Vertex AI TabNet builtin algorithm](get_started_with_tabnet.ipynb)
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Get the training data.
|
||||
- Configure training parameters for the Vertex AI TabNet container.
|
||||
- Train the model using Vertex AI Training using CSV data.
|
||||
- Upload the model as a Vertex AI Model resource.
|
||||
- Deploy the Vertex AI Model resource to a Vertex AI Endpoint resource.
|
||||
- Make a prediction with the deployed model.
|
||||
- Hyperparameter tuning the Vertex AI TabNet model.
|
||||
- Train the model using Vertex AI Training using BigQuery table.
|
||||
```
|
||||
|
||||
|
||||
### E2E Stage Example
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -38,15 +38,9 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_automl_training.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_automl_training.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_automl_training.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -71,11 +65,9 @@
|
||||
"id": "dataset:flowers,icn"
|
||||
},
|
||||
"source": [
|
||||
"### Datasets\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"#### Image\n",
|
||||
"\n",
|
||||
"The image dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in a given image from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -84,9 +76,9 @@
|
||||
"id": "dataset:gsod,lrg"
|
||||
},
|
||||
"source": [
|
||||
"#### Tabular\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The tabular dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -95,20 +87,9 @@
|
||||
"id": "dataset:happydb,tcn"
|
||||
},
|
||||
"source": [
|
||||
"#### Text\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The text dataset used for this tutorial is the [Happy Moments dataset](https://www.kaggle.com/ritresearch/happydb) from [Kaggle Datasets](https://www.kaggle.com/ritresearch/happydb). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "98eb93ec6faa"
|
||||
},
|
||||
"source": [
|
||||
"#### Video\n",
|
||||
"\n",
|
||||
"The video dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset](https://todo) from [MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model will predict the start frame where a golf swing begins."
|
||||
"The dataset used for this tutorial is the [Happy Moments dataset](https://www.kaggle.com/ritresearch/happydb) from [Kaggle Datasets](https://www.kaggle.com/ritresearch/happydb). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -127,12 +108,11 @@
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Train an image model\n",
|
||||
"- Export the image model as an edge model\n",
|
||||
"- Train a tabular model\n",
|
||||
"- Export the tabular model as a cloud model\n",
|
||||
"- Train a text model\n",
|
||||
"- Train a video model"
|
||||
"- Train an image model.\n",
|
||||
"- Export the image model as an edge model.\n",
|
||||
"- Train a tabular model.\n",
|
||||
"- Export the tabular model as a cloud model.\n",
|
||||
"- Train a text model."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -145,24 +125,9 @@
|
||||
"\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following are best practices for when to use AutoML:\n",
|
||||
"\n",
|
||||
"* **You have a limited amount of training data**\n",
|
||||
"**You have a limited amount of training data**\n",
|
||||
"\n",
|
||||
"* **You want to establish a baseline metric before experimenting with a custom model**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "fb3451ce8e47"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
"**You want to establish a baseline metric before experimenting with a custom model**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -173,7 +138,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages for executing the MLOps notebooks."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -184,18 +149,20 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-storage $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -233,23 +200,6 @@
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI, Compute Engine and Cloud Storage APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage_component).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n",
|
||||
"\n",
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
@@ -320,10 +270,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -350,63 +297,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3ffa6b6c7cdb"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex AI\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2b72272258fc"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -430,8 +320,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -442,9 +331,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
|
||||
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -464,7 +352,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -484,7 +372,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -507,7 +395,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -529,7 +417,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -560,7 +448,7 @@
|
||||
"source": [
|
||||
"## AutoML image models\n",
|
||||
"\n",
|
||||
"AutoML can train the following types of image models:\n",
|
||||
"AutoML can train the following types of models:\n",
|
||||
"\n",
|
||||
"- classification\n",
|
||||
"- objection detection\n",
|
||||
@@ -616,7 +504,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"FILE = IMPORT_FILE\n",
|
||||
"if \"IMPORT_FILES\" in globals():\n",
|
||||
" FILE = IMPORT_FILES[0]\n",
|
||||
"else:\n",
|
||||
" FILE = IMPORT_FILE\n",
|
||||
"\n",
|
||||
"count = ! gsutil cat $FILE | wc -l\n",
|
||||
"print(\"Number of Examples\", int(count[0]))\n",
|
||||
@@ -654,10 +545,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.ImageDataset.create(\n",
|
||||
" display_name=\"flowers_\" + TIMESTAMP,\n",
|
||||
"dataset = aip.ImageDataset.create(\n",
|
||||
" display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
|
||||
" gcs_source=[IMPORT_FILE],\n",
|
||||
" import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification,\n",
|
||||
" import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(dataset.resource_name)"
|
||||
@@ -702,8 +593,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dag = aiplatform.AutoMLImageTrainingJob(\n",
|
||||
" display_name=\"flowers_\" + TIMESTAMP,\n",
|
||||
"dag = aip.AutoMLImageTrainingJob(\n",
|
||||
" display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" prediction_type=\"classification\",\n",
|
||||
" multi_label=False,\n",
|
||||
" model_type=\"MOBILE_TF_LOW_LATENCY_1\",\n",
|
||||
@@ -721,14 +612,14 @@
|
||||
"source": [
|
||||
"#### Run the training pipeline\n",
|
||||
"\n",
|
||||
"Next, you run the created DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `dataset`: The `Dataset` resource to train the model.\n",
|
||||
"- `model_display_name`: The human readable name for the trained model.\n",
|
||||
"- `training_fraction_split`: The percentage of the dataset to use for training.\n",
|
||||
"- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
|
||||
"- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
|
||||
"- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of milli node-hours (1000 = node-hour).\n",
|
||||
"- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
|
||||
"- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.\n",
|
||||
"\n",
|
||||
"The `run` method when completed returns the `Model` resource.\n",
|
||||
@@ -746,7 +637,7 @@
|
||||
"source": [
|
||||
"model = dag.run(\n",
|
||||
" dataset=dataset,\n",
|
||||
" model_display_name=\"flowers_\" + TIMESTAMP,\n",
|
||||
" model_display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" training_fraction_split=0.8,\n",
|
||||
" validation_fraction_split=0.1,\n",
|
||||
" test_fraction_split=0.1,\n",
|
||||
@@ -762,7 +653,7 @@
|
||||
},
|
||||
"source": [
|
||||
"## Review model evaluation scores\n",
|
||||
"After your model training has finished, you can review the evaluation scores for it.\n",
|
||||
"After your model has finished training, you can review the evaluation scores for it.\n",
|
||||
"\n",
|
||||
"First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project."
|
||||
]
|
||||
@@ -776,13 +667,11 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get model resource ID\n",
|
||||
"models = aiplatform.Model.list(filter=\"display_name=flowers_\" + TIMESTAMP)\n",
|
||||
"models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
|
||||
"\n",
|
||||
"# Get a reference to the Model Service client\n",
|
||||
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
|
||||
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
|
||||
" client_options=client_options\n",
|
||||
")\n",
|
||||
"model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
|
||||
"\n",
|
||||
"model_evaluations = model_service_client.list_model_evaluations(\n",
|
||||
" parent=models[0].resource_name\n",
|
||||
@@ -832,7 +721,7 @@
|
||||
"source": [
|
||||
"### Get test item\n",
|
||||
"\n",
|
||||
"You will use an arbitrary example out of the dataset as a test item. Don't be concerned that the example was likely used in training the model. You are just looking at how to make a prediction."
|
||||
"You will use an arbitrary example out of the dataset as a test item. Don't be concerned that the example was likely used in training the model -- we just want to demonstrate how to make a prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -864,7 +753,7 @@
|
||||
"\n",
|
||||
"#### Request\n",
|
||||
"\n",
|
||||
"Since your test item is in a public Cloud Storage bucket in this example, you copy it to your bucket and read the contents of the image using `Cloud Storage SDK`. To pass the test data to the prediction service, you encode the bytes into base64 which makes the content safe from modification while transmitting binary data over the network.\n",
|
||||
"Since in this example your test item is in a Cloud Storage bucket, you open and read the contents of the image using `tf.io.gfile.Gfile()`. To pass the test data to the prediction service, you encode the bytes into base64 -- which makes the content safe from modification while transmitting binary data over the network.\n",
|
||||
"\n",
|
||||
"The format of each instance is:\n",
|
||||
"\n",
|
||||
@@ -886,66 +775,32 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1c1d53e89beb"
|
||||
"id": "predict_request:mbsdk,icn"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import base64\n",
|
||||
"\n",
|
||||
"from google.cloud import storage\n",
|
||||
"import tensorflow as tf\n",
|
||||
"\n",
|
||||
"# Copy the test image to the Cloud storage bucket as \"test.jpg\"\n",
|
||||
"test_image_local = \"{}/test.jpg\".format(BUCKET_URI)\n",
|
||||
"! gsutil cp $test_item $test_image_local\n",
|
||||
"\n",
|
||||
"# Download the test image in bytes format\n",
|
||||
"storage_client = storage.Client(project=PROJECT_ID)\n",
|
||||
"bucket = storage_client.bucket(bucket_name=BUCKET_NAME)\n",
|
||||
"test_content = bucket.get_blob(\"test.jpg\").download_as_bytes()\n",
|
||||
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
|
||||
" content = f.read()\n",
|
||||
"\n",
|
||||
"# The format of each instance should conform to the deployed model's prediction input schema.\n",
|
||||
"instances = [{\"content\": base64.b64encode(test_content).decode(\"utf-8\")}]\n",
|
||||
"instances = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
|
||||
"\n",
|
||||
"prediction = endpoint.predict(instances=instances)\n",
|
||||
"\n",
|
||||
"print(prediction)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3b1b67898533"
|
||||
},
|
||||
"source": [
|
||||
"#### Alternate method using [GFile](https://www.tensorflow.org/api_docs/python/tf/io/gfile/GFile)\n",
|
||||
"\n",
|
||||
"Alternatively, [GFile](https://www.tensorflow.org/api_docs/python/tf/io/gfile/GFile) method from tensorflow-io library can be used to read the data from Cloud storage directly. The following code snippet does the same :\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"import base64\n",
|
||||
"import tensorflow as tf\n",
|
||||
"\n",
|
||||
"# Read the test file using GFile\n",
|
||||
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
|
||||
" content = f.read()\n",
|
||||
"\n",
|
||||
"# The format of each instance should conform to the deployed model's prediction input schema.\n",
|
||||
"instances = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
|
||||
"\n",
|
||||
"prediction = endpoint.predict(instances=instances)\n",
|
||||
"\n",
|
||||
"print(prediction)\n",
|
||||
"```\n",
|
||||
"Nevertheless, `tf.io.gfile.GFile` supports multiple file system implementations, including local files, Google Cloud Storage (using a gs:// prefix), and HDFS (using an hdfs:// prefix)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "undeploy_model:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"#### Undeploy the model\n",
|
||||
"## Undeploy the model\n",
|
||||
"\n",
|
||||
"When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
|
||||
]
|
||||
@@ -991,7 +846,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = model.export_model(\n",
|
||||
" artifact_destination=BUCKET_URI, export_format_id=\"tflite\", sync=True\n",
|
||||
" artifact_destination=BUCKET_NAME, export_format_id=\"tflite\", sync=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model_package = response[\"artifactOutputUri\"]"
|
||||
@@ -1132,10 +987,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.TabularDataset.create(\n",
|
||||
" display_name=\"gsod_\" + TIMESTAMP,\n",
|
||||
"dataset = aip.TabularDataset.create(\n",
|
||||
" display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
|
||||
" bq_source=[IMPORT_FILE],\n",
|
||||
" labels={\"user_metadata\": BUCKET_NAME},\n",
|
||||
" labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\"\n",
|
||||
@@ -1191,7 +1046,9 @@
|
||||
" - regression:\n",
|
||||
" - `minimize-rmse`\n",
|
||||
" - `minimize-mae`\n",
|
||||
" - `minimize-rmsle`"
|
||||
" - `minimize-rmsle`\n",
|
||||
"\n",
|
||||
"The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1202,8 +1059,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dag = aiplatform.AutoMLTabularTrainingJob(\n",
|
||||
" display_name=\"gsod_\" + TIMESTAMP,\n",
|
||||
"dag = aip.AutoMLTabularTrainingJob(\n",
|
||||
" display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" optimization_prediction_type=\"regression\",\n",
|
||||
" optimization_objective=\"minimize-rmse\",\n",
|
||||
" column_transformations=TRANSFORMATIONS,\n",
|
||||
@@ -1220,7 +1077,7 @@
|
||||
"source": [
|
||||
"#### Run the training pipeline\n",
|
||||
"\n",
|
||||
"Next, you run the created DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `dataset`: The `Dataset` resource to train the model.\n",
|
||||
"- `model_display_name`: The human readable name for the trained model.\n",
|
||||
@@ -1246,7 +1103,7 @@
|
||||
"source": [
|
||||
"model = dag.run(\n",
|
||||
" dataset=dataset,\n",
|
||||
" model_display_name=\"gsod_\" + TIMESTAMP,\n",
|
||||
" model_display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" training_fraction_split=0.8,\n",
|
||||
" validation_fraction_split=0.1,\n",
|
||||
" test_fraction_split=0.1,\n",
|
||||
@@ -1277,13 +1134,11 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get model resource ID\n",
|
||||
"models = aiplatform.Model.list(filter=\"display_name=gsod_\" + TIMESTAMP)\n",
|
||||
"models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
|
||||
"\n",
|
||||
"# Get a reference to the Model Service client\n",
|
||||
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
|
||||
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
|
||||
" client_options=client_options\n",
|
||||
")\n",
|
||||
"model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
|
||||
"\n",
|
||||
"model_evaluations = model_service_client.list_model_evaluations(\n",
|
||||
" parent=models[0].resource_name\n",
|
||||
@@ -1322,7 +1177,7 @@
|
||||
"id": "undeploy_model:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"#### Undeploy the model\n",
|
||||
"## Undeploy the model\n",
|
||||
"\n",
|
||||
"When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
|
||||
]
|
||||
@@ -1363,7 +1218,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = model.export_model(\n",
|
||||
" artifact_destination=BUCKET_URI, export_format_id=\"tf-saved-model\", sync=True\n",
|
||||
" artifact_destination=BUCKET_NAME, export_format_id=\"tf-saved-model\", sync=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model_package = response[\"artifactOutputUri\"]"
|
||||
@@ -1495,7 +1350,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"FILE = IMPORT_FILE\n",
|
||||
"if \"IMPORT_FILES\" in globals():\n",
|
||||
" FILE = IMPORT_FILES[0]\n",
|
||||
"else:\n",
|
||||
" FILE = IMPORT_FILE\n",
|
||||
"\n",
|
||||
"count = ! gsutil cat $FILE | wc -l\n",
|
||||
"print(\"Number of Examples\", int(count[0]))\n",
|
||||
@@ -1533,10 +1391,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.TextDataset.create(\n",
|
||||
" display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
"dataset = aip.TextDataset.create(\n",
|
||||
" display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
|
||||
" gcs_source=[IMPORT_FILE],\n",
|
||||
" import_schema_uri=aiplatform.schema.dataset.ioformat.text.single_label_classification,\n",
|
||||
" import_schema_uri=aip.schema.dataset.ioformat.text.single_label_classification,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(dataset.resource_name)"
|
||||
@@ -1562,7 +1420,9 @@
|
||||
" - `sentiment`: A text sentiment analysis model.\n",
|
||||
" - `extraction`: A text entity extraction model.\n",
|
||||
"- `multi_label`: If a classification task, whether single (False) or multi-labeled (True).\n",
|
||||
"- `sentiment_max`: If a sentiment analysis task, the maximum sentiment value.\n"
|
||||
"- `sentiment_max`: If a sentiment analysis task, the maximum sentiment value.\n",
|
||||
"\n",
|
||||
"The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1573,7 +1433,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dag = aiplatform.AutoMLTextTrainingJob(\n",
|
||||
"dag = aip.AutoMLTextTrainingJob(\n",
|
||||
" display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" prediction_type=\"classification\",\n",
|
||||
" multi_label=False,\n",
|
||||
@@ -1590,7 +1450,7 @@
|
||||
"source": [
|
||||
"#### Run the training pipeline\n",
|
||||
"\n",
|
||||
"Next, you run the created DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `dataset`: The `Dataset` resource to train the model.\n",
|
||||
"- `model_display_name`: The human readable name for the trained model.\n",
|
||||
@@ -1641,13 +1501,11 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get model resource ID\n",
|
||||
"models = aiplatform.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
|
||||
"models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
|
||||
"\n",
|
||||
"# Get a reference to the Model Service client\n",
|
||||
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
|
||||
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
|
||||
" client_options=client_options\n",
|
||||
")\n",
|
||||
"model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
|
||||
"\n",
|
||||
"model_evaluations = model_service_client.list_model_evaluations(\n",
|
||||
" parent=models[0].resource_name\n",
|
||||
@@ -1684,7 +1542,7 @@
|
||||
"id": "undeploy_model:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"#### Undeploy the model\n",
|
||||
"## Undeploy the model\n",
|
||||
"\n",
|
||||
"When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
|
||||
]
|
||||
@@ -1828,7 +1686,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"FILE = IMPORT_FILE\n",
|
||||
"if \"IMPORT_FILES\" in globals():\n",
|
||||
" FILE = IMPORT_FILES[0]\n",
|
||||
"else:\n",
|
||||
" FILE = IMPORT_FILE\n",
|
||||
"\n",
|
||||
"count = ! gsutil cat $FILE | wc -l\n",
|
||||
"print(\"Number of Examples\", int(count[0]))\n",
|
||||
@@ -1865,10 +1726,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.VideoDataset.create(\n",
|
||||
" display_name=\"human_motion_\" + TIMESTAMP,\n",
|
||||
"dataset = aip.VideoDataset.create(\n",
|
||||
" display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
|
||||
" gcs_source=[IMPORT_FILE],\n",
|
||||
" import_schema_uri=aiplatform.schema.dataset.ioformat.video.classification,\n",
|
||||
" import_schema_uri=aip.schema.dataset.ioformat.video.classification,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(dataset.resource_name)"
|
||||
@@ -1892,7 +1753,9 @@
|
||||
"- `prediction_type`: The type task to train the model for.\n",
|
||||
" - `classification`: A video classification model.\n",
|
||||
" - `object_tracking`: A video object tracking model.\n",
|
||||
" - `action_recognition`: A video action recognition model."
|
||||
" - `action_recognition`: A video action recognition model.\n",
|
||||
"\n",
|
||||
"The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1903,8 +1766,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dag = aiplatform.AutoMLVideoTrainingJob(\n",
|
||||
" display_name=\"human_motion_\" + TIMESTAMP,\n",
|
||||
"dag = aip.AutoMLVideoTrainingJob(\n",
|
||||
" display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" prediction_type=\"classification\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
@@ -1919,7 +1782,7 @@
|
||||
"source": [
|
||||
"#### Run the training pipeline\n",
|
||||
"\n",
|
||||
"Next, you run the created DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `dataset`: The `Dataset` resource to train the model.\n",
|
||||
"- `model_display_name`: The human readable name for the trained model.\n",
|
||||
@@ -1941,7 +1804,7 @@
|
||||
"source": [
|
||||
"model = dag.run(\n",
|
||||
" dataset=dataset,\n",
|
||||
" model_display_name=\"human_motion_\" + TIMESTAMP,\n",
|
||||
" model_display_name=\"happydb_\" + TIMESTAMP,\n",
|
||||
" training_fraction_split=0.8,\n",
|
||||
" test_fraction_split=0.2,\n",
|
||||
")"
|
||||
@@ -1968,13 +1831,11 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get model resource ID\n",
|
||||
"models = aiplatform.Model.list(filter=\"display_name=human_motion_\" + TIMESTAMP)\n",
|
||||
"models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
|
||||
"\n",
|
||||
"# Get a reference to the Model Service client\n",
|
||||
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
|
||||
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
|
||||
" client_options=client_options\n",
|
||||
")\n",
|
||||
"model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
|
||||
"\n",
|
||||
"model_evaluations = model_service_client.list_model_evaluations(\n",
|
||||
" parent=models[0].resource_name\n",
|
||||
@@ -2038,7 +1899,16 @@
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial.\n"
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2049,11 +1919,66 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"delete_dataset = True\n",
|
||||
"delete_pipeline = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_endpoint = True\n",
|
||||
"delete_batchjob = True\n",
|
||||
"delete_customjob = True\n",
|
||||
"delete_hptjob = True\n",
|
||||
"delete_bucket = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"# Delete the dataset using the Vertex fully qualified identifier for the dataset\n",
|
||||
"try:\n",
|
||||
" if delete_dataset and \"dataset_id\" in globals():\n",
|
||||
" clients[\"dataset\"].delete_dataset(name=dataset_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n",
|
||||
"try:\n",
|
||||
" if delete_pipeline and \"pipeline_id\" in globals():\n",
|
||||
" clients[\"pipeline\"].delete_training_pipeline(name=pipeline_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the model using the Vertex fully qualified identifier for the model\n",
|
||||
"try:\n",
|
||||
" if delete_model and \"model_to_deploy_id\" in globals():\n",
|
||||
" clients[\"model\"].delete_model(name=model_to_deploy_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n",
|
||||
"try:\n",
|
||||
" if delete_endpoint and \"endpoint_id\" in globals():\n",
|
||||
" clients[\"endpoint\"].delete_endpoint(name=endpoint_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the batch job using the Vertex fully qualified identifier for the batch job\n",
|
||||
"try:\n",
|
||||
" if delete_batchjob and \"batch_job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_batch_prediction_job(name=batch_job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the custom job using the Vertex fully qualified identifier for the custom job\n",
|
||||
"try:\n",
|
||||
" if delete_customjob and \"job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_custom_job(name=job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n",
|
||||
"try:\n",
|
||||
" if delete_hptjob and \"hpt_job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_hyperparameter_tuning_job(name=hpt_job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -32,11 +32,6 @@
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Distributed Training\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/ml_ops/stage2/get_started_vertex_distributed_training.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_distributed_training.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
@@ -44,9 +39,8 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_distributed_training.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
"Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_distributed_training.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -62,7 +56,7 @@
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex Distributed Training. Please note: There are incompatibilities between Colab and Docker and the Docker section may not work until resolved by the platform."
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex Distributed Training."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -106,15 +100,6 @@
|
||||
"id": "recommendation:mlops,stage2,vertex,distributed_training"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
" \n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"Vertex AI\n",
|
||||
"Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/),\n",
|
||||
" to generate a cost estimate based on your projected usage.\n",
|
||||
"### Recommendations\n",
|
||||
"\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following are best practices for when to use Vertex AI Distributed Training:\n",
|
||||
@@ -141,64 +126,59 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XkYpRvOQyVYb"
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"source": [
|
||||
"### Install additional packages\n",
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the latest version of Vertex SDK for Python."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "xs_Kt8RcyXTC"
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "TjOXHg2VyajN"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "oQhwq1iozAxh"
|
||||
"id": "restart"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "zo3YFZXLzCRJ"
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
@@ -228,8 +208,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -283,7 +261,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "qohAA9fJulvP"
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -305,7 +283,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "8NKwwe7aulvQ"
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -314,81 +292,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "poKeKYG8ulvQ"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "MIpJGzF9ulvQ"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Vh6KDXB5ulvQ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -440,7 +343,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Moosy2rOulvR"
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -460,7 +363,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "56irx2CvulvS"
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -505,7 +408,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "wbvYPSTDulvS"
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -538,7 +441,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "PryARdnoulvT"
|
||||
"id": "accelerators:training,prediction,ngpu,mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -580,7 +483,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LhhUFw2nulvT"
|
||||
"id": "container:training,prediction"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -648,7 +551,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "vytMaukeulvT"
|
||||
"id": "machine:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -710,7 +613,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "mhw34XoOulvU"
|
||||
"id": "create_custom_pp_training_job:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -759,7 +662,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "IAaZpZyyulvU"
|
||||
"id": "examine_training_package"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -808,7 +711,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "zKzddzl6ulvV"
|
||||
"id": "taskpy_contents:mirrored,boston"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -864,13 +767,6 @@
|
||||
" strategy = tf.distribute.MultiWorkerMirroredStrategy()\n",
|
||||
" logging.info(\"Multi-worker Strategy distributed training\")\n",
|
||||
" logging.info('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n",
|
||||
" # Single Machine, multiple TPU devices\n",
|
||||
"elif args.distribute == 'tpu':\n",
|
||||
" cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=\"local\")\n",
|
||||
" tf.config.experimental_connect_to_cluster(cluster_resolver)\n",
|
||||
" tf.tpu.experimental.initialize_tpu_system(cluster_resolver)\n",
|
||||
" strategy = tf.distribute.TPUStrategy(cluster_resolver)\n",
|
||||
" print(\"All devices: \", tf.config.list_logical_devices('TPU'))\n",
|
||||
"\n",
|
||||
"logging.info('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n",
|
||||
"\n",
|
||||
@@ -929,11 +825,8 @@
|
||||
" else:\n",
|
||||
" task_type, task_id = None, None\n",
|
||||
"\n",
|
||||
" if args.distribute==\"tpu\":\n",
|
||||
" save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n",
|
||||
" model.save(args.model_dir, options=save_locally)\n",
|
||||
" # single, mirrored or primary for multiworker\n",
|
||||
" elif _is_chief(task_type, task_id):\n",
|
||||
" if _is_chief(task_type, task_id):\n",
|
||||
" model.save(args.model_dir)\n",
|
||||
" # non-primary workers for multi-workers\n",
|
||||
" else:\n",
|
||||
@@ -967,7 +860,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LFUHioqTulvV"
|
||||
"id": "tarball_training_script"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -992,7 +885,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LnUX0UkvulvV"
|
||||
"id": "run_custom_pp_training_job:mirrored"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1027,7 +920,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "iUWHFpPoulvW"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1049,7 +942,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-0gqCUTEulvW"
|
||||
"id": "model_delete:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1134,7 +1027,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "aXvPN8P6ulvX"
|
||||
"id": "create_custom_pp_training_job:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"### Create and run custom training job\n",
|
||||
@@ -1160,7 +1053,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "kYcFsVSEulvX"
|
||||
"id": "create_custom_pp_training_job:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1191,7 +1084,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "GHRxPU32ulvX"
|
||||
"id": "run_custom_pp_training_job:multiworker"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1218,7 +1111,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "92D_hbuVulvX"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"source": [
|
||||
"### Delete a custom training job\n",
|
||||
@@ -1230,7 +1123,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "CqrfWkB3ulvX"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1282,13 +1175,14 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "pGI2viDAulvY"
|
||||
"id": "write_docker_file:training,multiworker"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile custom/Dockerfile\n",
|
||||
"\n",
|
||||
"FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5\n",
|
||||
"WORKDIR /root\n",
|
||||
"\n",
|
||||
"WORKDIR /\n",
|
||||
"\n",
|
||||
@@ -1314,7 +1208,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7P8cdlFtulvY"
|
||||
"id": "name_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1334,7 +1228,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "jmw5cakNulvY"
|
||||
"id": "build_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1356,7 +1250,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "jJGLjU-TulvZ"
|
||||
"id": "test_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1378,7 +1272,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "GAXGjae7ulvZ"
|
||||
"id": "register_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1402,7 +1296,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "CEAnXBzCulvZ"
|
||||
"id": "worker_pool_primary"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1445,7 +1339,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "6dchPSfNulvZ"
|
||||
"id": "worker_pool_training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1481,7 +1375,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "m2VgmqEOulva"
|
||||
"id": "custom_job:worker_pool"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1505,7 +1399,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "hg8vnI_Wulva"
|
||||
"id": "run_custom_job:multiworker"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1519,7 +1413,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WT76Sc-culva"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"source": [
|
||||
"### Delete a custom training job\n",
|
||||
@@ -1531,7 +1425,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "I_IxVfuDulva"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1580,7 +1474,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "L8Av8ATVulvb"
|
||||
"id": "custom_job:worker_pool"
|
||||
},
|
||||
"source": [
|
||||
"### Create CustomJob with worker pool specifications\n",
|
||||
@@ -1596,7 +1490,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "TUWEP1Lmulvb"
|
||||
"id": "custom_job:worker_pool"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1608,7 +1502,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "_95FH8jeulvb"
|
||||
"id": "run_custom_job:multiworker"
|
||||
},
|
||||
"source": [
|
||||
"### Run the CustomJob\n",
|
||||
@@ -1620,7 +1514,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "IEbrY05Gulvb"
|
||||
"id": "run_custom_job:multiworker"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1634,7 +1528,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8R2Bnmwmulvb"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"source": [
|
||||
"### Delete a custom training job\n",
|
||||
@@ -1646,7 +1540,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "s1geVE3Lulvb"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1689,14 +1583,14 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "nQVPtknpulvb"
|
||||
"id": "docker_write:tpu"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile custom/Dockerfile\n",
|
||||
"FROM python:3.8\n",
|
||||
"\n",
|
||||
"WORKDIR /\n",
|
||||
"WORKDIR /root\n",
|
||||
"\n",
|
||||
"# Copies the trainer code to the docker image.\n",
|
||||
"COPY trainer /trainer\n",
|
||||
@@ -1728,11 +1622,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "J_d_zEXUulvc"
|
||||
"id": "docker_push:tpu"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRAIN_IMAGE = \"gcr.io/\" + PROJECT_ID + \"/tpu-train:latest\"\n",
|
||||
"TRAIN_IMAGE = f\"gcr.io/\" + PROJECT_ID + \"/tpu-train:latest\"\n",
|
||||
"\n",
|
||||
"os.chdir(\"custom\")\n",
|
||||
"! docker build --quiet --tag={TRAIN_IMAGE} .\n",
|
||||
@@ -1759,7 +1653,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "d514eU7lulvc"
|
||||
"id": "worker_pool_tpu"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1807,7 +1701,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "RruSqNfrulvc"
|
||||
"id": "custom_job:worker_pool"
|
||||
},
|
||||
"source": [
|
||||
"### Create CustomJob with worker pool specifications\n",
|
||||
@@ -1823,7 +1717,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2QvSqbbHulvc"
|
||||
"id": "custom_job:worker_pool"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1835,7 +1729,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Iw4L3UIfulvd"
|
||||
"id": "run_custom_job:multiworker"
|
||||
},
|
||||
"source": [
|
||||
"### Run the CustomJob\n",
|
||||
@@ -1847,7 +1741,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "zmqCNS78ulvd"
|
||||
"id": "run_custom_job:multiworker"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1861,7 +1755,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "gWZoH9QKulvd"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"source": [
|
||||
"### Delete a custom training job\n",
|
||||
@@ -1873,7 +1767,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Lt8BJ4iBulvd"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1893,7 +1787,13 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1901,15 +1801,70 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "U98Wzc01ulvd"
|
||||
"id": "cleanup"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"delete_dataset = True\n",
|
||||
"delete_pipeline = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_endpoint = True\n",
|
||||
"delete_batchjob = True\n",
|
||||
"delete_customjob = True\n",
|
||||
"delete_hptjob = True\n",
|
||||
"delete_bucket = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"# Delete the dataset using the Vertex fully qualified identifier for the dataset\n",
|
||||
"try:\n",
|
||||
" if delete_dataset and \"dataset_id\" in globals():\n",
|
||||
" clients[\"dataset\"].delete_dataset(name=dataset_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n",
|
||||
"try:\n",
|
||||
" if delete_pipeline and \"pipeline_id\" in globals():\n",
|
||||
" clients[\"pipeline\"].delete_training_pipeline(name=pipeline_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the model using the Vertex fully qualified identifier for the model\n",
|
||||
"try:\n",
|
||||
" if delete_model and \"model_to_deploy_id\" in globals():\n",
|
||||
" clients[\"model\"].delete_model(name=model_to_deploy_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n",
|
||||
"try:\n",
|
||||
" if delete_endpoint and \"endpoint_id\" in globals():\n",
|
||||
" clients[\"endpoint\"].delete_endpoint(name=endpoint_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the batch job using the Vertex fully qualified identifier for the batch job\n",
|
||||
"try:\n",
|
||||
" if delete_batchjob and \"batch_job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_batch_prediction_job(name=batch_job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the custom job using the Vertex fully qualified identifier for the custom job\n",
|
||||
"try:\n",
|
||||
" if delete_customjob and \"job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_custom_job(name=job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n",
|
||||
"try:\n",
|
||||
" if delete_hptjob and \"hpt_job_id\" in globals():\n",
|
||||
" clients[\"job\"].delete_hyperparameter_tuning_job(name=hpt_job_id)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -29,7 +29,7 @@
|
||||
"id": "title:generic,gcp"
|
||||
},
|
||||
"source": [
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Logging and Vertex AI Experiments\n",
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Logging and Vertex Experiments\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
@@ -38,15 +38,9 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -62,7 +56,7 @@
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Logging and Vertex AI Experiments."
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Logging and Vertex Experiments."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -99,7 +93,7 @@
|
||||
"source": [
|
||||
"### Recommendations\n",
|
||||
"\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following are some of the best practices for logging data when experimenting or formally training a model.\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following best practices for logging data when experimenting or formal training a model.\n",
|
||||
"\n",
|
||||
"#### Python Logging\n",
|
||||
"\n",
|
||||
@@ -111,14 +105,7 @@
|
||||
"\n",
|
||||
"#### Experiments\n",
|
||||
"\n",
|
||||
"Use Vertex AI Experiments in conjunction with logging when performing experiments to compare results for different experiment configurations.\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
"Use Vertex AI Experiments in conjunction with logging when doing experiments to compare results for different experiment configurations."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -129,7 +116,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages for executing this notebook."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -140,17 +127,20 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-logging $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -188,24 +178,6 @@
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI, Compute Engine, Cloud Storage and Cloud Logging APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage_component,logging).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
@@ -276,10 +248,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -306,63 +275,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f3bd8c0d0469"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex AI\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e0953a00668e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -372,7 +284,7 @@
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries"
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -383,9 +295,29 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging\n",
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_logging"
|
||||
},
|
||||
"source": [
|
||||
"#### Import logging\n",
|
||||
"\n",
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
"Import the logging package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_logging"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -407,7 +339,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION)"
|
||||
"aip.init(project=PROJECT_ID, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -424,9 +356,9 @@
|
||||
"- Send log output to console.\n",
|
||||
"- Send log output to a file.\n",
|
||||
"\n",
|
||||
"### Logging Levels in Python Logging\n",
|
||||
"### Logging Levels\n",
|
||||
"\n",
|
||||
"The logging levels in order (from least to highest) and each level inclusive of the previous level are :\n",
|
||||
"The logging levels in order (from least to highest) are, with each level inclusive of the previous level:\n",
|
||||
"\n",
|
||||
"1. Informational\n",
|
||||
"2. Warnings\n",
|
||||
@@ -466,7 +398,7 @@
|
||||
"source": [
|
||||
"### Setting logging level\n",
|
||||
"\n",
|
||||
"To set the logging level, you get the logging handler using `getLogger()`. You can have multiple logging handles. When `getLogger()` is called without any arguments, it gets the default handler named ROOT. With the handler, you set the logging level with the method `setLevel()`."
|
||||
"To set the logging level, you get the logging handler using `getLogger()`. You can have multiple logging handles. When `getLogger()` is called w/o arguments it gets the default handler, named ROOT. With the handler, you set the logging level with the method 'setLevel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -513,7 +445,7 @@
|
||||
"source": [
|
||||
"### Output to a local file\n",
|
||||
"\n",
|
||||
"You can preserve your logging output to a file that is local to where the Python script is running with the method `BasicConfig()`, that takes the following parameters:\n",
|
||||
"You can preserve your logging output to a file that is local to where the Python script is running with the method `BasicConfig()`, with the following paraneters:\n",
|
||||
"\n",
|
||||
"- `filename`: The file path to the local file to write the log output to.\n",
|
||||
"- `level`: Sets the level of logging that is written to the logging file.\n",
|
||||
@@ -550,7 +482,7 @@
|
||||
"- Send log output to storage.\n",
|
||||
"- Retrieve log output from storage.\n",
|
||||
"\n",
|
||||
"### Logging Levels in Cloud Logging\n",
|
||||
"### Logging Levels\n",
|
||||
"\n",
|
||||
"The logging levels in order (from least to highest) are, with each level inclusive of the previous level:\n",
|
||||
"\n",
|
||||
@@ -585,7 +517,7 @@
|
||||
"from google.cloud.logging.handlers import CloudLoggingHandler\n",
|
||||
"\n",
|
||||
"# Connect to the Cloud Logging service\n",
|
||||
"cl_client = google.cloud.logging.Client(project=PROJECT_ID)\n",
|
||||
"cl_client = google.cloud.logging.Client()\n",
|
||||
"handler = CloudLoggingHandler(cl_client, name=\"mylog\")\n",
|
||||
"\n",
|
||||
"# Create a logger instance and logging level\n",
|
||||
@@ -607,7 +539,7 @@
|
||||
"source": [
|
||||
"### Logging output\n",
|
||||
"\n",
|
||||
"Logging output at specific levels is identical to Python logging with respect to method and method names. The only difference is that you use your instance of the cloud logger in place of logging."
|
||||
"To log output at specific levels is identical in method, and method names, as in Python logging, except that you use your instance of the cloud logger in place of logging."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -635,7 +567,7 @@
|
||||
"To get the logged output, you:\n",
|
||||
"\n",
|
||||
"1. Retrieve the log handle to the service.\n",
|
||||
"2. Using the handle, call the method `list_entries()`.\n",
|
||||
"2. Using the handle call the method `list_entries()`\n",
|
||||
"3. Iterate through the entries."
|
||||
]
|
||||
},
|
||||
@@ -662,10 +594,10 @@
|
||||
"source": [
|
||||
"## Logging with Vertex AI Experiments and Vertex AI ML Metadata\n",
|
||||
"\n",
|
||||
"You can log results related to training experiments with `Vertex AI Experiments` and `ML Metadata` including:\n",
|
||||
"You can log results related to training experiments with `Vertex AI Experiments` and `ML Metadata`:\n",
|
||||
"\n",
|
||||
"- Preserve results of an experiment.\n",
|
||||
"- Track multiple runs i.e., training runs within an experiment.\n",
|
||||
"- Track multiple runs -- i.e., training runs -- within an experiment.\n",
|
||||
"- Track parameters (configuration) and metrics (results).\n",
|
||||
"- Retrieve and display the logged output.\n",
|
||||
"\n",
|
||||
@@ -680,29 +612,14 @@
|
||||
"source": [
|
||||
"### Create experiment for tracking training related metadata\n",
|
||||
"\n",
|
||||
"Setup tracking for parameters (configuration) and metrics (results) in each experiment:\n",
|
||||
"Setup tracking the parameters (configuration) and metrics (results) for each experiment:\n",
|
||||
"\n",
|
||||
"- `aiplatform.init()` - Create an experiment instance\n",
|
||||
"- `aiplatform.start_run()` - Track a specific run within the experiment.\n",
|
||||
"- `aip.init()` - Create an experiment instance\n",
|
||||
"- `aip.start_run()` - Track a specific run within the experiment.\n",
|
||||
"\n",
|
||||
"Learn more about [Introduction to Vertex AI ML Metadata](https://cloud.google.com/vertex-ai/docs/ml-metadata/introduction)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1ed46e349cf2"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Specify a name for the experiment\n",
|
||||
"EXPERIMENT_NAME = \"[your-experiment-name]\"\n",
|
||||
"\n",
|
||||
"if EXPERIMENT_NAME == \"[your-experiment-name]\":\n",
|
||||
" EXPERIMENT_NAME = \"example-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -711,9 +628,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create experiment\n",
|
||||
"aiplatform.init(experiment=EXPERIMENT_NAME)\n",
|
||||
"aiplatform.start_run(\"run-1\")"
|
||||
"EXPERIMENT_NAME = \"example-\" + TIMESTAMP\n",
|
||||
"aip.init(experiment=EXPERIMENT_NAME)\n",
|
||||
"aip.start_run(\"run-1\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -724,14 +641,14 @@
|
||||
"source": [
|
||||
"### Log parameters for the experiment\n",
|
||||
"\n",
|
||||
"Typically, an experiment is associated with a specific dataset and a model architecture. Within an experiment, you may have multiple training runs, where each run tries a different configuration. For example:\n",
|
||||
"Typically, an experiment is associated with a specific dataset and model architecture. Within an experiment, you may have multiple training runs, where each run tries a different configuration. As examples:\n",
|
||||
"\n",
|
||||
"- Dataset split\n",
|
||||
"- Dataset sampling and boosting\n",
|
||||
"- Depth and width of layers\n",
|
||||
"- Hyperparameters\n",
|
||||
"\n",
|
||||
"These configuration settings are referred to as parameters, which you store as key-value pairs using the method `log_params()`"
|
||||
"These configuration settings are referred to as parameters, which you store their key/value pair using the method `log_params()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -746,7 +663,7 @@
|
||||
"hyperparams[\"epochs\"] = 100\n",
|
||||
"hyperparams[\"batch_size\"] = 32\n",
|
||||
"hyperparams[\"learning_rate\"] = 0.01\n",
|
||||
"aiplatform.log_params(hyperparams)"
|
||||
"aip.log_params(hyperparams)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -757,14 +674,14 @@
|
||||
"source": [
|
||||
"### Log metrics for the experiment\n",
|
||||
"\n",
|
||||
"At the completion or termination of a run within an experiment, you can log results that you use to compare runs. For example:\n",
|
||||
"At the completion, or termination, of a run within an experiment, you can log results that you use to compare runs. As examples:\n",
|
||||
"\n",
|
||||
"- Evaluation metrics\n",
|
||||
"- Hyperparameter search selection\n",
|
||||
"- Time to train the model\n",
|
||||
"- Early stop trigger\n",
|
||||
"\n",
|
||||
"These results are referred to as metrics, which you store as key-value pairs using the method `log_metrics()`"
|
||||
"These results settings are referred to as metrics, which you store their key/value pair using the method `log_metrics()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -778,7 +695,7 @@
|
||||
"metrics = {}\n",
|
||||
"metrics[\"test_acc\"] = 98.7\n",
|
||||
"metrics[\"train_acc\"] = 99.3\n",
|
||||
"aiplatform.log_metrics(metrics)"
|
||||
"aip.log_metrics(metrics)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -800,11 +717,36 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"experiment_df = aiplatform.get_experiment_df()\n",
|
||||
"EXPERIMENT_NAME = \"example\"\n",
|
||||
"\n",
|
||||
"experiment_df = aip.get_experiment_df()\n",
|
||||
"experiment_df = experiment_df[experiment_df.experiment_name == EXPERIMENT_NAME]\n",
|
||||
"experiment_df.T"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "delete_experiment"
|
||||
},
|
||||
"source": [
|
||||
"### Delete the experiment\n",
|
||||
"\n",
|
||||
"Next, delete the experiment. You will need to get the context via the metadata to delete it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "delete_experiment"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"c = aiplatform.metadata._Context(EXPERIMENT_NAME)\n",
|
||||
"c.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -818,9 +760,15 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"### Delete the experiment\n",
|
||||
"\n",
|
||||
"Next, delete the experiment. You will need to get the context via the metadata to delete it."
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -831,8 +779,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"c = aiplatform.metadata._Context(EXPERIMENT_NAME)\n",
|
||||
"c.delete()"
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -38,20 +38,11 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_feature_store.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_feature_store.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_feature_store.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" \n",
|
||||
"</table>\n",
|
||||
"<br/><br/><br/>"
|
||||
]
|
||||
@@ -76,9 +67,9 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the `Movie Recommendations` dataset. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket, in Avro format.\n",
|
||||
"The dataset used for this tutorial is the Movie Recommendations. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket, in Avro format.\n",
|
||||
"\n",
|
||||
"This dataset is used to predict whether a person will watch a movie or not."
|
||||
"The dataset predicts whether a persons will watch a movie."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -89,7 +80,7 @@
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to use `Vertex AI Feature Store` when training and predicting with `Vertex AI`.\n",
|
||||
"In this tutorial, you learn how to use `Vertex AI Feature Store` for when training and prediction with `Vertex AI`.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
@@ -107,22 +98,6 @@
|
||||
"- Perform batch serving from a `Featurestore` resource."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "81c777b8ad32"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"- BigQuery\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -131,7 +106,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages for further running this notebook."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -142,18 +117,24 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"# Install the dependecies\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform google-cloud-bigquery pyarrow avro $USER_FLAG -q"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
" ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -261,10 +242,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -291,69 +269,15 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "29b110b44457"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "89788a802687"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
@@ -365,7 +289,28 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform\n",
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_bq"
|
||||
},
|
||||
"source": [
|
||||
"#### Import BigQuery\n",
|
||||
"\n",
|
||||
"Import the BigQuery package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_bq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import bigquery"
|
||||
]
|
||||
},
|
||||
@@ -375,7 +320,9 @@
|
||||
"id": "init_bq"
|
||||
},
|
||||
"source": [
|
||||
"Initialize Vertex AI and BigQuery clients."
|
||||
"### Create BigQuery client\n",
|
||||
"\n",
|
||||
"Create the BigQuery client."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -386,8 +333,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID)\n",
|
||||
"bqclient = bigquery.Client(project=PROJECT_ID)"
|
||||
"bqclient = bigquery.Client()"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -404,11 +350,11 @@
|
||||
"\n",
|
||||
"Now it's time to do a live prediction. You get a transaction from the cash register, but all it has is the credit card number and this transaction. It does not have the enriched data the model needs. During serving, the credit card number is used as an index to Feature Store to get the enriched data needed for the model.\n",
|
||||
"\n",
|
||||
"On the other hand, let's say the enriched data the model was trained on was timestamped on June 1st. The current transaction is from June 15th. Assume that the user has made other transactions between June 1st and 15th, and the enriched data has been continuously updated in Feature Store. But the model was trained on June 1st data. FeatureStore knows the version number and serves the June 1st version to the model (not the current June 15th). Otherwise, if you used June 15th data, you would have training-serving skew.\n",
|
||||
"Next problem. Let's say the enriched data the model was trained on was timestamp June 1. This transaction is June 15. Assume that the user has made other transactions between June 1 and 15, and the enriched data has been continuously updated in Feature Store. But the model was trained on June 1st data. FeatureStore knows the version number and serves the June 1 version to the model (not the current June 15); otherwise, if you used June 15 data you have training-serving skew.\n",
|
||||
"\n",
|
||||
"Another problem here is the data drift. Things change and suddenly one day, everybody is buying toilet paper! There is a significant change in the distribution of existing enriched data from the distribution that the deployed model was trained on. FeatureStore can detect changes/thresholds in distribution changes and trigger a notification for retraining the model.\n",
|
||||
"Next problem, data drift. Things change, suddenly one day everybody is buying toilet paper! There is a significant change in the distribution of the current stored enriched data from the distribution that the deployed model was trained on. FeatureStore can detect changes/thresholds in distribution changes and trigger a notification for retraining the model.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Feature Store API](https://cloud.google.com/vertex-ai/docs/featurestore)."
|
||||
"Learn more about [Vertex AI Feature Store API](https://cloud.google.com/vertex-ai/docs/featurestore)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -423,9 +369,9 @@
|
||||
"\n",
|
||||
" Featurestore -> EntityType -> Feature\n",
|
||||
"\n",
|
||||
"- `Featurestore`: the place to store your features.\n",
|
||||
"- `Featurestore`: the place to store your features\n",
|
||||
"- `EntityType`: under a `Featurestore`, an `EntityType` describes an object to be modeled, real one or virtual one.\n",
|
||||
"- `Feature`: under an `EntityType`, a `Feature` describes an attribute of the `EntityType`.\n",
|
||||
"- `Feature`: under an `EntityType`, a `Feature` describes an attribute of the `EntityType`\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Feature Store data model](https://cloud.google.com/vertex-ai/docs/featurestore/concepts).\n",
|
||||
"\n",
|
||||
@@ -461,7 +407,7 @@
|
||||
"# Represents featurestore resource path.\n",
|
||||
"FEATURESTORE_NAME = \"movies\"\n",
|
||||
"\n",
|
||||
"featurestore = aiplatform.Featurestore.create(\n",
|
||||
"featurestore = aip.Featurestore.create(\n",
|
||||
" featurestore_id=FEATURESTORE_NAME,\n",
|
||||
" online_store_fixed_node_count=1,\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
@@ -490,7 +436,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for featurestore in aiplatform.Featurestore.list():\n",
|
||||
"for featurestore in aip.Featurestore.list():\n",
|
||||
" print(featurestore)"
|
||||
]
|
||||
},
|
||||
@@ -517,7 +463,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"featurestore = featurestore = aiplatform.Featurestore(\n",
|
||||
"featurestore = featurestore = aip.Featurestore(\n",
|
||||
" featurestore_name=FEATURESTORE_NAME, project=PROJECT_ID, location=REGION\n",
|
||||
")\n",
|
||||
"print(featurestore)"
|
||||
@@ -576,7 +522,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def create_features(featurestore_name, entity_name, features):\n",
|
||||
" entity_type = aiplatform.EntityType(\n",
|
||||
" entity_type = aip.EntityType(\n",
|
||||
" entity_type_name=entity_name, featurestore_id=featurestore_name\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
@@ -627,7 +573,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for featurestore in aiplatform.Featurestore.list():\n",
|
||||
"for featurestore in aip.Featurestore.list():\n",
|
||||
" print(featurestore)"
|
||||
]
|
||||
},
|
||||
@@ -639,7 +585,7 @@
|
||||
"source": [
|
||||
"### Search `Feature` resources using a filter\n",
|
||||
"\n",
|
||||
"You can narrow your search of `Feature` resources using the method `list_features()` and specifying a `filter` string."
|
||||
"You can narrow your search of `Feature` resources using the method `list_features()` and specifying a `filter` filter."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -691,26 +637,17 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"features = aiplatform.Feature.search(query=\"value_type=DOUBLE\")\n",
|
||||
"features = aip.Feature.search(query=\"value_type=DOUBLE\")\n",
|
||||
"print(\"By data type\")\n",
|
||||
"for feature in features:\n",
|
||||
" print(features)\n",
|
||||
"\n",
|
||||
"aiplatform.Feature.search(query=\"feature_id=title\")\n",
|
||||
"aip.Feature.search(query=\"feature_id=title\")\n",
|
||||
"print(\"By Name\")\n",
|
||||
"for feature in features:\n",
|
||||
" print(features)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "277e9884cf37"
|
||||
},
|
||||
"source": [
|
||||
"Define paths to the feature data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -740,7 +677,7 @@
|
||||
"\n",
|
||||
"### Data layout\n",
|
||||
"\n",
|
||||
"Each imported `EntityType` resource data must have an ID. Also, each `EntityType` resource data item can optionally have a timestamp, sepecifying when the feature values were generated.\n",
|
||||
"Each imported `EntityType` resource data must have an ID; also, each `EntityType` resource data item can optionally have a timestamp, sepecifying when the feature values were generated.\n",
|
||||
"\n",
|
||||
"When importing, specify the following in your request:\n",
|
||||
"\n",
|
||||
@@ -748,7 +685,7 @@
|
||||
"- Data source URL\n",
|
||||
"- Destination: featurestore/entity types/features to be imported\n",
|
||||
"\n",
|
||||
"The feature values for `Movie Recommendations` dataset are in Avro format. The Avro schemas are as follows:\n",
|
||||
"The feature values for the movies dataset are in Avro format. The Avro schemas are as follows:\n",
|
||||
"\n",
|
||||
"**Users entity**:\n",
|
||||
"\n",
|
||||
@@ -860,7 +797,7 @@
|
||||
"source": [
|
||||
"#### Delete the entity types and corresponding features and feature values\n",
|
||||
"\n",
|
||||
"Now, in preparation to repeat the process of importing feature values but from a dataframe this time, you delete the existing entity types, and the corresponding content."
|
||||
"Next, in preparation to repeat importing feature values from a dataframe, you first delete the existing entity types, and corresponding content."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -885,7 +822,7 @@
|
||||
"source": [
|
||||
"## Create entity types for your `Featurestore` resource\n",
|
||||
"\n",
|
||||
"Next, you create the `EntityType` resources again for your `Featurestore` resource using the `create_entity_type()` method, with the following parameters:\n",
|
||||
"Next, you create the `EntityType` resources for your `Featurestore` resource using the `create_entity_type()` method, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `entity_type_id`: The name of the `EntityType` resource.\n",
|
||||
"- `description`: A description of the entity type."
|
||||
@@ -914,7 +851,7 @@
|
||||
"source": [
|
||||
"### Add `Feature` resources for your `EntityType` resources\n",
|
||||
"\n",
|
||||
"Further, you create the `Feature` resources again for each of the `EntityType` resources in your `Featurestore` resource using the `create_feature()` method, with the following parameters:\n",
|
||||
"Next, you create the `Feature` resources for each of the `EntityType` resources in your `Featurestore` resource using the `create_feature()` method, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `feature_id`: The name of the `Feature` resource.\n",
|
||||
"- `description`: A description of the feature.\n",
|
||||
@@ -930,7 +867,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def create_features(featurestore_name, entity_name, features):\n",
|
||||
" entity_type = aiplatform.EntityType(\n",
|
||||
" entity_type = aip.EntityType(\n",
|
||||
" entity_type_name=entity_name, featurestore_id=featurestore_name\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
@@ -962,15 +899,6 @@
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8715a3f719c8"
|
||||
},
|
||||
"source": [
|
||||
"Now, copy the `users` and `movies` data into avro files."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -1113,7 +1041,7 @@
|
||||
"source": [
|
||||
"## Batch Serving\n",
|
||||
"\n",
|
||||
"The Vertex AI Feature Store's batch serving service is optimized for serving large batches of features in real-time with high throughput, typically for training a model or batch prediction.\n",
|
||||
"The Vertex AI Feature Store batch serving service is optimized for serving large batches of features in real-time with high-throughput, typically for training a model or batch prediction.\n",
|
||||
"\n",
|
||||
"One can batch serve to the following destinations:\n",
|
||||
"\n",
|
||||
@@ -1165,7 +1093,7 @@
|
||||
"\n",
|
||||
"You batch serve entity data items to a BigQuery table using the `read_serve_to_bq()` method, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `bq_destination_output_uri`: The destination BigQuery table to receive the served features.\n",
|
||||
"- `bq_destination_output_uri`: The destination BigQuery table to serve the features to.\n",
|
||||
"- `serving_feature_ids`: A dictionary of entity type and corresponding features to serve.\n",
|
||||
"- `read_instances_uri`: A Cloud Storage location to read the entity data items from.\n",
|
||||
"\n",
|
||||
@@ -1198,7 +1126,6 @@
|
||||
"id": "delete_bq_dataset"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"### Delete a BigQuery dataset\n",
|
||||
"\n",
|
||||
"Use the method `delete_dataset()` to delete a BigQuery dataset along with all its tables, by setting the parameter `delete_contents` to `True`."
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -32,11 +32,6 @@
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Tensorboard\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_tensorboard.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_tensorboard.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
@@ -44,9 +39,8 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_tensorboard.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -87,75 +81,6 @@
|
||||
"- Using Vertex AI TensorBoard with Vertex AI Training."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "b132d4ef86d6"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "94a148f11da5"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
|
||||
"all the requirements to run this notebook. You can skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "56cb7f08a9e8"
|
||||
},
|
||||
"source": [
|
||||
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
|
||||
"You need the following:\n",
|
||||
"\n",
|
||||
"* The Google Cloud SDK\n",
|
||||
"* Git\n",
|
||||
"* Python 3\n",
|
||||
"* virtualenv\n",
|
||||
"* Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"The Google Cloud guide to [Setting up a Python development\n",
|
||||
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
|
||||
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
|
||||
"for meeting these requirements. The following steps provide a condensed set of\n",
|
||||
"instructions:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
|
||||
"\n",
|
||||
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
|
||||
"\n",
|
||||
"1. [Install\n",
|
||||
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
|
||||
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
|
||||
"\n",
|
||||
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
|
||||
"command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. Open this notebook in the Jupyter Notebook Dashboard.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -164,7 +89,7 @@
|
||||
"source": [
|
||||
"### Recommendations\n",
|
||||
"\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following are the best practices for visualizing your training with TensorBoard.\n",
|
||||
"When doing E2E MLOps on Google Cloud, the following best practices for visualizing your training with TensorBoard.\n",
|
||||
"\n",
|
||||
"#### Local TensorBoard\n",
|
||||
"\n",
|
||||
@@ -190,25 +115,6 @@
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "020040f91150"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -217,8 +123,20 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U tensorflow==2.8 $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -250,39 +168,6 @@
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d6a00c14b087"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2721ef0202d9"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -386,81 +271,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2700e693f1b3"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "885395904904"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "eff327d0552b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -484,7 +294,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -495,8 +305,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -516,7 +326,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -536,7 +346,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -576,7 +386,7 @@
|
||||
"):\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
@@ -600,7 +410,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -644,7 +454,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -674,15 +484,13 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (\n",
|
||||
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -855,9 +663,9 @@
|
||||
"\n",
|
||||
"You can upload your TensorBoard logs and share with others using `tensorboard dev` command. Once uploaded, a URL is returned to open up the TensorBoard instance in a brower for visualizing.\n",
|
||||
"\n",
|
||||
"*Note:* Your TensorBoard instance is publicly visible.\n",
|
||||
"*Note:* Your TensorBoard instance is publicly visable.\n",
|
||||
"\n",
|
||||
"*Note:* This cell is for demonstration purposes and must be ran in a terminal shell. In this example, while running within a notebook, the command will freeze since it is waiting for an interactive yes/no input. You can kill the command with a Ctrl C or kernel interupt.\n",
|
||||
"*Note:* In this example, while running within a notebook, the command will freeze since it is waiting for an interactive yes/no input. You can kill the command with a Ctrl C or kernel interupt.\n",
|
||||
"\n",
|
||||
"Learn more about [What is TensorBoard.dev](https://tensorboard.dev/)."
|
||||
]
|
||||
@@ -870,7 +678,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! tensorboard dev upload --logdir logs \\\n",
|
||||
"! tensorboard dev upload --logdir {LOG_DIR} \\\n",
|
||||
" --name \"Simple experiment with MNIST\" \\\n",
|
||||
" --description \"Training results\" \\\n",
|
||||
" --one_shot"
|
||||
@@ -898,7 +706,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TENSORBOARD_DISPLAY_NAME = \"example\"\n",
|
||||
"tensorboard = aiplatform.Tensorboard.create(display_name=TENSORBOARD_DISPLAY_NAME)\n",
|
||||
"tensorboard = aip.Tensorboard.create(display_name=TENSORBOARD_DISPLAY_NAME)\n",
|
||||
"tensorboard_resource_name = tensorboard.gca_resource.name\n",
|
||||
"print(\"TensorBoard resource name:\", tensorboard_resource_name)"
|
||||
]
|
||||
@@ -938,9 +746,9 @@
|
||||
"\n",
|
||||
"url = output[1].split(' ')[-1]\n",
|
||||
"\n",
|
||||
"#print(url)\n",
|
||||
"print(url)\n",
|
||||
"\n",
|
||||
"from IPython.display import display, HTML\n",
|
||||
"from IPython.core.display import display, HTML\n",
|
||||
"display(HTML(\"<a href='\" + url + \"'>click here for TensorBoard instance</a>\"))"
|
||||
]
|
||||
},
|
||||
@@ -1145,7 +953,7 @@
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_example.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_example.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1177,7 +985,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job = aiplatform.CustomTrainingJob(\n",
|
||||
"job = aip.CustomTrainingJob(\n",
|
||||
" display_name=\"example_\" + TIMESTAMP,\n",
|
||||
" script_path=\"custom/trainer/task.py\",\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
@@ -1213,7 +1021,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
|
||||
"\n",
|
||||
"EPOCHS = 20\n",
|
||||
"STEPS = 100\n",
|
||||
@@ -1335,8 +1143,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1348,14 +1162,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete the custom training job\n",
|
||||
"job.delete()\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -39,9 +39,8 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_pytorch.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -83,9 +82,8 @@
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
"* `Vertex AI Training`\n",
|
||||
"* `Vertex AI Model` resource\n",
|
||||
"\n",
|
||||
"- `Vertex AI Training`\n",
|
||||
"- `Vertex AI Model` resource\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
@@ -95,75 +93,6 @@
|
||||
"- Create a `Vertex AI Model` resource."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "85ee859437ed"
|
||||
},
|
||||
"source": [
|
||||
"## Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "5cd61a5dd9db"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
|
||||
"all the requirements to run this notebook. You can skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "7e689ee0bc3c"
|
||||
},
|
||||
"source": [
|
||||
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
|
||||
"You need the following:\n",
|
||||
"\n",
|
||||
"* The Google Cloud SDK\n",
|
||||
"* Git\n",
|
||||
"* Python 3\n",
|
||||
"* virtualenv\n",
|
||||
"* Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"The Google Cloud guide to [Setting up a Python development\n",
|
||||
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
|
||||
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
|
||||
"for meeting these requirements. The following steps provide a condensed set of\n",
|
||||
"instructions:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
|
||||
"\n",
|
||||
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
|
||||
"\n",
|
||||
"1. [Install\n",
|
||||
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
|
||||
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
|
||||
"\n",
|
||||
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
|
||||
"command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. Open this notebook in the Jupyter Notebook Dashboard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -183,10 +112,22 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
"! pip3 install --upgrade torchvision $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -344,7 +285,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -355,8 +296,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -376,7 +317,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -396,7 +337,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -419,7 +360,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -441,7 +382,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -471,15 +412,13 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (\n",
|
||||
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -622,7 +561,6 @@
|
||||
"# Add package information\n",
|
||||
"! touch custom/README.md\n",
|
||||
"\n",
|
||||
"# Instructions for installing package into environment of the docker image\n",
|
||||
"setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n",
|
||||
"! echo \"$setup_cfg\" > custom/setup.cfg\n",
|
||||
"\n",
|
||||
@@ -953,7 +891,7 @@
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_cifar10.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -964,7 +902,7 @@
|
||||
"source": [
|
||||
"### Make Pytorch container for prediction\n",
|
||||
"\n",
|
||||
"Currently, Vertex AI does not have a predefined container for making predictions with a deployed Pytorch model. No problem, you can assemble your own custom container. Typically, one would base the container on the `Torch Server`. For demonstration purpose, you build a placeholder container (not complete) that includes the latest `Torch Server` image, and push it to the `Container Registry`."
|
||||
"Currently, Vertex AI does not have a prefined container for making predictions with a deployed Pytorch model. No problem, you can assemble your own custom container. Typically, one would base the container on the `Torch Server`. For demonstration purpose, you build a placeholder container (not complete) that includes the latest `Torch Server` image, and push it to the `Container Registry`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1036,9 +974,9 @@
|
||||
"source": [
|
||||
"DISPLAY_NAME = \"cifar10_\" + TIMESTAMP\n",
|
||||
"\n",
|
||||
"job = aiplatform.CustomPythonPackageTrainingJob(\n",
|
||||
"job = aip.CustomPythonPackageTrainingJob(\n",
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_URI}/trainer_cifar10.tar.gz\",\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_NAME}/trainer_cifar10.tar.gz\",\n",
|
||||
" python_module_name=\"trainer.task\",\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
@@ -1071,7 +1009,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
|
||||
"\n",
|
||||
"DIRECT = False\n",
|
||||
"if DIRECT:\n",
|
||||
@@ -1183,7 +1121,7 @@
|
||||
"source": [
|
||||
"### Delete a custom training job\n",
|
||||
"\n",
|
||||
"After a training job is completed, you can delete the training job with the method `delete()`. Prior to completion, a training job can be cancelled with the method `cancel()`."
|
||||
"After a training job is completed, you can delete the training job with the method `delete()`. Prior to completion, a training job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1210,8 +1148,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Custom Job (Custom job deleted in previous cell)\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1223,12 +1167,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete the model using the Vertex model object\n",
|
||||
"model.delete()\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
"id": "title:generic,gcp"
|
||||
},
|
||||
"source": [
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex AI Training for R\n",
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Training for R\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
@@ -39,9 +39,8 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_r.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_r.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -57,7 +56,7 @@
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex AI Training for R. Please note that this notebook should be ran only in R notebook image (e.g., R4.1)."
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex Training for R."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -99,28 +98,6 @@
|
||||
"- Train a R model using `Vertex AI Trainingh` service with the R-to-Python training package."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0c997d8d92ce"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -132,25 +109,6 @@
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1fd00fa70a2a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -159,8 +117,22 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -192,39 +164,6 @@
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0e3cab0cc491"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "be929e7b4d76"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -244,8 +183,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -303,9 +240,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -342,7 +277,7 @@
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
|
||||
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
|
||||
"\n",
|
||||
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
|
||||
]
|
||||
@@ -355,7 +290,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -366,8 +301,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -387,7 +322,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -407,7 +342,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -430,8 +365,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import traceback\n",
|
||||
"\n",
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
@@ -454,7 +387,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1031,17 +964,14 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"try:\n",
|
||||
" INSTANCES = [\n",
|
||||
" {\"sepal_width\": 1, \"sepal_length\": 2, \"petal_width\": 3, \"petal_length\": 1},\n",
|
||||
" {\"sepal_width\": 4, \"sepal_length\": 2, \"petal_width\": 1, \"petal_length\": 1},\n",
|
||||
" ]\n",
|
||||
"INSTANCES = [\n",
|
||||
" {\"sepal_width\": 1, \"sepal_length\": 2, \"petal_width\": 3, \"petal_length\": 1},\n",
|
||||
" {\"sepal_width\": 4, \"sepal_length\": 2, \"petal_width\": 1, \"petal_length\": 1},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
" prediction = endpoint.predict(instances=INSTANCES)\n",
|
||||
"prediction = endpoint.predict(instances=INSTANCES)\n",
|
||||
"\n",
|
||||
" print(prediction)\n",
|
||||
"except:\n",
|
||||
" traceback.print_exc()"
|
||||
"print(prediction)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1312,7 +1242,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"CMDARGS = [\"--model-dir=\" + BUCKET_URI]\n",
|
||||
"CMDARGS = [\"--model-dir=\" + BUCKET_NAME]\n",
|
||||
"\n",
|
||||
"job.run(args=CMDARGS, replica_count=1, machine_type=TRAIN_COMPUTE, sync=True)"
|
||||
]
|
||||
@@ -1352,9 +1282,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Model (Already deleted in previous cells)\n",
|
||||
"- Endpoint (Already deleted in previous cells)\n",
|
||||
"- Custom Job (Already deleted in previous cells)\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1366,8 +1301,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -40,11 +40,7 @@
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
"Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -105,7 +101,62 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"You will not need special packages for this notebook."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -234,7 +285,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -245,8 +296,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -266,7 +317,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -286,7 +337,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -331,7 +382,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -364,8 +415,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
@@ -734,7 +783,7 @@
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_newsaggr.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_newsaggr.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -774,7 +823,7 @@
|
||||
"\n",
|
||||
"job = aip.CustomPythonPackageTrainingJob(\n",
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_URI}/trainer_newsaggr.tar.gz\",\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_NAME}/trainer_newsaggr.tar.gz\",\n",
|
||||
" python_module_name=\"trainer.task\",\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
@@ -808,7 +857,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
|
||||
"DATASET_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/00359/NewsAggregatorDataset.zip\"\n",
|
||||
"\n",
|
||||
"DIRECT = False\n",
|
||||
@@ -953,9 +1002,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Custom Job (already deleted in previous cell)\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -963,15 +1017,65 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b413063dfdcf"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete the model using the Vertex model object\n",
|
||||
"model.delete()\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -29,15 +29,9 @@
|
||||
"id": "title:generic,gcp"
|
||||
},
|
||||
"source": [
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex AI Training for XGBoost\n",
|
||||
"# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Training for XGBoost\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_xgboost.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_xgboost.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
@@ -45,9 +39,8 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_xgboost.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_xgboost.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -63,7 +56,7 @@
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex AI Training for XGBoost."
|
||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex Training for XGBoost."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -97,21 +90,7 @@
|
||||
"- Training using a Python package.\n",
|
||||
"- Report accuracy when hyperparameter tuning.\n",
|
||||
"- Save the model artifacts to Cloud Storage using GCSFuse.\n",
|
||||
"- Create a `Vertex AI Model` resource.\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
"- Create a `Vertex AI Model` resource."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -129,24 +108,55 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ncRJ_Dfdox9L"
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
"# ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
"# ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
"# ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -221,14 +231,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sKBTnvJpox9P"
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -246,7 +253,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JYtXOocrox9Q"
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -255,39 +262,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "NNc5Bf_NpPTq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -298,7 +272,7 @@
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
|
||||
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
|
||||
"\n",
|
||||
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
|
||||
]
|
||||
@@ -311,7 +285,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -322,8 +296,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -339,11 +313,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "aO4sKJfFox9R"
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -359,11 +333,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "yWnghzKFox9S"
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -404,11 +378,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JZg2sszQox9T"
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -437,12 +411,10 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "cQUrG4Mbox9T"
|
||||
"id": "accelerators:training,cpu,prediction,cpu,mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
@@ -481,7 +453,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "XujRA5ueox9U"
|
||||
"id": "container:training,prediction,xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -525,7 +497,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "UMPFgENkox9U"
|
||||
"id": "machine:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -589,7 +561,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "f4wS4eISox9V"
|
||||
"id": "examine_training_package:xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -644,7 +616,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "WiSnFuDoox9W"
|
||||
"id": "taskpy_contents:iris,xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -759,14 +731,14 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "dnmdycf6ox9X"
|
||||
"id": "tarball_training_script"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_iris.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_iris.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -798,7 +770,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "rVEMz1xqox9X"
|
||||
"id": "create_custom_pp_training_job:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -806,7 +778,7 @@
|
||||
"\n",
|
||||
"job = aip.CustomPythonPackageTrainingJob(\n",
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_URI}/trainer_iris.tar.gz\",\n",
|
||||
" python_package_gcs_uri=f\"{BUCKET_NAME}/trainer_iris.tar.gz\",\n",
|
||||
" python_module_name=\"trainer.task\",\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
@@ -837,11 +809,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AoUfpBqVox9Y"
|
||||
"id": "prepare_custom_cmdargs:iris,xgboost"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
|
||||
"MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
|
||||
"DATASET_DIR = \"gs://cloud-samples-data/ai-platform/iris\"\n",
|
||||
"\n",
|
||||
"ROUNDS = 20\n",
|
||||
@@ -886,7 +858,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JCruQq1aox9Y"
|
||||
"id": "run_custom_job:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -927,7 +899,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "KBM_KLMSox9Y"
|
||||
"id": "list_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -950,7 +922,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "lHPMHbSyox9Z"
|
||||
"id": "custom_job_wait:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -972,7 +944,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "tlYg7Sp-ox9Z"
|
||||
"id": "delete_job"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -992,7 +964,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Custom Job (Custome Training job is remove in previous step)\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1000,12 +979,65 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JyWy23gDox9a"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -66,17 +66,6 @@ The steps performed include:
|
||||
- Execute a Vertex AI pipeline.
|
||||
```
|
||||
|
||||
[Get Started with Dataproc components](get_started_with_dataproc_pipeline_components.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- DataprocPySparkBatchOp for PySpark batch workloads.
|
||||
- DataprocSparkBatchOp for Spark batch workloads.
|
||||
- DataprocSparkSqlBatchOp for running Spark SQL batch workloads.
|
||||
- DataprocSparkRBatchOp for running SparkR batch workloads.
|
||||
```
|
||||
|
||||
[Get Started with Vertex AI AutoML components](get_started_with_automl_pipeline_components.ipynb)
|
||||
|
||||
```
|
||||
|
||||
+171
-305
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -38,15 +38,9 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -73,7 +67,7 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in the given image from the five classes of flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -100,15 +94,7 @@
|
||||
" - Training a Vertex AI AutoML trained model.\n",
|
||||
" - Test the serving binary with a batch prediction job.\n",
|
||||
" - Deploying a Vertex AI AutoML trained model.\n",
|
||||
"- Execute a Vertex AI pipeline.\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
"- Execute a Vertex AI pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -119,7 +105,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages for executing this MLOps notebook."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -130,22 +116,24 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
" \n",
|
||||
"! pip3 install tensorflow-io==0.18 $USER_FLAG -q\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" google-cloud-pipeline-components \\\n",
|
||||
" google-cloud-logging \\\n",
|
||||
" pyarrow \\\n",
|
||||
" kfp $USER_FLAG -q"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
" ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -156,7 +144,7 @@
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so that it can find the packages."
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -253,10 +241,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -283,63 +268,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c38be665ca50"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e0953a00668e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -363,8 +291,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -375,8 +302,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -396,7 +323,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -416,7 +343,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -427,7 +354,7 @@
|
||||
"source": [
|
||||
"#### Service Account\n",
|
||||
"\n",
|
||||
"You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
|
||||
"**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -451,15 +378,13 @@
|
||||
"source": [
|
||||
"if (\n",
|
||||
" SERVICE_ACCOUNT == \"\"\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
" or SERVICE_ACCOUNT is None\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
"):\n",
|
||||
" shell_output = ! gcloud projects describe $PROJECT_ID | sed -nre 's:.*projectNumber\\: (.*):\\1:p'\n",
|
||||
" SERVICE_ACCOUNT = (\n",
|
||||
" shell_output[0].replace(\"'\", \"\") + \"-compute@developer.gserviceaccount.com\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -470,7 +395,7 @@
|
||||
"source": [
|
||||
"#### Set service account access for Vertex AI Pipelines\n",
|
||||
"\n",
|
||||
"Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account."
|
||||
"Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -481,9 +406,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -492,7 +417,32 @@
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries"
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_tf"
|
||||
},
|
||||
"source": [
|
||||
"#### Import TensorFlow\n",
|
||||
"\n",
|
||||
"Import the TensorFlow package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -503,14 +453,22 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import base64\n",
|
||||
"import tensorflow as tf"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_kfp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import google.cloud.aiplatform as aiplatform\n",
|
||||
"import tensorflow as tf\n",
|
||||
"from kfp import dsl\n",
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2.dsl import Artifact, Input, Output, component"
|
||||
"from kfp.v2.dsl import component"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -532,7 +490,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -572,7 +530,7 @@
|
||||
"- Takes as input the region and Model artifacts returned from an AutoML training component.\n",
|
||||
"- Create a client interface to the Vertex AI Model service (`metadata[\"resource_name\"]).\n",
|
||||
"- Construct the resource ID for the model from the model artifact parameter.\n",
|
||||
"- Retrieve the model evaluation.\n",
|
||||
"- Retrieve the model evaluation\n",
|
||||
"- Return the model evaluation as a string."
|
||||
]
|
||||
},
|
||||
@@ -584,6 +542,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from kfp.v2.dsl import Artifact, Input, Model, Output\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
|
||||
"def evaluateAutoMLModelOp(\n",
|
||||
" model: Input[Artifact], region: str, model_evaluation: Output[Artifact]\n",
|
||||
@@ -616,7 +577,7 @@
|
||||
"1. Use the prebuilt component `ImageDatasetCreateOp` to create a Vertex AI Dataset resource, where:\n",
|
||||
" - The display name for the dataset is passed into the pipeline.\n",
|
||||
" - The import file for the dataset is passed into the pipeline.\n",
|
||||
" - The component returns the dataset resource as `outputs[\"dataset\"]`.\n",
|
||||
" - The component returns the dataset resource as `outputs[\"dataset\"]`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"2. Use the prebuilt component `AutoMLImageTrainingJobRunOp` to train a Vertex AI AutoML Model resource, where:\n",
|
||||
@@ -635,12 +596,12 @@
|
||||
" - The component returns the endpoint resource as `outputs[\"endpoint\"]`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"5. Use the prebuilt component `ModelDeployOp` to deploy the trained AutoML model where:\n",
|
||||
"5. Use the prebuilt component `ModelDeployOp` to deploy the trained AutoML model to, where:\n",
|
||||
" - The display name for the dataset is passed into the pipeline.\n",
|
||||
" - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
|
||||
" - The endpoint is the output from the `EndpointCreateOp`.\n",
|
||||
" - The endpoint is the output from the `EndpointCreateOp`\n",
|
||||
"\n",
|
||||
"*Note:* Since each component is executed as a graph node in its own execution context, you pass the parameter `project` for each component op, in constrast to doing a `aiplatform.init(project=project)` if this was a Python script calling the SDK methods directly within the same execution context."
|
||||
"*Note:* Since each component is executed as a graph node in its own execution context, you pass the parameter `project` for each component op, in constrast to doing a `aip.init(project=project)` if this was a Python script calling the SDK methods directly within the same execution context."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -651,7 +612,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_URI)\n",
|
||||
"from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_NAME)\n",
|
||||
"DEPLOY_COMPUTE = \"n1-standard-4\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -666,13 +629,12 @@
|
||||
" project: str = PROJECT_ID,\n",
|
||||
" region: str = REGION,\n",
|
||||
"):\n",
|
||||
" from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
|
||||
"\n",
|
||||
" dataset_op = gcc_aip.ImageDatasetCreateOp(\n",
|
||||
" project=project,\n",
|
||||
" display_name=display_name,\n",
|
||||
" gcs_source=import_file,\n",
|
||||
" import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification,\n",
|
||||
" import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" training_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
|
||||
@@ -710,12 +672,11 @@
|
||||
" display_name=display_name,\n",
|
||||
" ).after(batch_op)\n",
|
||||
"\n",
|
||||
" _ = gcc_aip.ModelDeployOp(\n",
|
||||
" deploy_op = gcc_aip.ModelDeployOp(\n",
|
||||
" model=training_op.outputs[\"model\"],\n",
|
||||
" endpoint=endpoint_op.outputs[\"endpoint\"],\n",
|
||||
" automatic_resources_min_replica_count=1,\n",
|
||||
" automatic_resources_max_replica_count=1,\n",
|
||||
" traffic_split={\"0\": 100},\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
@@ -727,7 +688,7 @@
|
||||
"source": [
|
||||
"### Get test item(s)\n",
|
||||
"\n",
|
||||
"In the pipeline, you do a batch prediction on your Vertex model. You will use arbitrary examples from the dataset as test items. Don't be concerned that the examples were likely used while training the model. This step is just to demonstrate how to make a prediction."
|
||||
"Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -772,11 +733,11 @@
|
||||
"file_1 = test_item_1.split(\"/\")[-1]\n",
|
||||
"file_2 = test_item_2.split(\"/\")[-1]\n",
|
||||
"\n",
|
||||
"! gsutil cp $test_item_1 $BUCKET_URI/$file_1\n",
|
||||
"! gsutil cp $test_item_2 $BUCKET_URI/$file_2\n",
|
||||
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
|
||||
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
|
||||
"\n",
|
||||
"test_item_1 = BUCKET_URI + \"/\" + file_1\n",
|
||||
"test_item_2 = BUCKET_URI + \"/\" + file_2"
|
||||
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
|
||||
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -787,14 +748,14 @@
|
||||
"source": [
|
||||
"### Make the batch input file\n",
|
||||
"\n",
|
||||
"Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can only be in JSONL format. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains key/value pairs:\n",
|
||||
"Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can only be in JSONL. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
|
||||
"\n",
|
||||
"- `content`: The Cloud Storage path to the image.\n",
|
||||
"- `mime_type`: The content type. In our example, it is a `jpeg` file.\n",
|
||||
"\n",
|
||||
"For example:\n",
|
||||
"\n",
|
||||
" {'content': '[your-bucket]/file1.jpg', 'mime_type': 'jpeg'}"
|
||||
" {'content': '[your-bucket]/file1.jpg', 'mime_type': 'jpeg'}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -805,7 +766,11 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gcs_input_uri = BUCKET_URI + \"/test.jsonl\"\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import tensorflow as tf\n",
|
||||
"\n",
|
||||
"gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
|
||||
"with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
|
||||
" data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
|
||||
" f.write(json.dumps(data) + \"\\n\")\n",
|
||||
@@ -845,7 +810,7 @@
|
||||
" pipeline_func=pipeline, package_path=\"automl_icn_training.json\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"automl_icn_training\",\n",
|
||||
" template_path=\"automl_icn_training.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -946,21 +911,9 @@
|
||||
"print(\"automl-image-training-job\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"automl-image-training-job\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
|
||||
"print(\"\\n\")\n",
|
||||
"print(model_id)\n",
|
||||
"print(\"endpoint-create\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"resourceName\"\n",
|
||||
"]\n",
|
||||
"print(\"\\n\")\n",
|
||||
"print(endpoint_id)\n",
|
||||
"print(\"model-deploy\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
@@ -976,12 +929,7 @@
|
||||
" output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"gcsOutputDirectory\"\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"batch_job_id = output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"resourceName\"\n",
|
||||
"]"
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1009,117 +957,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "endpoint_load:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"#### Load an endpoint\n",
|
||||
"\n",
|
||||
"The 'Endpoint' initializer will load an endpoint from an endpoint identifier."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "endpoint_load:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = aiplatform.Endpoint(endpoint_id)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "make_prediction"
|
||||
},
|
||||
"source": [
|
||||
"## Send a online prediction request\n",
|
||||
"\n",
|
||||
"Send a online prediction request to your deployed model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "get_test_item"
|
||||
},
|
||||
"source": [
|
||||
"### Get test item\n",
|
||||
"\n",
|
||||
"You will use an arbitrary example out of the dataset as a test item. Don't be concerned that the example was likely used while training the model. This step is just to demonstrate how to make a prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "get_test_item:automl,icn,csv"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
|
||||
"if len(str(test_item[0]).split(\",\")) == 3:\n",
|
||||
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
|
||||
"else:\n",
|
||||
" test_item, test_label = str(test_item[0]).split(\",\")\n",
|
||||
"\n",
|
||||
"print(test_item, test_label)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "predict_request:mbsdk,icn"
|
||||
},
|
||||
"source": [
|
||||
"### Make the prediction\n",
|
||||
"\n",
|
||||
"Now that your `Model` resource is deployed to an `Endpoint` resource, you can do online predictions by sending prediction requests to the Endpoint resource.\n",
|
||||
"\n",
|
||||
"#### Request\n",
|
||||
"\n",
|
||||
"Since in this example your test item is in a Cloud Storage bucket, you open and read the contents of the image using `tf.io.gfile.Gfile()`. To pass the test data to the prediction service, you encode the bytes into base64 which makes the content safe from modification while transmitting binary data over the network.\n",
|
||||
"\n",
|
||||
"The format of each instance is:\n",
|
||||
"\n",
|
||||
" { 'content': { 'b64': base64_encoded_bytes } }\n",
|
||||
"\n",
|
||||
"Since the `predict()` method can take multiple items (instances), send your single test item as a list of one test item.\n",
|
||||
"\n",
|
||||
"#### Response\n",
|
||||
"\n",
|
||||
"The response from the `predict()` call is a Python dictionary with the following entries:\n",
|
||||
"\n",
|
||||
"- `ids`: The internal assigned unique identifiers for each prediction request.\n",
|
||||
"- `displayNames`: The class names for each class label.\n",
|
||||
"- `confidences`: The predicted confidence, between 0 and 1, per class label.\n",
|
||||
"- `deployed_model_id`: The Vertex AI identifier for the deployed Model resource which did the predictions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "predict_request:mbsdk,icn"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
|
||||
" content = f.read()\n",
|
||||
"\n",
|
||||
"# The format of each instance should conform to the deployed model's prediction input schema.\n",
|
||||
"instances = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
|
||||
"\n",
|
||||
"prediction = endpoint.predict(instances=instances)\n",
|
||||
"\n",
|
||||
"print(prediction)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "9d347472d5ba"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"# Cleaning up\n",
|
||||
@@ -1127,40 +965,17 @@
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial.\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"#### Delete the Vertex AI Model, Endpoint and BatchPredictionJob resources\n",
|
||||
"\n",
|
||||
"Undelpoy and delete the Vertex AI Model, Endpoint and BatchPredictionJob resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "baa3e1071f7b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.undeploy_all()\n",
|
||||
"endpoint.delete()\n",
|
||||
"\n",
|
||||
"model = aiplatform.Model(model_id)\n",
|
||||
"model.delete()\n",
|
||||
"\n",
|
||||
"batch_job = aiplatform.BatchPredictionJob(batch_job_id)\n",
|
||||
"batch_job.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a802da1f6fa7"
|
||||
},
|
||||
"source": [
|
||||
"#### Delete the Cloud Storage bucket\n",
|
||||
"\n",
|
||||
"Set `delete_bucket` to *True* to delete the Cloud storage bucket used in this notebook."
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1171,10 +986,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = False\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+94
-160
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -33,20 +33,14 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bq_tfdv_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bq_tfdv_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bq_tfdv_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bq_tfdv_pipeline_components.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -99,28 +93,6 @@
|
||||
"- Execute a Vertex AI pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0c997d8d92ce"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -132,25 +104,6 @@
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1fd00fa70a2a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -159,9 +112,20 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U tensorflow $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade kfp $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -212,8 +176,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -271,9 +233,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -300,81 +260,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "648aa9824ac6"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "fc52bba17ee3"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "535223fa4b84"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -398,7 +283,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -409,8 +294,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -430,7 +315,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -450,7 +335,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -461,9 +346,7 @@
|
||||
"source": [
|
||||
"#### Service Account\n",
|
||||
"\n",
|
||||
"**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below.\n",
|
||||
"\n",
|
||||
"*Note:* The code for automatically finding your service account works on a user-managed Workbench AI noteboook. If you are using a fully-managed notebook or colab, you will need to manually enter your service account."
|
||||
"**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -492,7 +375,7 @@
|
||||
"):\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
@@ -515,9 +398,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -599,7 +482,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -714,7 +597,7 @@
|
||||
" return dataset.column_names\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/dataset_bq\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/dataset_bq\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(\n",
|
||||
@@ -727,9 +610,9 @@
|
||||
"):\n",
|
||||
" create_op = create_dataset_bq(bq_table, display_name, project)\n",
|
||||
"\n",
|
||||
" _ = get_dataset_source(create_op.output)\n",
|
||||
" source_op = get_dataset_source(create_op.output)\n",
|
||||
"\n",
|
||||
" _ = get_column_names(create_op.output)\n",
|
||||
" column_names_op = get_column_names(create_op.output)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"dataset_bq.json\")\n",
|
||||
@@ -928,7 +811,7 @@
|
||||
" return (stats_file, schema_file)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/dataset_stats\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/dataset_stats\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(\n",
|
||||
@@ -936,7 +819,7 @@
|
||||
")\n",
|
||||
"def pipeline(dataset_id: str, label: str, bucket: str):\n",
|
||||
"\n",
|
||||
" _ = statistics(dataset_id, label, bucket)\n",
|
||||
" stats_op = statistics(dataset_id, label, bucket)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"dataset_stats.json\")\n",
|
||||
@@ -948,7 +831,7 @@
|
||||
" parameter_values={\n",
|
||||
" \"dataset_id\": dataset_id,\n",
|
||||
" \"label\": \"mean_temp\",\n",
|
||||
" \"bucket\": BUCKET_URI,\n",
|
||||
" \"bucket\": BUCKET_NAME,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
@@ -1018,7 +901,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Vertex AI dataset\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1030,17 +920,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"# Create reference to Vertex AI dataset created in pipeline\n",
|
||||
"dataset = aip.TabularDataset(dataset_id)\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"# delete Vertex AI dataset\n",
|
||||
"dataset.delete()\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+62
-213
@@ -33,20 +33,14 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -105,28 +99,6 @@
|
||||
"- Make a prediction with the deployed Vertex AI model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0c997d8d92ce"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -142,34 +114,28 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1fd00fa70a2a"
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-OxtcyNNJ39g"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U tensorflow $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
"! pip3 install --upgrade kfp $USER_FLAG\n"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
" ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -187,7 +153,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "fIuF_ZjxJ39h"
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -220,8 +186,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -275,13 +239,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c1Rim3ogJ39j"
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -299,7 +261,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "hdkr5x2jJ39k"
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -308,81 +270,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "UG2SHSlTJ39k"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ETQR4H1HJ39k"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "9M66jv07J39l"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -407,7 +294,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -435,7 +322,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "V97jQQuiJ39m"
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -455,7 +342,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7PN6kSQtJ39m"
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -479,7 +366,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "M4WZi4CDJ39n"
|
||||
"id": "set_service_account"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -520,7 +407,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "mI3IJONMJ39n"
|
||||
"id": "set_service_account:pipelines"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -563,7 +450,8 @@
|
||||
"import json\n",
|
||||
"\n",
|
||||
"from kfp import dsl\n",
|
||||
"from kfp.v2 import compiler"
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2.dsl import component"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -581,7 +469,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r7p4Iv8_J39o"
|
||||
"id": "import_bq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -603,7 +491,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "_K5tP8oJJ39p"
|
||||
"id": "import_tf"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -625,11 +513,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "uAnLpS9cJ39p"
|
||||
"id": "init_aip:mbsdk,all"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -647,7 +535,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "I9RloZo9J39p"
|
||||
"id": "init_bq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -677,7 +565,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1-mE_7kXJ39p"
|
||||
"id": "accelerators:prediction,mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -710,7 +598,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AmHM8whxJ39q"
|
||||
"id": "container:prediction"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -783,7 +671,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LvND7iTpJ39r"
|
||||
"id": "create_bqml_pipeline:tabular"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -829,11 +717,11 @@
|
||||
" query=f\"CREATE OR REPLACE MODEL {dataset}.{model} OPTIONS (model_type='dnn_classifier', labels=['{label}'], num_trials={num_trials}) AS SELECT * FROM `{bq_table}` WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL\",\n",
|
||||
" ).after(bq_dataset)\n",
|
||||
"\n",
|
||||
" _ = BigqueryEvaluateModelJobOp(\n",
|
||||
" bq_eval = BigqueryEvaluateModelJobOp(\n",
|
||||
" project=PROJECT_ID, location=\"US\", model=bq_model.outputs[\"model\"]\n",
|
||||
" ).after(bq_model)\n",
|
||||
"\n",
|
||||
" _ = BigqueryPredictModelJobOp(\n",
|
||||
" bq_predict = BigqueryPredictModelJobOp(\n",
|
||||
" project=project,\n",
|
||||
" location=location,\n",
|
||||
" model=bq_model.outputs[\"model\"],\n",
|
||||
@@ -877,7 +765,7 @@
|
||||
" display_name=display_name,\n",
|
||||
" ).after(model_upload)\n",
|
||||
"\n",
|
||||
" _ = ModelDeployOp(\n",
|
||||
" deploy_model = ModelDeployOp(\n",
|
||||
" model=model_upload.outputs[\"model\"],\n",
|
||||
" endpoint=endpoint.outputs[\"endpoint\"],\n",
|
||||
" dedicated_resources_min_replica_count=min_replica_count,\n",
|
||||
@@ -919,23 +807,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "l2FMs74-J39r"
|
||||
"id": "run_pipeline:bqml"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If DEPLOY_GPU is None, keeping gpu as no accelerator and accelerator_count as 0\n",
|
||||
"accelerator_count = 0\n",
|
||||
"if DEPLOY_GPU:\n",
|
||||
" gpu = DEPLOY_GPU.name\n",
|
||||
" accelerator_count = 1\n",
|
||||
"else:\n",
|
||||
" gpu = \"ACCELERATOR_TYPE_UNSPECIFIED\" # Unspecified accelerator type, which means no accelerator.\n",
|
||||
" accelerator_count = 0\n",
|
||||
"\n",
|
||||
"print(\"gpu=\", gpu)\n",
|
||||
"print(\"accelerator_count=\", accelerator_count)\n",
|
||||
"\n",
|
||||
"MODEL_DIR = BUCKET_URI + \"/bqmodel\"\n",
|
||||
"MODEL_DIR = BUCKET_NAME + \"/bqmodel\"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"bqml.json\")\n",
|
||||
"\n",
|
||||
@@ -955,8 +831,8 @@
|
||||
" \"machine_type\": \"n1-standard-4\",\n",
|
||||
" \"min_replica_count\": 1,\n",
|
||||
" \"max_replica_count\": 1,\n",
|
||||
" \"accelerator_type\": gpu,\n",
|
||||
" \"accelerator_count\": accelerator_count,\n",
|
||||
" \"accelerator_type\": DEPLOY_GPU.name,\n",
|
||||
" \"accelerator_count\": DEPLOY_NGPU,\n",
|
||||
" \"project\": PROJECT_ID,\n",
|
||||
" \"location\": \"US\",\n",
|
||||
" },\n",
|
||||
@@ -981,7 +857,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2OM8zzJXJ39s"
|
||||
"id": "view_pipleline_results:bqml"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1060,9 +936,6 @@
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"model-upload\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"model-upload\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"endpoint-create\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
|
||||
@@ -1095,7 +968,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1UTEiNi9J39s"
|
||||
"id": "delete_pipeline"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1117,7 +990,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gPEt5GMAJ39s"
|
||||
"id": "endpoint_load:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1164,7 +1037,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sesK_MSdJ39t"
|
||||
"id": "make_test_items:bqml,penguins"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1210,7 +1083,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "u5_cgdKQJ39t"
|
||||
"id": "endpoint_predict:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1238,41 +1111,15 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "W0rhdoHmJ39t"
|
||||
"id": "delete:bqml,penguins"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"try:\n",
|
||||
" job = bqclient.delete_model(f\"{PROJECT_ID}.bqml_tutorial.penguins_model\")\n",
|
||||
" job = bqclient.delete_model(\"bqml_tutorial.penguins_model\")\n",
|
||||
"except:\n",
|
||||
" pass\n",
|
||||
"job = bqclient.delete_dataset(f\"{PROJECT_ID}.bqml_tutorial\", delete_contents=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e776f9a3bdc4"
|
||||
},
|
||||
"source": [
|
||||
"#### Delete the Vertex AI Model and Endpoint resources\n",
|
||||
"\n",
|
||||
"Next, undelpoy and delete the Vertex AI Model and Endpoint resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "63462e0480f0"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.undeploy_all()\n",
|
||||
"endpoint.delete()\n",
|
||||
"\n",
|
||||
"model = aip.Model(model_id)\n",
|
||||
"model.delete()"
|
||||
"job = bqclient.delete_dataset(\"bqml_tutorial\", delete_contents=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1293,15 +1140,17 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ufWUEbnZJ39u"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_all:\n",
|
||||
" # (DEVELOPER TODO) Find generated resources from pipeline and delete\n",
|
||||
"\n",
|
||||
" if \"BUCKET_URI\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+160
-312
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -33,20 +33,14 @@
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_custom_training_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_custom_training_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_custom_training_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_custom_training_pipeline_components.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -100,37 +94,9 @@
|
||||
" - Training a Vertex AI custom trained model.\n",
|
||||
" - Test the serving binary with a batch prediction job.\n",
|
||||
" - Deploying a Vertex AI custom trained model.\n",
|
||||
"- Execute a Vertex AI pipeline.\n",
|
||||
"- Construct a pipeline for:\n",
|
||||
" - Construct a custom training component.\n",
|
||||
" - Convert custom training component to CustomTrainingJobOp.\n",
|
||||
" - Training a Vertex AI custom trained model using the converted component.\n",
|
||||
" - Deploying a Vertex AI custom trained model.\n",
|
||||
"- Execute a Vertex AI pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0c997d8d92ce"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -142,25 +108,6 @@
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1fd00fa70a2a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -169,10 +116,24 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U tensorflow $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"! pip3 install --upgrade kfp $USER_FLAG\n"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
" ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -223,8 +184,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -282,9 +241,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -311,81 +268,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "648aa9824ac6"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "fc52bba17ee3"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "535223fa4b84"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -409,8 +291,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -421,8 +302,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -442,7 +323,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -462,7 +343,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -525,9 +406,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -609,7 +490,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -648,7 +529,7 @@
|
||||
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
|
||||
" DEPLOY_GPU, DEPLOY_NGPU = (\n",
|
||||
@@ -1136,7 +1017,7 @@
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_flowers.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_flowers.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1196,20 +1077,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_training\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_training\".format(BUCKET_NAME)\n",
|
||||
"DEPLOY_COMPUTE = \"n1-standard-4\"\n",
|
||||
"\n",
|
||||
"# If TRAIN_GPU is None, keeping gpu as no accelerator and accelerator_count as 0\n",
|
||||
"gpu = \"ACCELERATOR_TYPE_UNSPECIFIED\"\n",
|
||||
"accelerator_count = 0\n",
|
||||
"\n",
|
||||
"if TRAIN_GPU:\n",
|
||||
" gpu = TRAIN_GPU.name\n",
|
||||
" accelerator_count = 1\n",
|
||||
"else:\n",
|
||||
" gpu = \"ACCELERATOR_TYPE_UNSPECIFIED\" # Unspecified accelerator type, which means no accelerator.\n",
|
||||
" accelerator_count = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(\n",
|
||||
" name=\"flowers-custom-training\",\n",
|
||||
@@ -1251,8 +1121,8 @@
|
||||
" args=[\"--epochs\", \"50\", \"--image-width\", \"32\", \"--image-height\", \"32\"],\n",
|
||||
" replica_count=1,\n",
|
||||
" machine_type=TRAIN_COMPUTE,\n",
|
||||
" accelerator_type=gpu,\n",
|
||||
" accelerator_count=accelerator_count,\n",
|
||||
" accelerator_type=TRAIN_GPU.name,\n",
|
||||
" accelerator_count=TRAIN_NGPU,\n",
|
||||
" # Serving - As part of this operation, the model is registered to Vertex AI\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
" model_display_name=display_name,\n",
|
||||
@@ -1278,7 +1148,7 @@
|
||||
" display_name=display_name,\n",
|
||||
" ).after(batch_op)\n",
|
||||
"\n",
|
||||
" _ = ModelDeployOp(\n",
|
||||
" deploy_op = ModelDeployOp(\n",
|
||||
" model=training_op.outputs[\"model\"],\n",
|
||||
" endpoint=endpoint_op.outputs[\"endpoint\"],\n",
|
||||
" dedicated_resources_min_replica_count=1,\n",
|
||||
@@ -1340,11 +1210,11 @@
|
||||
"file_1 = test_item_1.split(\"/\")[-1]\n",
|
||||
"file_2 = test_item_2.split(\"/\")[-1]\n",
|
||||
"\n",
|
||||
"! gsutil cp $test_item_1 $BUCKET_URI/$file_1\n",
|
||||
"! gsutil cp $test_item_2 $BUCKET_URI/$file_2\n",
|
||||
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
|
||||
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
|
||||
"\n",
|
||||
"test_item_1 = BUCKET_URI + \"/\" + file_1\n",
|
||||
"test_item_2 = BUCKET_URI + \"/\" + file_2"
|
||||
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
|
||||
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1373,7 +1243,11 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gcs_input_uri = BUCKET_URI + \"/test.jsonl\"\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import tensorflow as tf\n",
|
||||
"\n",
|
||||
"gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
|
||||
"with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
|
||||
" data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
|
||||
" f.write(json.dumps(data) + \"\\n\")\n",
|
||||
@@ -1423,7 +1297,7 @@
|
||||
" \"import_file\": IMPORT_FILE,\n",
|
||||
" \"batch_files\": [gcs_input_uri],\n",
|
||||
" \"display_name\": \"flowers\" + TIMESTAMP,\n",
|
||||
" \"python_package\": f\"{BUCKET_URI}/trainer_flowers.tar.gz\",\n",
|
||||
" \"python_package\": f\"{BUCKET_NAME}/trainer_flowers.tar.gz\",\n",
|
||||
" \"python_module\": \"trainer.task\",\n",
|
||||
" \"project\": PROJECT_ID,\n",
|
||||
" \"region\": REGION,\n",
|
||||
@@ -1487,53 +1361,25 @@
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/gcp_resources\"\n",
|
||||
" )\n",
|
||||
" EVAL_METRICS = (\n",
|
||||
" PIPELINE_ROOT\n",
|
||||
" + \"/\"\n",
|
||||
" + PROJECT_NUMBER\n",
|
||||
" + \"/\"\n",
|
||||
" + JOB_ID\n",
|
||||
" + \"/\"\n",
|
||||
" + output_task_name\n",
|
||||
" + \"_\"\n",
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/evaluation_metrics\"\n",
|
||||
" )\n",
|
||||
" if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
|
||||
" ! gsutil cat $EXECUTE_OUTPUT\n",
|
||||
" return EXECUTE_OUTPUT\n",
|
||||
" break\n",
|
||||
" elif tf.io.gfile.exists(GCP_RESOURCES):\n",
|
||||
" ! gsutil cat $GCP_RESOURCES\n",
|
||||
" return GCP_RESOURCES\n",
|
||||
" elif tf.io.gfile.exists(EVAL_METRICS):\n",
|
||||
" ! gsutil cat $EVAL_METRICS\n",
|
||||
" return EVAL_METRICS\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
" return None\n",
|
||||
" return EXECUTE_OUTPUT\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(\"image-dataset-create\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"image-dataset-create\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"dataset_id = output[\"artifacts\"][\"dataset\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"custompythonpackagetrainingjob-run\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"custompythonpackagetrainingjob-run\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"endpoint-create\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"resourceName\"\n",
|
||||
"]\n",
|
||||
"print(\"model-deploy\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
@@ -1546,13 +1392,7 @@
|
||||
" output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"gcsOutputDirectory\"\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"batch_job_id = output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"resourceName\"\n",
|
||||
"]\n",
|
||||
"print(\"\\n\\n\")"
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1577,49 +1417,6 @@
|
||||
"pipeline.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d614c239d74c"
|
||||
},
|
||||
"source": [
|
||||
"#### Delete the Vertex AI Model, Endpoint and BatchPredictionJob resources\n",
|
||||
"\n",
|
||||
"Next, delete the daatset, undelpoy and delete the Vertex AI Model, Endpoint and BathPredictionJob resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "417791a1a7e2"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aip.ImageDataset(dataset_id)\n",
|
||||
"try:\n",
|
||||
" dataset.delete()\n",
|
||||
"except:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"endpoint = aip.Endpoint(endpoint_id)\n",
|
||||
"endpoint.undeploy_all()\n",
|
||||
"try:\n",
|
||||
" endpoint.delete()\n",
|
||||
"except:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"model = aip.Model(model_id)\n",
|
||||
"try:\n",
|
||||
" model.delete()\n",
|
||||
"except:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"batch_job = aip.BatchPredictionJob(batch_job_id)\n",
|
||||
"batch_job.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1662,6 +1459,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google_cloud_pipeline_components.v1.custom_job import utils\n",
|
||||
"from kfp.v2.dsl import Artifact\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component(\n",
|
||||
@@ -1818,7 +1616,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_cifar10_training\".format(BUCKET_URI)\n",
|
||||
"from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_cifar10_training\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(name=\"custom-model-training-sample-pipeline\")\n",
|
||||
@@ -1832,8 +1632,6 @@
|
||||
" location: str = REGION,\n",
|
||||
" deploy_image: str = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
|
||||
"):\n",
|
||||
" from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
|
||||
"\n",
|
||||
" custom_job_op = custom_job_training_op(\n",
|
||||
" model_dir=model_dir,\n",
|
||||
" lr=lr,\n",
|
||||
@@ -1859,7 +1657,7 @@
|
||||
" display_name=display_name,\n",
|
||||
" ).after(model_upload_op)\n",
|
||||
"\n",
|
||||
" _ = gcc_aip.ModelDeployOp(\n",
|
||||
" deploy_op = gcc_aip.ModelDeployOp(\n",
|
||||
" model=model_upload_op.outputs[\"model\"],\n",
|
||||
" endpoint=endpoint_op.outputs[\"endpoint\"],\n",
|
||||
" dedicated_resources_min_replica_count=1,\n",
|
||||
@@ -1937,6 +1735,46 @@
|
||||
"PROJECT_NUMBER = pipeline.gca_resource.name.split(\"/\")[1]\n",
|
||||
"print(PROJECT_NUMBER)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def print_pipeline_output(job, output_task_name):\n",
|
||||
" JOB_ID = job.name\n",
|
||||
" print(JOB_ID)\n",
|
||||
" for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
|
||||
" TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
|
||||
" EXECUTE_OUTPUT = (\n",
|
||||
" PIPELINE_ROOT\n",
|
||||
" + \"/\"\n",
|
||||
" + PROJECT_NUMBER\n",
|
||||
" + \"/\"\n",
|
||||
" + JOB_ID\n",
|
||||
" + \"/\"\n",
|
||||
" + output_task_name\n",
|
||||
" + \"_\"\n",
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/executor_output.json\"\n",
|
||||
" )\n",
|
||||
" GCP_RESOURCES = (\n",
|
||||
" PIPELINE_ROOT\n",
|
||||
" + \"/\"\n",
|
||||
" + PROJECT_NUMBER\n",
|
||||
" + \"/\"\n",
|
||||
" + JOB_ID\n",
|
||||
" + \"/\"\n",
|
||||
" + output_task_name\n",
|
||||
" + \"_\"\n",
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/gcp_resources\"\n",
|
||||
" )\n",
|
||||
" if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
|
||||
" ! gsutil cat $EXECUTE_OUTPUT\n",
|
||||
" break\n",
|
||||
" elif tf.io.gfile.exists(GCP_RESOURCES):\n",
|
||||
" ! gsutil cat $GCP_RESOURCES\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
" return EXECUTE_OUTPUT\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(\"custom-train-model\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"custom-train-model\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
@@ -1946,17 +1784,9 @@
|
||||
"print(\"model-upload\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"model-upload\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
|
||||
"print(\"endpoint-create\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"output = !gsutil cat $artifacts\n",
|
||||
"output = json.loads(output[0])\n",
|
||||
"endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
|
||||
" \"resourceName\"\n",
|
||||
"]\n",
|
||||
"print(\"model-deploy\")\n",
|
||||
"artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
|
||||
"print(\"\\n\\n\")"
|
||||
@@ -1984,49 +1814,6 @@
|
||||
"pipeline.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "93e69fc8b9e3"
|
||||
},
|
||||
"source": [
|
||||
"#### Delete the Vertex AI Model and Endpoint resource\n",
|
||||
"\n",
|
||||
"Next, undelpoy and delete the Vertex AI Model and Endpoint resources."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c9d18ae084b1"
|
||||
},
|
||||
"source": [
|
||||
"#### Delete the Vertex model and endpoint\n",
|
||||
"\n",
|
||||
"Next, undelpoy and delete the Vertex Model and Endpoint resource."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "dccf71121d0b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.undeploy_all()\n",
|
||||
"try:\n",
|
||||
" endpoint.delete()\n",
|
||||
"except:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"model = aip.Model(model_id)\n",
|
||||
"try:\n",
|
||||
" model.delete()\n",
|
||||
"except:\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -2038,7 +1825,17 @@
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial."
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -2049,10 +1846,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = False\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
-1640
File diff suppressed because it is too large
Load Diff
+132
-140
@@ -32,11 +32,6 @@
|
||||
"# E2E ML on GCP: MLOps stage 3 : formalization: get started with Hyperparameter Tuning pipeline components\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
@@ -44,8 +39,7 @@
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
@@ -100,20 +94,7 @@
|
||||
" - If the metrics exceed a specified threshold.\n",
|
||||
" - Get the location of the model artifacts for the best tuned model.\n",
|
||||
" - Upload the model artifacts to a `Vertex AI Model` resource.\n",
|
||||
"- Execute a Vertex AI pipeline.\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
"- Execute a Vertex AI pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -131,26 +112,28 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LR9HQnyiMoT5"
|
||||
"id": "install_mlops"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
"! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
"! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
"! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
"! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
"# ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
"# ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade torchvision $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade rpy2 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
|
||||
" ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -168,7 +151,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "VeBfL2pmMoT7"
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -254,14 +237,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7iewOt9NMoT8"
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -279,7 +259,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Y-vhpfibMoT9"
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -288,39 +268,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-V_6SvMUNUa1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -344,7 +291,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -355,8 +302,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -372,11 +319,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2smRgc53MoT-"
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -392,11 +339,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ME1Tr9j_MoT-"
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -414,7 +361,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "EIivrR-3MoT-"
|
||||
"id": "set_service_account"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -431,15 +378,13 @@
|
||||
"source": [
|
||||
"if (\n",
|
||||
" SERVICE_ACCOUNT == \"\"\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
" or SERVICE_ACCOUNT is None\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
"):\n",
|
||||
" shell_output = ! gcloud projects describe $PROJECT_ID | sed -nre 's:.*projectNumber\\: (.*):\\1:p'\n",
|
||||
" SERVICE_ACCOUNT = (\n",
|
||||
" shell_output[0].replace(\"'\", \"\") + \"-compute@developer.gserviceaccount.com\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -457,13 +402,13 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "mtwsjYnIMoT_"
|
||||
"id": "set_service_account:pipelines"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -519,7 +464,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "DtTIHh_KMoUA"
|
||||
"id": "import_tf"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -541,11 +486,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sUFJPDW0MoUA"
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -574,19 +519,17 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "A6dzi4cXMoUA"
|
||||
"id": "accelerators:training,prediction,ngpu,mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
|
||||
" TRAIN_GPU, TRAIN_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
|
||||
" DEPLOY_GPU, DEPLOY_NGPU = (\n",
|
||||
@@ -617,7 +560,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gxai072KMoUB"
|
||||
"id": "container:prediction"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -673,7 +616,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LEYjL1ojMoUB"
|
||||
"id": "machine:training,prediction"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -729,7 +672,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "UIoZdxPqMoUC"
|
||||
"id": "examine_training_package"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -792,7 +735,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "uk4MAGErMoUC"
|
||||
"id": "taskpy_contents:dataset,horses_or_humans"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -929,14 +872,14 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "bHcLQLGkMoUD"
|
||||
"id": "tarball_training_script"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! rm -f custom.tar custom.tar.gz\n",
|
||||
"! tar cvf custom.tar custom\n",
|
||||
"! gzip custom.tar\n",
|
||||
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_horses_or_humans.tar.gz"
|
||||
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_horses_or_humans.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -969,7 +912,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "MaMK4AoBMoUE"
|
||||
"id": "write_docker_file:training,tf-dlvm"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1002,7 +945,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "9qVJGXT2MoUE"
|
||||
"id": "name_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1022,7 +965,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7-dkQP8hMoUE"
|
||||
"id": "build_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1044,7 +987,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7-kh6QBLMoUF"
|
||||
"id": "register_container:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1074,11 +1017,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Epzlh8M-MoUF"
|
||||
"id": "create_hpt_pipeline:icn"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_tuning\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_tuning\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
|
||||
@@ -1146,7 +1089,9 @@
|
||||
" threshold_op.output == \"true\",\n",
|
||||
" name=\"deploy_decision\",\n",
|
||||
" ):\n",
|
||||
" _ = hyperparameter_tuning_job.GetHyperparametersOp(trial=best_trial_op.output)\n",
|
||||
" best_hyperparameters_op = hyperparameter_tuning_job.GetHyperparametersOp(\n",
|
||||
" trial=best_trial_op.output\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model_dir_op = model_dir(base_output_directory, best_trial_op.output)\n",
|
||||
"\n",
|
||||
@@ -1160,7 +1105,7 @@
|
||||
" },\n",
|
||||
" ).after(model_dir_op)\n",
|
||||
"\n",
|
||||
" _ = ModelUploadOp(\n",
|
||||
" model_upload_op = ModelUploadOp(\n",
|
||||
" project=project,\n",
|
||||
" display_name=display_name,\n",
|
||||
" unmanaged_container_model=import_unmanaged_model_op.outputs[\"artifact\"],\n",
|
||||
@@ -1194,26 +1139,13 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "aEm7RuwMMoUG"
|
||||
"id": "create_hpt_specs"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google_cloud_pipeline_components.experimental import \\\n",
|
||||
" hyperparameter_tuning_job\n",
|
||||
"\n",
|
||||
"gpu = \"ACCELERATOR_TYPE_UNSPECIFIED\"\n",
|
||||
"accelerator_count = 0\n",
|
||||
"\n",
|
||||
"if TRAIN_GPU:\n",
|
||||
" gpu = TRAIN_GPU.name\n",
|
||||
" accelerator_count = 1\n",
|
||||
"\n",
|
||||
"else:\n",
|
||||
" gpu = \"ACCELERATOR_TYPE_UNSPECIFIED\"\n",
|
||||
" accelerator_count = (\n",
|
||||
" 0 # same problem with accelerator_count, if we keep is as \"None\" its not\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"CMDARGS = [\n",
|
||||
" \"--epochs=10\",\n",
|
||||
"]\n",
|
||||
@@ -1223,8 +1155,8 @@
|
||||
" {\n",
|
||||
" \"machine_spec\": {\n",
|
||||
" \"machine_type\": TRAIN_COMPUTE,\n",
|
||||
" \"accelerator_type\": gpu,\n",
|
||||
" \"accelerator_count\": accelerator_count,\n",
|
||||
" \"accelerator_type\": TRAIN_GPU.name,\n",
|
||||
" \"accelerator_count\": TRAIN_NGPU,\n",
|
||||
" },\n",
|
||||
" \"replica_count\": 1,\n",
|
||||
" \"container_spec\": {\"image_uri\": TRAIN_IMAGE, \"args\": CMDARGS},\n",
|
||||
@@ -1273,7 +1205,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "42YNp9Y9MoUG"
|
||||
"id": "run_pipeline:hpt"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1314,7 +1246,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4X3jrdX1MoUH"
|
||||
"id": "view_pipleline_results:hpt,horses_or_humans"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1426,7 +1358,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "hS53o3FcMoUH"
|
||||
"id": "delete_pipeline"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1446,6 +1378,14 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
@@ -1453,18 +1393,70 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "laAQFM4aoBm3"
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "get_started_with_hpt_pipeline_components.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -38,16 +38,9 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_kubeflow_pipelines.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_kubeflow_pipelines.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_kubeflow_pipelines.ipynb\">\n",
|
||||
" Open in Google Cloud Notebooks\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -74,7 +67,7 @@
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to use `Kubeflow Pipelines`(KFP).\n",
|
||||
"In this tutorial, you learn how to use `Kubeflow Pipelines`.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
@@ -85,17 +78,8 @@
|
||||
"- Building KFP lightweight Python function components.\n",
|
||||
"- Assembling and compiling KFP components into a pipeline.\n",
|
||||
"- Executing a KFP pipeline using Vertex AI Pipelines.\n",
|
||||
"- Loading component and pipeline definitions from a source code repository.\n",
|
||||
"- Building sequential, parallel, multiple output components.\n",
|
||||
"- Building control flow into pipelines.\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
"- Building control flow into pipelines."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -106,7 +90,7 @@
|
||||
"source": [
|
||||
"## Installations\n",
|
||||
"\n",
|
||||
"Install the following packages for executing this MLOps notebooks."
|
||||
"Install *one time* the packages for executing the MLOps notebooks."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -117,20 +101,20 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
" \n",
|
||||
"! pip3 install tensorflow-io==0.18 $USER_FLAG -q\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" pyarrow \\\n",
|
||||
" kfp $USER_FLAG -q"
|
||||
"ONCE_ONLY = False\n",
|
||||
"if ONCE_ONLY:\n",
|
||||
" ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
|
||||
" ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
|
||||
" ! pip3 install --upgrade kfp $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -238,10 +222,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -268,63 +249,6 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "b5627478895e"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "49ee8894d674"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -348,8 +272,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = \"gs://{}\".format(BUCKET_NAME)"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -360,8 +283,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -381,7 +304,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -401,7 +324,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -412,7 +335,7 @@
|
||||
"source": [
|
||||
"#### Service Account\n",
|
||||
"\n",
|
||||
"You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
|
||||
"**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -436,15 +359,13 @@
|
||||
"source": [
|
||||
"if (\n",
|
||||
" SERVICE_ACCOUNT == \"\"\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
" or SERVICE_ACCOUNT is None\n",
|
||||
" or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
|
||||
"):\n",
|
||||
" shell_output = ! gcloud projects describe $PROJECT_ID | sed -nre 's:.*projectNumber\\: (.*):\\1:p'\n",
|
||||
" SERVICE_ACCOUNT = (\n",
|
||||
" shell_output[0].replace(\"'\", \"\") + \"-compute@developer.gserviceaccount.com\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -455,7 +376,7 @@
|
||||
"source": [
|
||||
"#### Set service account access for Vertex AI Pipelines\n",
|
||||
"\n",
|
||||
"Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account."
|
||||
"Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -466,9 +387,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -477,7 +398,10 @@
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries"
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -487,11 +411,42 @@
|
||||
"id": "import_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_tf"
|
||||
},
|
||||
"source": [
|
||||
"#### Import TensorFlow\n",
|
||||
"\n",
|
||||
"Import the TensorFlow package into your Python environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_tf"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorflow as tf"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_kfp:namedtuple"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import NamedTuple\n",
|
||||
"\n",
|
||||
"import google.cloud.aiplatform as aiplatform\n",
|
||||
"import tensorflow as tf\n",
|
||||
"from kfp import dsl\n",
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2.dsl import component"
|
||||
@@ -516,7 +471,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -531,14 +486,14 @@
|
||||
"\n",
|
||||
" 1. Design the pipeline workflow.\n",
|
||||
" 2. Compile the pipeline.\n",
|
||||
" 3. Schedule pipeline execution (or run now).\n",
|
||||
" 3. Schedule execution (or run now) the pipeline.\n",
|
||||
" 4. Get the pipeline results.\n",
|
||||
"\n",
|
||||
"Pipelines are designed using domain specific language (DSL). Vertex AI Pipelines support both KFP DSL and TFX DSL for designing pipelines.\n",
|
||||
"Pipelines are designed using language specific domain specific language (DSL). Vertex AI Pipelines support both KFP DSL and TFX DSL for designing pipelines.\n",
|
||||
"\n",
|
||||
"In addition to designing components, you can use a wide variety of pre-built Google Cloud Pipeline Components for Vertex AI services.\n",
|
||||
"\n",
|
||||
"Learn more about [Building a pipeline](https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline)."
|
||||
"Learn more about [Building a pipeline](https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -547,9 +502,9 @@
|
||||
"id": "pipelines_intro:helloworld"
|
||||
},
|
||||
"source": [
|
||||
"## Basic pipeline\n",
|
||||
"## Basic pipeline introduction\n",
|
||||
"\n",
|
||||
"This step demonstrates the basics of constructing and executing a pipeline. You do the following:\n",
|
||||
"This demonstrates the basics of constructing and executing a pipeline. You do the following:\n",
|
||||
"\n",
|
||||
"1. Design a simple Python function based component to output the input string.\n",
|
||||
"2. Construct a pipeline that uses the component.\n",
|
||||
@@ -567,8 +522,8 @@
|
||||
"\n",
|
||||
"To create a KFP component from a Python function, you add the KFP DSL decorator `@component` to the function. In this example, the decorator takes the following parameters:\n",
|
||||
"\n",
|
||||
"- `output_component_file`(optional): write the component description to a YAML file such that the component is portable.\n",
|
||||
"- `base_image`(optional): The interpreter for executing the Python function. By default it is Python 3.7"
|
||||
"- `output_component_file`: (optional) write the component description to a YAML file such that the component is portable.\n",
|
||||
"- `base_image`: (optional): The interpreter for executing the Python function. By default it is Python 3.7"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -611,7 +566,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/hello_world\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/hello_world\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(\n",
|
||||
@@ -620,8 +575,7 @@
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
")\n",
|
||||
"def pipeline(text: str = \"hi there\"):\n",
|
||||
" hello_world_task = hello_world(text)\n",
|
||||
" return hello_world_task"
|
||||
" hello_world_task = hello_world(text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -634,7 +588,7 @@
|
||||
"\n",
|
||||
"Once the design of the pipeline is completed, the next step is to compile it. The pipeline definition is compiled into a JSON formatted file, which is transportable and can be interpreted by both KFP and Vertex AI Pipelines.\n",
|
||||
"\n",
|
||||
"Compile the pipeline with the Compiler().compile() method using the following parameters:\n",
|
||||
"You compile the pipeline with the method Compiler().compile(), with the following parameters:\n",
|
||||
"\n",
|
||||
"- `pipeline_func`: The corresponding DSL function that defines the pipeline.\n",
|
||||
"- `package_path`: The JSON file to write the transportable compiled pipeline to."
|
||||
@@ -661,14 +615,14 @@
|
||||
"source": [
|
||||
"### Execute the hello world pipeline\n",
|
||||
"\n",
|
||||
"Now that the pipeline is compiled, you can execute it by:\n",
|
||||
"Now that the pipeline is compiled, you can execute by:\n",
|
||||
"\n",
|
||||
"- Creating a Vertex AI PipelineJob with the following parameters:\n",
|
||||
"- Create a Vertex AI PipelineJob, with the following parameters:\n",
|
||||
" - `display_name`: The human readable name for the job.\n",
|
||||
" - `template_path`: The compiled JSON pipeline definition.\n",
|
||||
" - `template_path`: Thee compiled JSON pipeline definition.\n",
|
||||
" - `pipeline_root`: Where to write output artifacts to.\n",
|
||||
"\n",
|
||||
"Click on the generated link below `INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:` to see your job run in the Cloud Console."
|
||||
"Click on the generated link below `INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:` to see your run in the Cloud Console."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -679,7 +633,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"hello_world\",\n",
|
||||
" template_path=\"hello_world.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -740,29 +694,14 @@
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/gcp_resources\"\n",
|
||||
" )\n",
|
||||
" EVAL_METRICS = (\n",
|
||||
" PIPELINE_ROOT\n",
|
||||
" + \"/\"\n",
|
||||
" + PROJECT_NUMBER\n",
|
||||
" + \"/\"\n",
|
||||
" + JOB_ID\n",
|
||||
" + \"/\"\n",
|
||||
" + output_task_name\n",
|
||||
" + \"_\"\n",
|
||||
" + str(TASK_ID)\n",
|
||||
" + \"/evaluation_metrics\"\n",
|
||||
" )\n",
|
||||
" if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
|
||||
" ! gsutil cat $EXECUTE_OUTPUT\n",
|
||||
" return EXECUTE_OUTPUT\n",
|
||||
" break\n",
|
||||
" elif tf.io.gfile.exists(GCP_RESOURCES):\n",
|
||||
" ! gsutil cat $GCP_RESOURCES\n",
|
||||
" return GCP_RESOURCES\n",
|
||||
" elif tf.io.gfile.exists(EVAL_METRICS):\n",
|
||||
" ! gsutil cat $EVAL_METRICS\n",
|
||||
" return EVAL_METRICS\n",
|
||||
" break\n",
|
||||
"\n",
|
||||
" return None\n",
|
||||
" return EXECUTE_OUTPUT\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print_pipeline_output(pipeline, \"hello-world\")"
|
||||
@@ -776,7 +715,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -802,7 +741,7 @@
|
||||
"\n",
|
||||
" hello_world_op = components.load_component_from_file('./hello_world.yaml').\n",
|
||||
"\n",
|
||||
"You can also use the `load_component_from_url` method, if your component YAML file is stored online, such as in a git repository."
|
||||
"You can also use the load_component_from_url method, if your component YAML file is stored online, such as if in a git repo."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -815,7 +754,7 @@
|
||||
"source": [
|
||||
"from kfp import components\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/hello_world-v2\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/hello_world-v2\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"hello_world_op = components.load_component_from_file(\"./hello_world.yaml\")\n",
|
||||
"\n",
|
||||
@@ -826,13 +765,12 @@
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
")\n",
|
||||
"def pipeline(text: str = \"hi there\"):\n",
|
||||
" hello_world_task = hello_world_op(text)\n",
|
||||
" return hello_world_task\n",
|
||||
" hellow_world_task = hello_world_op(text)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"hello_world-v2.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"hello_world-v2\",\n",
|
||||
" template_path=\"hello_world-v2.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -851,74 +789,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "delete_pipeline"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pipeline.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "load_component_pipeline_git:helloworld"
|
||||
},
|
||||
"source": [
|
||||
"### Loading components and pipeline YAML definitions from source control\n",
|
||||
"\n",
|
||||
"By storing the component and pipeline definitions in a source repository, like Github, you can version control your components and pipelines, as follows:\n",
|
||||
"\n",
|
||||
"- Use the method `load_component_from_url()`.\n",
|
||||
"\n",
|
||||
"- Pull the raw file format version from the repo. For github, that will be in the form of:\n",
|
||||
"\n",
|
||||
" https://raw.githubusercontent.com/....\n",
|
||||
"\n",
|
||||
"- Specify the version of the component/pipeline. For github, that will be the branch."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "load_component_pipeline_git:helloworld"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"VERSION = \"main\"\n",
|
||||
"hello_world_op = components.load_component_from_url(\n",
|
||||
" f\"https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.yaml\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"! wget https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.json -O hello_git_example.json\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
" display_name=\"hello_world-git\",\n",
|
||||
" template_path=\"hello_git_example.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"pipeline.run()\n",
|
||||
"\n",
|
||||
"! rm -f hello_git_example.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "delete_pipeline"
|
||||
},
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -958,7 +829,7 @@
|
||||
" return np.mean(values)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/numpy_mean\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/numpy_mean\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@dsl.pipeline(\n",
|
||||
@@ -966,12 +837,11 @@
|
||||
")\n",
|
||||
"def pipeline(values: list = [2, 3]):\n",
|
||||
" numpy_task = numpy_mean(values)\n",
|
||||
" return numpy_task\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"numpy_mean.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"numpy_mean\",\n",
|
||||
" template_path=\"numpy_mean.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -992,7 +862,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1027,7 +897,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/add_div2\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/add_div2\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component(output_component_file=\"add.yaml\", base_image=\"python:3.9\")\n",
|
||||
@@ -1046,12 +916,11 @@
|
||||
"def pipeline(v1: int = 4, v2: int = 5):\n",
|
||||
" add_task = add(v1, v2)\n",
|
||||
" div2_task = div_by_2(add_task.output)\n",
|
||||
" return div2_task\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"add_div2.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"add_div2\",\n",
|
||||
" template_path=\"add_div2.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1072,7 +941,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1094,7 +963,7 @@
|
||||
"source": [
|
||||
"### Multiple output pipeline\n",
|
||||
"\n",
|
||||
"Next, you design and execute a pipeline where a first component has multiple outputs, which are then used as inputs to the next component. To distinguish between the outputs, when used as inputs to the next component, you follow:\n",
|
||||
"Next, you design and execute a pipeline where a first component has multiple outputs, which are then used as inputs to the next component. To distinquish between the outputs, when used as inputs to the next component, you do:\n",
|
||||
"\n",
|
||||
"1. Set the function return type to `NamedTuple`.\n",
|
||||
"2. In NamedTuple, specify a name and type for each output, in the specified order.\n",
|
||||
@@ -1109,7 +978,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/multi_output\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/multi_output\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component()\n",
|
||||
@@ -1143,12 +1012,11 @@
|
||||
" multi_output_task.outputs[\"output_1\"],\n",
|
||||
" multi_output_task.outputs[\"output_2\"],\n",
|
||||
" )\n",
|
||||
" return concat_task\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"multi_output.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"multi-output\",\n",
|
||||
" template_path=\"multi_output.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1169,7 +1037,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1191,9 +1059,9 @@
|
||||
"source": [
|
||||
"## Parallel tasks in component\n",
|
||||
"\n",
|
||||
"Next, you design and execute a pipeline with parallel tasks. In this example, one parallel task adds up a list of integers and another substracts them. Note that the compiler knows these two tasks can be run in parallel, because their input is not dependent on the output of the other task.\n",
|
||||
"Next, you design and execute a pipeline with parallel tasks. In this example, one parallel task adds up a list of integers and another substracts them. Note that the compiler knows these two tasks can be ran in parallel, because their input is not dependent on the output of the other task.\n",
|
||||
"\n",
|
||||
"Finally, the `add_int` task waits on the two parallel tasks to complete, and then adds together the two outputs."
|
||||
"Finally, the add task waits on the two parallel tasks to complete, and then adds together the two outputs."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1204,14 +1072,14 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/parallel\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/parallel\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component()\n",
|
||||
"def add_list(values: list) -> int:\n",
|
||||
" ret = 0\n",
|
||||
" for value in values:\n",
|
||||
" ret = value + ret\n",
|
||||
" ret += 1\n",
|
||||
" return ret\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -1219,12 +1087,12 @@
|
||||
"def sub_list(values: list) -> int:\n",
|
||||
" ret = 0\n",
|
||||
" for value in values:\n",
|
||||
" ret = value - ret\n",
|
||||
" ret -= 1\n",
|
||||
" return ret\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component()\n",
|
||||
"def add_int(value1: int, value2: int) -> int:\n",
|
||||
"def add(value1: int, value2: int) -> int:\n",
|
||||
" return value1 + value2\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -1234,13 +1102,12 @@
|
||||
"def pipeline(values: list = [1, 2, 3]):\n",
|
||||
" add_list_task = add_list(values)\n",
|
||||
" sub_list_task = sub_list(values)\n",
|
||||
" add_task = add_int(add_list_task.output, sub_list_task.output)\n",
|
||||
" return add_task\n",
|
||||
" add_task = add(add_list_task.output, sub_list_task.output)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"parallel.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"parallel\",\n",
|
||||
" template_path=\"parallel.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1261,7 +1128,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1283,7 +1150,7 @@
|
||||
"source": [
|
||||
"## Control flow in pipeline\n",
|
||||
"\n",
|
||||
"While Python control statements(e.g., if/else, for) can be used in a component, they cannot be used in a pipeline function. Each task in a pipeline function runs as a node in a graph. Thus a control flow statement also has to run as a graph node. To support this, KFP provides a set of DSL statements that implement control flow as a graph node."
|
||||
"While Python control statements, e.g., if/else, for, can be used in a component, they cannot be used in the pipeline function. Each task in the pipeline function runs as a node in a graph. Thus a control flow statement also has to run as a graph node. To support this, KFP provides a set of DSL statements that implement control flow as a graph node."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1294,7 +1161,7 @@
|
||||
"source": [
|
||||
"### dsl.ParallelFor\n",
|
||||
"\n",
|
||||
"The statement `dsl.ParallelFor()` implements a `for` loop, where each iteration in the `for` loop runs in parallel."
|
||||
"The statement `dsl.ParallelFor()` implements a for loop, where each iteration in the for loop runs in parallel."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1305,7 +1172,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/parallel_for\".format(BUCKET_URI)\n",
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/parallel_for\".format(BUCKET_NAME)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@component()\n",
|
||||
@@ -1327,12 +1194,11 @@
|
||||
" with dsl.ParallelFor(values) as item:\n",
|
||||
" output = double(item).output\n",
|
||||
" echo_task = echo(output)\n",
|
||||
" return echo_task\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"parallel_for.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"parallel-for\",\n",
|
||||
" template_path=\"parallel_for.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1353,7 +1219,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1375,15 +1241,7 @@
|
||||
"source": [
|
||||
"### dsl.Condition\n",
|
||||
"\n",
|
||||
"The statement `dsl.Condition()` implements an `if` statement. There is no support for an `else` or `elif` statement. You use a separate `dsl.Condition()` for each value you want to test for. For example, if the output from a task is `1` or `0`, you will have two `dsl.Condition()` statements, one for 1 and one for 0.\n",
|
||||
"\n",
|
||||
"The condition in `dsl.Condition()` is evaluated at run-time, not compile time. As such it is not Python code anymore. The condition is of type `ConditionOperator`. This operator has three parts:\n",
|
||||
"\n",
|
||||
"1. PipelineParam or task output\n",
|
||||
"2. == or !=\n",
|
||||
"3. string or integer value\n",
|
||||
"\n",
|
||||
"A `dsl.Condition()` can be named using the `name` parameter while defining the condition."
|
||||
"The statement `dsl.Condition()` implements an `if` statement. There is no support for an `else` or `elif` statement. You use a separate `dsl.Condition()` for each value you want to test for. For example, if the output from a task is `True` or `False`, you will have two `dsl.Condition()` statements, one for True and one for False."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1422,12 +1280,11 @@
|
||||
" task = heads()\n",
|
||||
" with dsl.Condition(flip_task.output == 0, name=\"false_clause\"):\n",
|
||||
" task = tails()\n",
|
||||
" return task\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"condition.json\")\n",
|
||||
"\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"condition\",\n",
|
||||
" template_path=\"condition.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1448,7 +1305,7 @@
|
||||
"source": [
|
||||
"### Delete a pipeline job\n",
|
||||
"\n",
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the `delete()` method. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
"After a pipeline job is completed, you can delete the pipeline job with the method `delete()`. Prior to completion, a pipeline job can be canceled with the method `cancel()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1468,14 +1325,16 @@
|
||||
"id": "pipeline_errata"
|
||||
},
|
||||
"source": [
|
||||
"## Errata\n",
|
||||
"\n",
|
||||
"### Caching in pipeline components\n",
|
||||
"\n",
|
||||
"When running a pipeline with Vertex AI Pipelines, the outcome state of each task is cached. With caching, if the pipeline is run again, and the compiled definition of the task and state has not changed, the cached output will be used instead of running the task again.\n",
|
||||
"When running a pipeline with Vertex AI Pipelines, the outcome state of each task is cached. With caching, if the pipeline is ran again, and the compiled definition of the task and state has not changed, the cached output will be used instead of running the task again.\n",
|
||||
"\n",
|
||||
"To override caching, i.e., force run the task, you set the parameter `enable_caching` to `False` when creating the Vertex AI Pipeline job.\n",
|
||||
"To override caching, i.e., forceable run the task, you set the parameter `enable_caching` to `False` when creating the Vertex AI Pipeline job.\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"pipeline = aiplatform.PipelineJob(\n",
|
||||
"pipeline = aip.PipelineJob(\n",
|
||||
" display_name=\"example\",\n",
|
||||
" template_path=\"example.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
@@ -1485,11 +1344,11 @@
|
||||
"\n",
|
||||
"### Asynchronous execution of pipeline\n",
|
||||
"\n",
|
||||
"When running a pipeline with the method `run()`, the pipeline is run synchronously. To run asynchronously, you use the method `submit()`. Once the job has started, your Python script can continue to execute. To block execution, you can use the method `wait()`.\n",
|
||||
"When running a pipeline with the method `run()`, the pipeline is ran synchronously. To run asynchronously, you use the method `submit()`. Once the job has started, your Python script can continue to execute. Then when you need to block execution using the method `wait()`.\n",
|
||||
"\n",
|
||||
"### Setting machine resources for pipeline steps\n",
|
||||
"\n",
|
||||
"By default, Vertex AI Pipelines automatically finds the best matching machine type to run the component. You can override and specify the machine resources on a per component basis, when you invoke the component in a pipeline, as follows:\n",
|
||||
"By default, Vertex AI Pipelines will automatically find the best matching machine type to run the component. You can override and specify the machine resources on a per component basis, when you invoke the component in a pipeline, as follows:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"@dsl.pipeline(name='my-pipeline')\n",
|
||||
@@ -1517,9 +1376,15 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"### Cloud Storage Bucket\n",
|
||||
"\n",
|
||||
"Set `delete_bucket` to True to delete the Cloud storage bucket used in this notebook."
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1530,10 +1395,61 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = False\n",
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" try:\n",
|
||||
" if \"dataset\" in globals():\n",
|
||||
" dataset.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" try:\n",
|
||||
" if \"model\" in globals():\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" try:\n",
|
||||
" if \"endpoint\" in globals():\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the AutoML or Pipeline training job\n",
|
||||
" try:\n",
|
||||
" if \"dag\" in globals():\n",
|
||||
" dag.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the custom training job\n",
|
||||
" try:\n",
|
||||
" if \"job\" in globals():\n",
|
||||
" job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the batch prediction job using the Vertex batch prediction object\n",
|
||||
" try:\n",
|
||||
" if \"batch_predict_job\" in globals():\n",
|
||||
" batch_predict_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
|
||||
" try:\n",
|
||||
" if \"hpt_job\" in globals():\n",
|
||||
" hpt_job.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+88
-83
@@ -32,20 +32,14 @@
|
||||
"# E2E ML on GCP: MLOps stage 3 : Get started with rapid prototyping with AutoML and BQML\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_rapid_prototyping_bqml_automl.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_rapid_prototyping_bqml_automl.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_rapid_prototyping.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_rapid_prototyping_bqml_automl.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_rapid_prototyping.ipynb\">\n",
|
||||
" Open in Vertex Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
@@ -258,7 +252,9 @@
|
||||
"if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"else:\n",
|
||||
" USER_FLAG = \"\""
|
||||
" USER_FLAG = \"\"\n",
|
||||
"\n",
|
||||
"! pip3 install --quiet --upgrade google-cloud-aiplatform {USER_FLAG}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -278,7 +274,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --quiet --upgrade google-cloud-aiplatform {USER_FLAG}\n",
|
||||
"! pip3 install {USER_FLAG} --quiet -U google-cloud-pipeline-components==1.0 kfp\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-bigquery"
|
||||
]
|
||||
@@ -419,8 +414,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
@@ -478,9 +471,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -676,7 +667,6 @@
|
||||
"from typing import NamedTuple\n",
|
||||
"\n",
|
||||
"import google.cloud.aiplatform as aip\n",
|
||||
"from google.cloud import bigquery\n",
|
||||
"from kfp import dsl\n",
|
||||
"from kfp.v2 import compiler\n",
|
||||
"from kfp.v2.dsl import Artifact, Input, Metrics, Output, component"
|
||||
@@ -1439,17 +1429,6 @@
|
||||
"- `validate_infrastructure`: Validate the deployed model serving infrastructure."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "040e82bc1646"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"DISPLAY_NAME = \"rapid-prototyping\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -1474,7 +1453,8 @@
|
||||
" from google_cloud_pipeline_components.types import artifact_types\n",
|
||||
" from google_cloud_pipeline_components.v1.bigquery import (\n",
|
||||
" BigqueryCreateModelJobOp, BigqueryEvaluateModelJobOp,\n",
|
||||
" BigqueryExportModelJobOp)\n",
|
||||
" BigqueryExportModelJobOp, BigqueryPredictModelJobOp,\n",
|
||||
" BigqueryQueryJobOp)\n",
|
||||
" from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
|
||||
" ModelDeployOp)\n",
|
||||
" from google_cloud_pipeline_components.v1.model import ModelUploadOp\n",
|
||||
@@ -1690,7 +1670,7 @@
|
||||
"PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root\"\n",
|
||||
"image_prefix = REGION.split(\"-\")[0]\n",
|
||||
"BQML_SERVING_CONTAINER_IMAGE_URI = (\n",
|
||||
" f\"{image_prefix}-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-8:latest\"\n",
|
||||
" f\"{image_prefix}-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-6:latest\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"BQ_DATASET = \"rapid_prototype\" # j90wipxexhrgq3cquanc5\" # @param {type:\"string\"}\n",
|
||||
@@ -1698,6 +1678,7 @@
|
||||
"BQ_LOCATION = BQ_LOCATION.upper()\n",
|
||||
"BQML_EXPORT_LOCATION = f\"{BUCKET_URI}/artifacts/bqml\"\n",
|
||||
"\n",
|
||||
"DISPLAY_NAME = \"rapid-prototyping\"\n",
|
||||
"ENDPOINT_DISPLAY_NAME = f\"{DISPLAY_NAME}_endpoint\"\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(\n",
|
||||
@@ -1725,7 +1706,7 @@
|
||||
" template_path=PIPELINE_JSON_PKG_PATH,\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
" parameter_values=pipeline_params,\n",
|
||||
" enable_caching=False,\n",
|
||||
" enable_caching=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response = pipeline_job.submit()"
|
||||
@@ -1777,72 +1758,96 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = True\n",
|
||||
"delete = True # set to True if you want to delete resources created in this tutorial.\n",
|
||||
"\n",
|
||||
"print(\"Will delete endpoint\")\n",
|
||||
"\n",
|
||||
"endpoints = aip.Endpoint.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
|
||||
")\n",
|
||||
"endpoint = endpoints[0]\n",
|
||||
"endpoint.undeploy_all()\n",
|
||||
"aip.Endpoint.delete(endpoint.resource_name)\n",
|
||||
"print(\"Deleted endpoint:\", endpoint)\n",
|
||||
"delete_vertex_dataset = True and delete\n",
|
||||
"delete_pipeline = True and delete\n",
|
||||
"delete_model = True and delete\n",
|
||||
"delete_endpoint = True and delete\n",
|
||||
"delete_batchjob = True and delete\n",
|
||||
"delete_bucket = True and delete\n",
|
||||
"delete_bq_dataset = True and delete\n",
|
||||
"\n",
|
||||
"print(\"Will delete models\")\n",
|
||||
"suffix_list = [\"bqml\", \"automl\", \"best\"]\n",
|
||||
"for suffix in suffix_list:\n",
|
||||
"try:\n",
|
||||
" if delete_endpoint and \"DISPLAY_NAME\" in globals():\n",
|
||||
" print(\"Will delete endpoint\")\n",
|
||||
" endpoints = aip.Endpoint.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" endpoint = endpoints[0]\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" aip.Endpoint.delete(endpoint.resource_name)\n",
|
||||
" print(\"Deleted endpoint:\", endpoint)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_model and \"DISPLAY_NAME\" in globals():\n",
|
||||
" print(\"Will delete models\")\n",
|
||||
" suffix_list = [\"bqml\", \"automl\", \"best\"]\n",
|
||||
" for suffix in suffix_list:\n",
|
||||
" try:\n",
|
||||
" model_display_name = f\"{DISPLAY_NAME}_{suffix}\"\n",
|
||||
" print(\"Will delete model with name \" + model_display_name)\n",
|
||||
" models = aip.Model.list(\n",
|
||||
" filter=f\"display_name={model_display_name}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model = models[0]\n",
|
||||
" aip.Model.delete(model)\n",
|
||||
" print(\"Deleted model:\", model)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_vertex_dataset and \"DISPLAY_NAME\" in globals():\n",
|
||||
" print(\"Will delete Vertex dataset\")\n",
|
||||
" try:\n",
|
||||
" model_display_name = f\"{DISPLAY_NAME}_{suffix}\"\n",
|
||||
" print(\"Will delete model with name \" + model_display_name)\n",
|
||||
" models = aip.Model.list(\n",
|
||||
" filter=f\"display_name={model_display_name}\", order_by=\"create_time\"\n",
|
||||
" datasets = aip.TabularDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model = models[0]\n",
|
||||
" aip.Model.delete(model)\n",
|
||||
" print(\"Deleted model:\", model)\n",
|
||||
" dataset = datasets[0]\n",
|
||||
" aip.TabularDataset.delete(dataset)\n",
|
||||
" print(\"Deleted Vertex dataset:\", dataset)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(\"Will delete Vertex dataset\")\n",
|
||||
"try:\n",
|
||||
" if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
|
||||
" pipelines = aip.PipelineJob.list(\n",
|
||||
" filter=f\"pipeline_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" pipeline = pipelines[0]\n",
|
||||
" aip.PipelineJob.delete(pipeline)\n",
|
||||
" print(\"Deleted pipeline:\", pipeline)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"datasets = aip.TabularDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
")\n",
|
||||
"if delete_bq_dataset and \"DISPLAY_NAME\" in globals():\n",
|
||||
" from google.cloud import bigquery\n",
|
||||
"\n",
|
||||
"dataset = datasets[0]\n",
|
||||
"aip.TabularDataset.delete(dataset)\n",
|
||||
"print(\"Deleted Vertex dataset:\", dataset)\n",
|
||||
" try:\n",
|
||||
" # Construct a BigQuery client object.\n",
|
||||
"\n",
|
||||
" bq_client = bigquery.Client(project=PROJECT_ID, location=BQ_LOCATION)\n",
|
||||
"\n",
|
||||
" # TODO(developer): Set model_id to the ID of the model to fetch.\n",
|
||||
" dataset_id = f\"{PROJECT_ID}.{BQ_DATASET}\"\n",
|
||||
"\n",
|
||||
" print(f\"Will delete BQ dataset '{dataset_id}' from location {BQ_LOCATION}.\")\n",
|
||||
" # Use the delete_contents parameter to delete a dataset and its contents.\n",
|
||||
" # Use the not_found_ok parameter to not receive an error if the dataset has already been deleted.\n",
|
||||
" bq_client.delete_dataset(\n",
|
||||
" dataset_id, delete_contents=True, not_found_ok=True\n",
|
||||
" ) # Make an API request.\n",
|
||||
"\n",
|
||||
" print(f\"Deleted BQ dataset '{dataset_id}' from location {BQ_LOCATION}.\")\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"pipelines = aip.PipelineJob.list(\n",
|
||||
" filter=f\"pipeline_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
")\n",
|
||||
"pipeline = pipelines[0]\n",
|
||||
"aip.PipelineJob.delete(pipeline)\n",
|
||||
"print(\"Deleted pipeline:\", pipeline)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Construct a BigQuery client object.\n",
|
||||
"\n",
|
||||
"bq_client = bigquery.Client(project=PROJECT_ID, location=BQ_LOCATION)\n",
|
||||
"\n",
|
||||
"# TODO(developer): Set dataset_id to the ID of the dataset to fetch.\n",
|
||||
"dataset_id = f\"{PROJECT_ID}.{BQ_DATASET}\"\n",
|
||||
"\n",
|
||||
"print(f\"Will delete BQ dataset '{dataset_id}' from location {BQ_LOCATION}.\")\n",
|
||||
"# Use the delete_contents parameter to delete a dataset and its contents.\n",
|
||||
"# Use the not_found_ok parameter to not receive an error if the dataset has already been deleted.\n",
|
||||
"bq_client.delete_dataset(\n",
|
||||
" dataset_id, delete_contents=True, not_found_ok=True\n",
|
||||
") # Make an API request.\n",
|
||||
"\n",
|
||||
"print(f\"Deleted BQ dataset '{dataset_id}' from location {BQ_LOCATION}.\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
"if delete_bucket and \"BUCKET_URI\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
# Stage 5: Deployment
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
## Recommendations
|
||||
|
||||
|
||||
<img src='stage5.png'>
|
||||
|
||||
## Notebooks
|
||||
|
||||
### Get Started
|
||||
|
||||
|
||||
[Get started with Vertex AI Endpoints](get_started_with_vertex_endpoints.ipynb)
|
||||
|
||||
```
|
||||
The steps include:
|
||||
|
||||
- Creating an `Endpoint` resource.
|
||||
- List all `Endpoint` resources.
|
||||
- List `Endpoint` resources by query filter.
|
||||
- Configuring the serving binary of a `Model` resource for deployment to an `Endpoint` resource.
|
||||
- Deploying a single `Model` resource to an `Endpoint` resource.
|
||||
- Get deployment settings for a deployed `Model` resource.
|
||||
- Configuring auto-scaling.
|
||||
- Deploying multiple `Model` resources to an `Endpoint` resource and configuring a traffic split.
|
||||
- Dynamically change the traffic split for an `Endpoint` resource.
|
||||
- Undeploy a single `Model` resource to an `Endpoint` resource.
|
||||
- Undeploy all `Model` resources from an `Endpoint` resource.
|
||||
- Delete an `Endpoint` resource.
|
||||
- In pipeline: Create an `Endpoint` resource and deploy an existing `Model` resource to the `Endpoint` resource.
|
||||
- In pipeline: Deploy an existing `Model` resource to an existing `Endpoint` resource.
|
||||
```
|
||||
|
||||
[Get started with Vertex AI Private Endpoints](get_started_with_vertex_private_endpoints.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Creating a `Private Endpoint` resource.
|
||||
- Configure a VPC peering connection.
|
||||
- Configuring the serving binary of a `Model` resource for deployment to a `Private Endpoint` resource.
|
||||
- Deploying a `Model` resource to a `Private Endpoint` resource.
|
||||
- Send a prediction request to a `Private Endpoint`
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -23,9 +23,7 @@ This stage may be done entirely by MLOps. We recommend:
|
||||
- Features that dynamically change per example (e.g., bank balance) are stored in Vertex Feature Store.
|
||||
|
||||
|
||||
<img src='stage6a.png'>
|
||||
<img src='stage6b.png'>
|
||||
<img src='stage6c.png'>
|
||||
<img src='stage6.png'>
|
||||
|
||||
## Notebooks
|
||||
|
||||
@@ -58,74 +56,3 @@ The steps performed include:
|
||||
- Upload and deploy the model serving container to Vertex AI Endpoint.
|
||||
- Make a prediction request.
|
||||
```
|
||||
|
||||
[Get started with Vertex AI Raw Prediction](get_started_with_raw_predict.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a pretrained tabular classification model artifacts for a TensorFlow 1.x estimator.
|
||||
- Upload the TensorFlow estimator model as a `Vertex AI Model` resource.
|
||||
- Creating an `Endpoint` resource.
|
||||
- Deploying the `Model` resource to an `Endpoint` resource.
|
||||
- Make an online raw prediction to the `Model` resource instance deployed to the `Endpoint` resource.
|
||||
```
|
||||
|
||||
|
||||
[Get started with serving functions for TensorFlow model](get_started_with_tf_serving_function.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a pretrained image classification model from TensorFlow Hub.
|
||||
- Create a serving function to receive compressed image data, and output decomopressed preprocessed data for the model input.
|
||||
- Upload the TensorFlow Hub model and serving function as a `Vertex AI Model` resource.
|
||||
- Creating an `Endpoint` resource.
|
||||
- Deploying the `Model` resource to an `Endpoint` resource.
|
||||
- Make an online prediction to the `Model` resource instance deployed to the `Endpoint` resource.
|
||||
```
|
||||
|
||||
[Get started with TensorFlow Serving](get_started_with_tf_serving.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a pretrained image classification model from TensorFlow Hub.
|
||||
- Create a serving function to receive compressed image data, and output decomopressed preprocessed data for the model input.
|
||||
- Upload the TensorFlow Hub model and serving function as a `Vertex AI Model` resource.
|
||||
- Creating an `Endpoint` resource.
|
||||
- Deploying the `Model` resource to an `Endpoint` resource with `TensorFlow Serving` serving binary.
|
||||
- Make an online prediction to the `Model` resource instance deployed to the `Endpoint` resource.
|
||||
```
|
||||
|
||||
[Get started with FastAPI serving binary](get_started_with_fastapi.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a pretrained image classification model from TensorFlow Hub.
|
||||
- Create a serving function to receive compressed image data, and output decomopressed preprocessed data for the model input.
|
||||
- Upload the TensorFlow Hub model and serving function as a `Vertex AI Model` resource.
|
||||
- Creating an `Endpoint` resource.
|
||||
- Deploying the `Model` resource to an `Endpoint` resource with `FastAPI` custom serving binary.
|
||||
- Make an online prediction to the `Model` resource instance deployed to the `Endpoint` resource.
|
||||
```
|
||||
|
||||
[Get started with Vertex AI Prediction optimized TensorFlow run-time container](get_started_with_optimized_tfe_bert.ipynb)
|
||||
|
||||
```
|
||||
The steps performed include:
|
||||
|
||||
- Download a pretrained BERT model from TensorFlow Hub.
|
||||
- Fine-tune (transfer learning) the BERT model as a binary classifier.
|
||||
- Upload the TensorFlow Hub model as a `Vertex AI Model` resource, with standard TensorFlow serving container.
|
||||
- Upload the TensorFlow Hub model as a `Vertex AI Model` resource, with `Vertex AI Prediction optimized TensorFlow runtime` container
|
||||
- Create two `Endpoint` resources.
|
||||
- Deploying both `Model` resources to separate `Endpoint` resources.
|
||||
- Make the same online prediction requests to both `Model` resource instances deployed to the `Endpoint` resources.
|
||||
- Compare the prediction accuracy between the two deployed `Model` resources.
|
||||
- Configuring container settings for fine-tune control of optimizations.
|
||||
- Create a `Private Endpoint` resource.
|
||||
- Deploy the `Model` resoure with then `Vertex AI Prediction optimized TensorFlow runtime` to the `Private Endpoint` resource.
|
||||
- Make an online prediction request to the `Private Endpoint` resource.
|
||||
```
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,909 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "title:generic,gcp"
|
||||
},
|
||||
"source": [
|
||||
"# E2E ML on GCP: MLOps stage 6 : Get started with TensorFlow serving functions with Vertex AI Raw Prediction\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_vertex_raw_predict.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_vertex_raw_predict.ipynb\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
"<br/><br/><br/>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how use `Vertex AI Raw Prediction` to send raw HTTP content directly to a model deployed to a `Vertex AI Endpoint`. \n",
|
||||
"\n",
|
||||
"For example, the HTTP server for pre-built `Vertex AI` deployment containers does not support the HTTP request body for TensorFlow 1.x estimators. Using raw predict, one can send raw content through the HTTP server that is presented to the model input as-is -- no canonical processing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c9402cfbdc2d"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to use `Vertex AI Raw Prediction` on a `Vertex AI Endpoint` resource.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services and resources:\n",
|
||||
"\n",
|
||||
"- `Vertex AI Raw Prediction`\n",
|
||||
"- `Vertex AI Models`\n",
|
||||
"- `Vertex AI Endpoints`\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Download a pretrained tabular classification model artifacts for a TensorFlow 1.x estimator.\n",
|
||||
"- Upload the TensorFlow estimator model as a `Vertex AI Model` resource.\n",
|
||||
"- Creating an `Endpoint` resource.\n",
|
||||
"- Deploying the `Model` resource to an `Endpoint` resource.\n",
|
||||
"- Make an online raw prediction to the `Model` resource instance deployed to the `Endpoint` resource."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dataset:iris,lcn"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"This tutorial uses a pre-trained tabular classification model from a public Cloud Storage bucket, which is trained on the Penguins dataset (https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "costs"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "install_aip"
|
||||
},
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the following packages to execute this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_aip"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook\n",
|
||||
"if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"else:\n",
|
||||
" USER_FLAG = \"\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"! pip3 install tensorflow-hub $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hhq5zEbGg0XX"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "EzrelQZ22IZj"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "before_you_begin"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### GPU runtime\n",
|
||||
"\n",
|
||||
"*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
|
||||
"\n",
|
||||
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "autoset_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID:\", PROJECT_ID)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_gcloud_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gcloud config set project $PROJECT_ID"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable, which is used for operations\n",
|
||||
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
|
||||
"\n",
|
||||
"- Americas: `us-central1`\n",
|
||||
"- Europe: `europe-west4`\n",
|
||||
"- Asia Pacific: `asia-east1`\n",
|
||||
"\n",
|
||||
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "timestamp"
|
||||
},
|
||||
"source": [
|
||||
"#### Timestamp\n",
|
||||
"\n",
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "gcp_authenticate"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"**Click Create service account**.\n",
|
||||
"\n",
|
||||
"In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gcp_authenticate"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "bucket:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"### Create a Cloud Storage bucket\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
|
||||
"\n",
|
||||
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "autoset_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
|
||||
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"source": [
|
||||
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aip\n",
|
||||
"import tensorflow as tf\n",
|
||||
"import tensorflow_hub as hub"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "accelerators:training,cpu,prediction,cpu,mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"#### Set hardware accelerators\n",
|
||||
"\n",
|
||||
"You can set hardware accelerators for training and prediction.\n",
|
||||
"\n",
|
||||
"Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
|
||||
"\n",
|
||||
" (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
|
||||
"\n",
|
||||
"Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n",
|
||||
"\n",
|
||||
"*Note*: TF releases before 2.3 for GPU support will fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3. This is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "accelerators:training,cpu,prediction,cpu,mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
|
||||
" DEPLOY_GPU, DEPLOY_NGPU = (\n",
|
||||
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
|
||||
" int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "container:training,prediction"
|
||||
},
|
||||
"source": [
|
||||
"#### Set pre-built containers\n",
|
||||
"\n",
|
||||
"Set the pre-built Docker container image for prediction.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "container:training,prediction"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING_TF\"):\n",
|
||||
" TF = os.getenv(\"IS_TESTING_TF\")\n",
|
||||
"else:\n",
|
||||
" TF = \"2.5\".replace(\".\", \"-\")\n",
|
||||
"\n",
|
||||
"if TF[0] == \"2\":\n",
|
||||
" if DEPLOY_GPU:\n",
|
||||
" DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
|
||||
" else:\n",
|
||||
" DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
|
||||
"else:\n",
|
||||
" if DEPLOY_GPU:\n",
|
||||
" DEPLOY_VERSION = \"tf-gpu.{}\".format(TF)\n",
|
||||
" else:\n",
|
||||
" DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n",
|
||||
"\n",
|
||||
"DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
|
||||
" REGION.split(\"-\")[0], DEPLOY_VERSION\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "machine:training"
|
||||
},
|
||||
"source": [
|
||||
"#### Set machine type\n",
|
||||
"\n",
|
||||
"Next, set the machine type to use for prediction.\n",
|
||||
"\n",
|
||||
"- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for prediction.\n",
|
||||
" - `machine type`\n",
|
||||
" - `n1-standard`: 3.75GB of memory per vCPU.\n",
|
||||
" - `n1-highmem`: 6.5GB of memory per vCPU\n",
|
||||
" - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
|
||||
" - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
|
||||
"\n",
|
||||
"*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "machine:training"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n",
|
||||
" MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n",
|
||||
"else:\n",
|
||||
" MACHINE_TYPE = \"n1-standard\"\n",
|
||||
"\n",
|
||||
"VCPU = \"4\"\n",
|
||||
"DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
|
||||
"print(\"Train machine type\", DEPLOY_COMPUTE)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d8128b8ff025"
|
||||
},
|
||||
"source": [
|
||||
"## Get pretrained model from the public Cloud Storage location\n",
|
||||
"\n",
|
||||
"For demonstration purposes, this tutorial uses a pretrained model TensorFlow 1.x estimator tabular classification Model, which is then uploaded to a `Vertex AI Model` resource. Once you have a `Vertex AI Model` resource, the model can be deployed to a `Vertex AI Endpoint` resource.\n",
|
||||
"\n",
|
||||
"### Download the pretrained model\n",
|
||||
"\n",
|
||||
"Download the pretrained TensorFlow estimator model artifacts from the public Cloud Storage, and then upload the model artifacts to your own Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c55fa4c826f7"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_DIR = BUCKET_URI + \"/model\"\n",
|
||||
"\n",
|
||||
"! gsutil cp -r gs://cloud-samples-data/vertex-ai/google-cloud-aiplatform-ci-artifacts/models/penguins/estimator/ {MODEL_DIR}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e8ce91147c93"
|
||||
},
|
||||
"source": [
|
||||
"## Upload the TensorFlow estimator model to a `Vertex AI Model` resource\n",
|
||||
"\n",
|
||||
"Finally, you upload the model artifacts from the TFHub model and serving function into a `Vertex AI Model` resource.\n",
|
||||
"\n",
|
||||
"*Note:* When you upload the model artifacts to a `Vertex AI Model` resource, you specify the corresponding deployment container image."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ad61e1429512"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = aip.Model.upload(\n",
|
||||
" display_name=\"example_\" + TIMESTAMP,\n",
|
||||
" artifact_uri=MODEL_DIR,\n",
|
||||
" serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "628de0914ba1"
|
||||
},
|
||||
"source": [
|
||||
"## Creating an `Endpoint` resource\n",
|
||||
"\n",
|
||||
"You create an `Endpoint` resource using the `Endpoint.create()` method. At a minimum, you specify the display name for the endpoint. Optionally, you can specify the project and location (region); otherwise the settings are inherited by the values you set when you initialized the Vertex AI SDK with the `init()` method.\n",
|
||||
"\n",
|
||||
"In this example, the following parameters are specified:\n",
|
||||
"\n",
|
||||
"- `display_name`: A human readable name for the `Endpoint` resource.\n",
|
||||
"- `project`: Your project ID.\n",
|
||||
"- `location`: Your region.\n",
|
||||
"- `labels`: (optional) User defined metadata for the `Endpoint` in the form of key/value pairs.\n",
|
||||
"\n",
|
||||
"This method returns an `Endpoint` object.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "0ea443f9593b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = aip.Endpoint.create(\n",
|
||||
" display_name=\"example_\" + TIMESTAMP,\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" location=REGION,\n",
|
||||
" labels={\"your_key\": \"your_value\"},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(endpoint)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ca3fa3f6a894"
|
||||
},
|
||||
"source": [
|
||||
"## Deploying `Model` resources to an `Endpoint` resource.\n",
|
||||
"\n",
|
||||
"You can deploy one of more `Vertex AI Model` resource instances to the same endpoint. Each `Vertex AI Model` resource that is deployed will have its own deployment container for the serving binary. \n",
|
||||
"\n",
|
||||
"BLAH\n",
|
||||
"\n",
|
||||
"In the next example, you deploy the `Vertex AI Model` resource to a `Vertex AI Endpoint` resource. The `Vertex AI Model` resource already has defined for it the deployment container image. To deploy, you specify the following additional configuration settings:\n",
|
||||
"\n",
|
||||
"- The machine type.\n",
|
||||
"- The (if any) type and number of GPUs.\n",
|
||||
"- Static, manual or auto-scaling of VM instances.\n",
|
||||
"\n",
|
||||
"In this example, you deploy the model with the minimal amount of specified parameters, as follows:\n",
|
||||
"\n",
|
||||
"- `model`: The `Model` resource.\n",
|
||||
"- `deployed_model_displayed_name`: The human readable name for the deployed model instance.\n",
|
||||
"- `machine_type`: The machine type for each VM instance.\n",
|
||||
"\n",
|
||||
"Do to the requirements to provision the resource, this may take upto a few minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4e93b034a72f"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = endpoint.deploy(\n",
|
||||
" model=model,\n",
|
||||
" deployed_model_display_name=\"example_\" + TIMESTAMP,\n",
|
||||
" machine_type=DEPLOY_COMPUTE,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(endpoint)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "make_test_items:bqml,penguins"
|
||||
},
|
||||
"source": [
|
||||
"#### Make prediction instances\n",
|
||||
"\n",
|
||||
"Next, you prepare a prediction request using a synthetic example. In this example, the model format is a TensorFlow 1.x estimator format. This model format takes a request signature not supported by the HTTP server in the `Vertex AI` prebuilt TensorFlow serving containers.\n",
|
||||
"\n",
|
||||
"For this model format, you use the `raw_predict()` to pass as-is a request that matches directly the serving interfac of the model, with the following request format:\n",
|
||||
"\n",
|
||||
" http_body -> {\n",
|
||||
" 'signature_name' : serving_signature,\n",
|
||||
" 'instances': [ {instance_1}, {instance_2}, ... ]\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" instance -> { 'feature_1': value_1, 'feature_2': value_2, ... }\n",
|
||||
"\n",
|
||||
" serving_signature -> \"predict\"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "make_test_items:bqml,penguins"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"from google.api import httpbody_pb2\n",
|
||||
"from google.cloud import aiplatform_v1\n",
|
||||
"\n",
|
||||
"DATA = {\n",
|
||||
" \"signature_name\": \"predict\",\n",
|
||||
" \"instances\": [\n",
|
||||
" {\n",
|
||||
" \"island\": \"DREAM\",\n",
|
||||
" \"culmen_length_mm\": 36.6,\n",
|
||||
" \"culmen_depth_mm\": 18.4,\n",
|
||||
" \"flipper_length_mm\": 184.0,\n",
|
||||
" \"body_mass_g\": 3475.0,\n",
|
||||
" \"sex\": \"FEMALE\",\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"http_body = httpbody_pb2.HttpBody(\n",
|
||||
" data=json.dumps(DATA).encode(\"utf-8\"),\n",
|
||||
" content_type=\"application/json\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"req = aiplatform_v1.RawPredictRequest(\n",
|
||||
" http_body=http_body, endpoint=endpoint.resource_name\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "endpoint_predict:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"### Make a prediction\n",
|
||||
"\n",
|
||||
"Finally, you make the prediction request using Vertex AI Prediction service."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "endpoint_predict:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)\n",
|
||||
"client_options = {\"api_endpoint\": API_ENDPOINT}\n",
|
||||
"\n",
|
||||
"pred_client = aip.gapic.PredictionServiceClient(client_options=client_options)\n",
|
||||
"\n",
|
||||
"response = pred_client.raw_predict(req)\n",
|
||||
"print(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TpV-iwP9qw9c"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sx_vKniMq9ZX"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_endpoint = True\n",
|
||||
"\n",
|
||||
"if delete_endpoint:\n",
|
||||
" try:\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_model:\n",
|
||||
" try:\n",
|
||||
" model.delete()\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -rf {BUCKET_URI}"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "get_started_with_raw_predict.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB |
File diff suppressed because it is too large
Load Diff
@@ -1,137 +0,0 @@
|
||||
import itertools
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from collections import defaultdict
|
||||
|
||||
import numpy as np
|
||||
import tensorflow as tf
|
||||
|
||||
logging = tf.get_logger()
|
||||
logging.propagate = False
|
||||
logging.setLevel("INFO")
|
||||
|
||||
def benchmark_qps(send_request, requests, qps):
|
||||
logging.info("Running benchmark at {} qps".format(qps))
|
||||
# List appends are thread safe
|
||||
num_requests = len(requests)
|
||||
success = []
|
||||
error = []
|
||||
latency = []
|
||||
|
||||
def _make_call(i):
|
||||
"""Send a request to using specified method and measure observed latency."""
|
||||
start_time = time.time()
|
||||
try:
|
||||
_ = send_request(requests[i])
|
||||
success.append(1)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
error.append(1)
|
||||
|
||||
latency.append(time.time() - start_time)
|
||||
if len(latency) % (qps * 10) == 0:
|
||||
logging.info("received {} responses.".format(len(latency)))
|
||||
|
||||
thread_lst = []
|
||||
miss_rate_percent = []
|
||||
start_time = time.time()
|
||||
previous_worker_start = start_time
|
||||
for i in range(num_requests):
|
||||
thread = threading.Thread(target=_make_call, args=(i,))
|
||||
thread_lst.append(thread)
|
||||
thread.start()
|
||||
if i % (qps * 10) == 0 and i != 0:
|
||||
logging.info("sent {} requests.".format(i))
|
||||
|
||||
# send requests at a constant rate and adjust for the time it took to send previous request
|
||||
pause = 1.0 / qps - (time.time() - previous_worker_start)
|
||||
if pause > 0:
|
||||
time.sleep(pause)
|
||||
else:
|
||||
missed_delay = (
|
||||
100 * ((time.time() - previous_worker_start) - 1.0 / qps) / (1.0 / qps)
|
||||
)
|
||||
miss_rate_percent.append(missed_delay)
|
||||
previous_worker_start = time.time()
|
||||
|
||||
for thread in thread_lst:
|
||||
thread.join()
|
||||
|
||||
acc_time = time.time() - start_time
|
||||
|
||||
avg_miss_rate_percent = 0
|
||||
if len(miss_rate_percent) > 0:
|
||||
avg_miss_rate_percent = np.average(miss_rate_percent)
|
||||
logging.warning(
|
||||
"couldn't keep up at current QPS rate, average miss rate:{:.2f}%".format(
|
||||
avg_miss_rate_percent
|
||||
)
|
||||
)
|
||||
|
||||
logging.info(
|
||||
"num_qps:{} requests/second: {:.2f} #success:{} #error:{} "
|
||||
"latencies: [avg:{:.2f}ms p50:{:.2f}ms p90:{:.2f}ms p99:{:.2f}ms]".format(
|
||||
qps,
|
||||
num_requests / acc_time,
|
||||
sum(success),
|
||||
sum(error),
|
||||
np.average(latency) * 1000,
|
||||
np.percentile(latency, 50) * 1000,
|
||||
np.percentile(latency, 90) * 1000,
|
||||
np.percentile(latency, 99) * 1000,
|
||||
)
|
||||
)
|
||||
return {
|
||||
"reqested_qps": qps,
|
||||
"actual_qps": num_requests / acc_time,
|
||||
"success": sum(success),
|
||||
"error": sum(error),
|
||||
"time": acc_time,
|
||||
"avg_latency": np.average(latency) * 1000,
|
||||
"p50": np.percentile(latency, 50) * 1000,
|
||||
"p90": np.percentile(latency, 90) * 1000,
|
||||
"p99": np.percentile(latency, 99) * 1000,
|
||||
"avg_miss_rate_percent": avg_miss_rate_percent,
|
||||
}
|
||||
|
||||
|
||||
def benchmark(
|
||||
send_request,
|
||||
build_request,
|
||||
request_file_path,
|
||||
qps_list,
|
||||
duration_sec,
|
||||
model_name="default",
|
||||
):
|
||||
requests = []
|
||||
with tf.io.gfile.GFile(request_file_path, "r") as f:
|
||||
for line in f:
|
||||
row_dict = json.loads(line)
|
||||
requests.append(build_request(row_dict, model_name))
|
||||
|
||||
results = []
|
||||
for qps in qps_list:
|
||||
num_requests = max(qps * duration_sec, 10)
|
||||
requests_for_qps = list(
|
||||
itertools.islice(itertools.cycle(requests), num_requests)
|
||||
)
|
||||
logging.info(f"benchmarking at {qps} QPS, sending {num_requests} requests")
|
||||
results.append(benchmark_qps(send_request, requests_for_qps, qps))
|
||||
|
||||
columns = [
|
||||
"reqested_qps",
|
||||
"actual_qps",
|
||||
"avg_latency",
|
||||
"p50",
|
||||
"p90",
|
||||
"p99",
|
||||
"success",
|
||||
"error",
|
||||
]
|
||||
merged_results = defaultdict(list)
|
||||
for result in results:
|
||||
for column in columns:
|
||||
merged_results[column].append(result[column])
|
||||
|
||||
return merged_results
|
||||
-2103
File diff suppressed because it is too large
Load Diff
-2579
File diff suppressed because it is too large
Load Diff
@@ -19,6 +19,4 @@
|
||||
/ml_metadata/vertex-pipelines-ml-metadata.ipynb @sararob
|
||||
/vizier/gapic-vizier-multi-objective-optimization.ipynb @halio-g
|
||||
/feature_store/gapic-feature-store.ipynb @protorganizer @diemtvu
|
||||
/managed_notebooks @GoogleCloudPlatform/notebooks-team
|
||||
/pipelines/google_cloud_pipeline_components_bqml_text.ipynb @inardini
|
||||
/pipelines/google_cloud_pipelines_dataproc_tabular @inardini
|
||||
/managed_notebooks @GoogleCloudPlatform/notebooks-team
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -45,7 +45,6 @@
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_forecasting_batch.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
@@ -174,8 +173,19 @@
|
||||
"else:\n",
|
||||
" USER_FLAG = \"\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
|
||||
"! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_tensorflow"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -409,7 +419,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -420,8 +430,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -441,7 +451,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -461,7 +471,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -506,7 +516,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -788,15 +798,6 @@
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "99b7a9287ba6"
|
||||
},
|
||||
"source": [
|
||||
"`batch_predict` can export predictions either to BigQuery or GCS. The BQ option is commented out below and the predictions will be exported to the BUCKET_URI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -806,15 +807,14 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PREDICTION_DATASET_BQ_PATH = (\n",
|
||||
" \"bq://bigquery-public-data:iowa_liquor_sales_forecasting.2021_sales_predict\"\n",
|
||||
" \"bq://bigquery-public-data:iowa_liquor_sales_forecasting.2020_sales_train\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"batch_prediction_job = model.batch_predict(\n",
|
||||
" job_display_name=f\"iowa_liquor_sales_forecasting_predictions_{TIMESTAMP}\",\n",
|
||||
" bigquery_source=PREDICTION_DATASET_BQ_PATH,\n",
|
||||
" instances_format=\"bigquery\",\n",
|
||||
" # bigquery_destination_prefix=batch_predict_bq_output_uri_prefix,\n",
|
||||
" gcs_destination_prefix=BUCKET_URI,\n",
|
||||
" bigquery_destination_prefix=batch_predict_bq_output_uri_prefix,\n",
|
||||
" predictions_format=\"bigquery\",\n",
|
||||
" sync=False,\n",
|
||||
")\n",
|
||||
@@ -1018,9 +1018,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this to true only if you'd like to delete your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"# Delete dataset\n",
|
||||
"dataset.delete()\n",
|
||||
"\n",
|
||||
@@ -1033,8 +1030,8 @@
|
||||
"# Delete batch prediction job\n",
|
||||
"batch_prediction_job.delete()\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -1,916 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "copyright"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "title"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI SDK: AutoML training tabular regression model for online prediction using BigQuery\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
"<br/><br/><br/>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "overview:automl"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use the Vertex AI SDK for Python to create tabular regression models and do online prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dataset:gsod,lrg"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "objective:automl,training,online_prediction"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you create an AutoML tabular regression model and deploy for online prediction from a Python script using the Vertex AI SDK. You can alternatively create and deploy models using the `gcloud` command-line tool or online using the Cloud Console.\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Create a Vertex `Dataset` resource.\n",
|
||||
"- Train the model.\n",
|
||||
"- View the model evaluation.\n",
|
||||
"- Deploy the `Model` resource to a serving `Endpoint` resource.\n",
|
||||
"- Make a prediction.\n",
|
||||
"- Undeploy the `Model`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "costs"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "setup_local"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"If you are using Colab or Google Cloud Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
|
||||
"\n",
|
||||
"Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
|
||||
"\n",
|
||||
"- The Cloud Storage SDK\n",
|
||||
"- Git\n",
|
||||
"- Python 3\n",
|
||||
"- virtualenv\n",
|
||||
"- Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
|
||||
"\n",
|
||||
"2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
|
||||
"\n",
|
||||
"3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
|
||||
"\n",
|
||||
"4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"6. Open this notebook in the Jupyter Notebook Dashboard.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "install_aip:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the latest version of Vertex AI SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook\n",
|
||||
"if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"else:\n",
|
||||
" USER_FLAG = \"\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "install_storage"
|
||||
},
|
||||
"source": [
|
||||
"Install the latest GA version of *google-cloud-storage* library as well."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "install_storage"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install -U google-cloud-storage $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "restart"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "before_you_begin:nogpu"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### GPU runtime\n",
|
||||
"\n",
|
||||
"This tutorial does not require a GPU runtime.\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
|
||||
"\n",
|
||||
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
|
||||
"\n",
|
||||
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "autoset_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID:\", PROJECT_ID)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_gcloud_project_id"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gcloud config set project $PROJECT_ID"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable, which is used for operations\n",
|
||||
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
|
||||
"\n",
|
||||
"- Americas: `us-central1`\n",
|
||||
"- Europe: `europe-west4`\n",
|
||||
"- Asia Pacific: `asia-east1`\n",
|
||||
"\n",
|
||||
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "timestamp"
|
||||
},
|
||||
"source": [
|
||||
"#### Timestamp\n",
|
||||
"\n",
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "timestamp"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "gcp_authenticate"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"**Click Create service account**.\n",
|
||||
"\n",
|
||||
"In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gcp_authenticate"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "autoset_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "create_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"source": [
|
||||
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "validate_bucket"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "setup_vars"
|
||||
},
|
||||
"source": [
|
||||
"### Set up variables\n",
|
||||
"\n",
|
||||
"Next, set up some variables used throughout the tutorial.\n",
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"## Initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tutorial_start:automl"
|
||||
},
|
||||
"source": [
|
||||
"# Tutorial\n",
|
||||
"\n",
|
||||
"Now you are ready to start creating your own AutoML tabular regression model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "import_file:u_dataset,bq"
|
||||
},
|
||||
"source": [
|
||||
"#### Location of BigQuery training data.\n",
|
||||
"\n",
|
||||
"Now set the variable `IMPORT_FILE` to the location of the data table in BigQuery."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "import_file:gsod,bq,lrg"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"IMPORT_FILE = \"bq://bigquery-public-data.samples.gsod\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "create_dataset:tabular,bq,lrg"
|
||||
},
|
||||
"source": [
|
||||
"### Create the Dataset\n",
|
||||
"\n",
|
||||
"Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
|
||||
"\n",
|
||||
"- `display_name`: The human readable name for the `Dataset` resource.\n",
|
||||
"- `bq_source`: Alternatively, import data items from a BigQuery table into the `Dataset` resource.\n",
|
||||
"\n",
|
||||
"This operation may take several minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "create_dataset:tabular,bq,lrg"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = aiplatform.TabularDataset.create(\n",
|
||||
" display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
|
||||
" bq_source=[IMPORT_FILE],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\"\n",
|
||||
"\n",
|
||||
"print(dataset.resource_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_transformations:gsod"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRANSFORMATIONS = [\n",
|
||||
" {\"auto\": {\"column_name\": \"year\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"month\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"day\"}},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "create_automl_pipeline:tabular,lrg,transformations"
|
||||
},
|
||||
"source": [
|
||||
"### Create and run training pipeline\n",
|
||||
"\n",
|
||||
"To train an AutoML model, you perform two steps: 1) create a training pipeline, and 2) run the pipeline.\n",
|
||||
"\n",
|
||||
"#### Create training pipeline\n",
|
||||
"\n",
|
||||
"An AutoML training pipeline is created with the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `display_name`: The human readable name for the `TrainingJob` resource.\n",
|
||||
"- `optimization_prediction_type`: The type task to train the model for.\n",
|
||||
" - `classification`: A tabuar classification model.\n",
|
||||
" - `regression`: A tabular regression model.\n",
|
||||
"- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
|
||||
"- `optimization_objective`: The optimization objective to minimize or maximize.\n",
|
||||
" - binary classification:\n",
|
||||
" - `minimize-log-loss`\n",
|
||||
" - `maximize-au-roc`\n",
|
||||
" - `maximize-au-prc`\n",
|
||||
" - `maximize-precision-at-recall`\n",
|
||||
" - `maximize-recall-at-precision`\n",
|
||||
" - multi-class classification:\n",
|
||||
" - `minimize-log-loss`\n",
|
||||
" - regression:\n",
|
||||
" - `minimize-rmse`\n",
|
||||
" - `minimize-mae`\n",
|
||||
" - `minimize-rmsle`\n",
|
||||
"\n",
|
||||
"The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "create_automl_pipeline:tabular,lrg,transformations"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job = aiplatform.AutoMLTabularTrainingJob(\n",
|
||||
" display_name=\"gsod_\" + TIMESTAMP,\n",
|
||||
" optimization_prediction_type=\"regression\",\n",
|
||||
" optimization_objective=\"minimize-rmse\",\n",
|
||||
" column_transformations=TRANSFORMATIONS,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(job)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "run_automl_pipeline:tabular"
|
||||
},
|
||||
"source": [
|
||||
"#### Run the training pipeline\n",
|
||||
"\n",
|
||||
"Next, you run the DAG (object 'job') to start the training job by invoking the method `run`, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `dataset`: The `Dataset` resource to train the model.\n",
|
||||
"- `model_display_name`: The human readable name for the trained model.\n",
|
||||
"- `training_fraction_split`: The percentage of the dataset to use for training.\n",
|
||||
"- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
|
||||
"- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
|
||||
"- `target_column`: The name of the column to train as the label.\n",
|
||||
"- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
|
||||
"- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.\n",
|
||||
"\n",
|
||||
"The `run` method when completed returns the `Model` resource.\n",
|
||||
"\n",
|
||||
"The execution of the training pipeline will take upto 8 hours."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "run_automl_pipeline:tabular"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = job.run(\n",
|
||||
" dataset=dataset,\n",
|
||||
" model_display_name=\"gsod_\" + TIMESTAMP,\n",
|
||||
" training_fraction_split=0.6,\n",
|
||||
" validation_fraction_split=0.2,\n",
|
||||
" test_fraction_split=0.2,\n",
|
||||
" budget_milli_node_hours=8000,\n",
|
||||
" disable_early_stopping=False,\n",
|
||||
" target_column=label_column,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "evaluate_the_model:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"## Review model evaluation scores\n",
|
||||
"After your model has finished training, you can review the evaluation scores for it.\n",
|
||||
"\n",
|
||||
"First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "evaluate_the_model:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get model resource ID\n",
|
||||
"models = aiplatform.Model.list(filter=\"display_name=gsod_\" + TIMESTAMP)\n",
|
||||
"\n",
|
||||
"# Get a reference to the Model Service client\n",
|
||||
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
|
||||
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
|
||||
" client_options=client_options\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model_evaluations = model_service_client.list_model_evaluations(\n",
|
||||
" parent=models[0].resource_name\n",
|
||||
")\n",
|
||||
"model_evaluation = list(model_evaluations)[0]\n",
|
||||
"print(model_evaluation)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "deploy_model:mbsdk,dedicated"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy the model\n",
|
||||
"\n",
|
||||
"Next, deploy your model for online prediction. To deploy the model, you invoke the `deploy` method, with the following parameters:\n",
|
||||
"\n",
|
||||
"- `machine_type`: The type of compute machine."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "deploy_model:mbsdk,dedicated"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = model.deploy(machine_type=\"n1-standard-4\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "make_prediction"
|
||||
},
|
||||
"source": [
|
||||
"## Send a online prediction request\n",
|
||||
"\n",
|
||||
"Send a online prediction to your deployed model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "make_test_item:automl,online_prediction"
|
||||
},
|
||||
"source": [
|
||||
"### Make test item\n",
|
||||
"\n",
|
||||
"You will use synthetic data as a test data item. Don't be concerned that we are using synthetic data -- we just want to demonstrate how to make a prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "make_test_item:automl,tabular,gsod"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"INSTANCE = {\"year\": \"1932\", \"month\": \"11\", \"day\": \"6\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "predict_request:mbsdk,lrg"
|
||||
},
|
||||
"source": [
|
||||
"### Make the prediction\n",
|
||||
"\n",
|
||||
"Now that your `Model` resource is deployed to an `Endpoint` resource, you can do online predictions by sending prediction requests to the `Endpoint` resource.\n",
|
||||
"\n",
|
||||
"#### Request\n",
|
||||
"\n",
|
||||
"The format of each instance is:\n",
|
||||
"\n",
|
||||
" [feature_list]\n",
|
||||
"\n",
|
||||
"Since the predict() method can take multiple items (instances), send your single test item as a list of one test item.\n",
|
||||
"\n",
|
||||
"#### Response\n",
|
||||
"\n",
|
||||
"The response from the predict() call is a Python dictionary with the following entries:\n",
|
||||
"\n",
|
||||
"- `ids`: The internal assigned unique identifiers for each prediction request.\n",
|
||||
"- `value`: The predicted value for each prediction.\n",
|
||||
"- `deployed_model_id`: The Vertex AI identifier for the deployed `Model` resource which did the predictions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "predict_request:mbsdk,lrg"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"instances_list = [INSTANCE]\n",
|
||||
"\n",
|
||||
"prediction = endpoint.predict(instances_list)\n",
|
||||
"print(prediction)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "undeploy_model:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"## Undeploy the model\n",
|
||||
"\n",
|
||||
"When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "undeploy_model:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.undeploy_all()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"source": [
|
||||
"# Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- AutoML Training Job"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "cleanup:mbsdk"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_all = True\n",
|
||||
"\n",
|
||||
"if delete_all:\n",
|
||||
" # Delete the dataset using the Vertex dataset object\n",
|
||||
" dataset.delete()\n",
|
||||
"\n",
|
||||
" # Delete the model using the Vertex model object\n",
|
||||
" model.delete()\n",
|
||||
"\n",
|
||||
" # Delete the endpoint using the Vertex endpoint object\n",
|
||||
" endpoint.delete()\n",
|
||||
"\n",
|
||||
" # Delete the AutoML trainig job\n",
|
||||
" job.delete()\n",
|
||||
"\n",
|
||||
" if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "sdk_automl_tabular_regression_online_bq.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2020 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -43,12 +43,6 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
@@ -61,7 +55,7 @@
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use the Vertex AI SDK for Python to train and deploy a custom tabular classification model for online prediction."
|
||||
"This tutorial demonstrates how to use the Vertex SDK for Python to train and deploy a custom tabular classification model for online prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -122,7 +116,7 @@
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the latest version of Vertex AI SDK for Python."
|
||||
"Install the latest version of Vertex SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -431,8 +425,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://[your-bucket-name]\"\n",
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}"
|
||||
"BUCKET_NAME = \"\" # @param {type:\"string\"}\n",
|
||||
"REGION = \"us-central1\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -443,11 +437,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP\n",
|
||||
"\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -467,7 +458,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI\n"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -487,7 +478,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -507,9 +498,9 @@
|
||||
"id": "import_aip"
|
||||
},
|
||||
"source": [
|
||||
"### Import Vertex AI SDK for Python\n",
|
||||
"### Import Vertex SDK for Python\n",
|
||||
"\n",
|
||||
"Import the Vertex AI SDK for Python into your Python environment and initialize it."
|
||||
"Import the Vertex SDK for Python into your Python environment and initialize it."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -520,15 +511,13 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"from google.cloud import aiplatform, bigquery\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from google.cloud.aiplatform import gapic as aip\n",
|
||||
"\n",
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -586,8 +575,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRAIN_VERSION = \"tf-gpu.2-8\"\n",
|
||||
"DEPLOY_VERSION = \"tf2-gpu.2-8\"\n",
|
||||
"TRAIN_VERSION = \"tf-gpu.2-4\"\n",
|
||||
"DEPLOY_VERSION = \"tf2-gpu.2-4\"\n",
|
||||
"\n",
|
||||
"TRAIN_IMAGE = \"us-docker.pkg.dev/vertex-ai/training/{}:latest\".format(TRAIN_VERSION)\n",
|
||||
"DEPLOY_IMAGE = \"us-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(DEPLOY_VERSION)\n",
|
||||
@@ -676,7 +665,11 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"# Calculate mean and std across all rows\n",
|
||||
"from google.cloud import bigquery\n",
|
||||
"\n",
|
||||
"NA_VALUES = [\"NA\", \".\"]\n",
|
||||
"\n",
|
||||
@@ -731,7 +724,7 @@
|
||||
" json.dump(mean_and_std, outfile)\n",
|
||||
"\n",
|
||||
"# Save to the staging bucket\n",
|
||||
"! gsutil cp {MEAN_AND_STD_JSON_FILE} {BUCKET_URI}"
|
||||
"! gsutil cp {MEAN_AND_STD_JSON_FILE} {BUCKET_NAME}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -814,7 +807,7 @@
|
||||
" \"--epochs=\" + str(EPOCHS),\n",
|
||||
" \"--batch_size=\" + str(BATCH_SIZE),\n",
|
||||
" \"--distribute=\" + TRAIN_STRATEGY,\n",
|
||||
" \"--mean_and_std_json_file=\" + f\"{BUCKET_URI}/{MEAN_AND_STD_JSON_FILE}\",\n",
|
||||
" \"--mean_and_std_json_file=\" + f\"{BUCKET_NAME}/{MEAN_AND_STD_JSON_FILE}\",\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
@@ -860,9 +853,9 @@
|
||||
"from google.cloud import storage\n",
|
||||
"\n",
|
||||
"# Read environmental variables\n",
|
||||
"training_data_uri = os.getenv(\"AIP_TRAINING_DATA_URI\")\n",
|
||||
"validation_data_uri = os.getenv(\"AIP_VALIDATION_DATA_URI\")\n",
|
||||
"test_data_uri = os.getenv(\"AIP_TEST_DATA_URI\")\n",
|
||||
"training_data_uri = os.environ[\"AIP_TRAINING_DATA_URI\"]\n",
|
||||
"validation_data_uri = os.environ[\"AIP_VALIDATION_DATA_URI\"]\n",
|
||||
"test_data_uri = os.environ[\"AIP_TEST_DATA_URI\"]\n",
|
||||
"\n",
|
||||
"# Read args\n",
|
||||
"parser = argparse.ArgumentParser()\n",
|
||||
@@ -1135,7 +1128,7 @@
|
||||
"# Train the model\n",
|
||||
"model.fit(dataset_train, epochs=args.epochs, validation_data=dataset_validation)\n",
|
||||
"\n",
|
||||
"tf.saved_model.save(model, os.getenv(\"AIP_MODEL_DIR\"))\n",
|
||||
"tf.saved_model.save(model, os.environ[\"AIP_MODEL_DIR\"])\n",
|
||||
"\n",
|
||||
"df_test.head()"
|
||||
]
|
||||
@@ -1182,7 +1175,7 @@
|
||||
" display_name=JOB_NAME,\n",
|
||||
" script_path=\"task.py\",\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
" requirements=[\"google-cloud-bigquery>=2.20.0\", \"db-dtypes\"],\n",
|
||||
" requirements=[\"google-cloud-bigquery>=2.20.0\"],\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
@@ -1508,6 +1501,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_training_job = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_endpoint = True\n",
|
||||
"\n",
|
||||
"# Warning: Setting this to true will delete everything in your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
@@ -1520,8 +1517,8 @@
|
||||
"# Delete the endpoint\n",
|
||||
"endpoint.delete()\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil -m rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+2421
File diff suppressed because it is too large
Load Diff
+2402
File diff suppressed because it is too large
Load Diff
+2309
File diff suppressed because it is too large
Load Diff
+2385
File diff suppressed because it is too large
Load Diff
@@ -171,7 +171,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
|
||||
"! pip install {USER_FLAG} --upgrade git+https://github.com/googleapis/python-aiplatform.git@v1.6.0"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -266,7 +266,7 @@
|
||||
"\n",
|
||||
"# Get your Google Cloud project ID from gcloud\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID: \", PROJECT_ID)"
|
||||
]
|
||||
@@ -543,22 +543,29 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"FEATURESTORE_ID = \"movie_prediction\"\n",
|
||||
"try:\n",
|
||||
" create_lro = admin_client.create_featurestore(\n",
|
||||
" featurestore_service_pb2.CreateFeaturestoreRequest(\n",
|
||||
" parent=BASE_RESOURCE_PATH,\n",
|
||||
" featurestore_id=FEATURESTORE_ID,\n",
|
||||
" featurestore=featurestore_pb2.Featurestore(\n",
|
||||
" online_serving_config=featurestore_pb2.Featurestore.OnlineServingConfig(\n",
|
||||
" fixed_node_count=1\n",
|
||||
" ),\n",
|
||||
"create_lro = admin_client.create_featurestore(\n",
|
||||
" featurestore_service_pb2.CreateFeaturestoreRequest(\n",
|
||||
" parent=BASE_RESOURCE_PATH,\n",
|
||||
" featurestore_id=FEATURESTORE_ID,\n",
|
||||
" featurestore=featurestore_pb2.Featurestore(\n",
|
||||
" online_serving_config=featurestore_pb2.Featurestore.OnlineServingConfig(\n",
|
||||
" fixed_node_count=1\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" # Wait for LRO to finish and get the LRO result.\n",
|
||||
" print(create_lro.result())\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "57V8eVcB5VFZ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Wait for LRO to finish and get the LRO result.\n",
|
||||
"print(create_lro.result())"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -583,41 +590,6 @@
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "018ab19d934f"
|
||||
},
|
||||
"source": [
|
||||
"Auto scaling is available in v1beta1 since v1.11. Below is the example for the `CreateFeaturestoreRequest` with auto-scaling, use it with `aiplatform_v1beta1.FeaturestoreServiceClient` to create Featurestore:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "aea39718b5d3"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud.aiplatform_v1beta1.types import \\\n",
|
||||
" featurestore as v1beta1_featurestore_pb2\n",
|
||||
"from google.cloud.aiplatform_v1beta1.types import \\\n",
|
||||
" featurestore_service as v1beta1_featurestore_service_pb2\n",
|
||||
"\n",
|
||||
"create_featurestore_request = v1beta1_featurestore_service_pb2.CreateFeaturestoreRequest(\n",
|
||||
" parent=BASE_RESOURCE_PATH,\n",
|
||||
" featurestore_id=FEATURESTORE_ID,\n",
|
||||
" featurestore=v1beta1_featurestore_pb2.Featurestore(\n",
|
||||
" online_serving_config=v1beta1_featurestore_pb2.Featurestore.OnlineServingConfig(\n",
|
||||
" scaling=v1beta1_featurestore_pb2.Featurestore.OnlineServingConfig.Scaling(\n",
|
||||
" min_node_count=1, max_node_count=5\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -635,20 +607,18 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"try:\n",
|
||||
" users_entity_type_lro = admin_client.create_entity_type(\n",
|
||||
" featurestore_service_pb2.CreateEntityTypeRequest(\n",
|
||||
" parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
|
||||
" entity_type_id=\"users\",\n",
|
||||
" entity_type=entity_type_pb2.EntityType(\n",
|
||||
" description=\"Users entity\",\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
"users_entity_type_lro = admin_client.create_entity_type(\n",
|
||||
" featurestore_service_pb2.CreateEntityTypeRequest(\n",
|
||||
" parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
|
||||
" entity_type_id=\"users\",\n",
|
||||
" entity_type=entity_type_pb2.EntityType(\n",
|
||||
" description=\"Users entity\",\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" # Similarly, wait for EntityType creation operation.\n",
|
||||
" print(users_entity_type_lro.result())\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
")\n",
|
||||
"\n",
|
||||
"# Similarly, wait for EntityType creation operation.\n",
|
||||
"print(users_entity_type_lro.result())"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -660,19 +630,16 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create movies entity type without a monitoring configuration.\n",
|
||||
"try:\n",
|
||||
" movies_entity_type_lro = admin_client.create_entity_type(\n",
|
||||
" featurestore_service_pb2.CreateEntityTypeRequest(\n",
|
||||
" parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
|
||||
" entity_type_id=\"movies\",\n",
|
||||
" entity_type=entity_type_pb2.EntityType(description=\"Movies entity\"),\n",
|
||||
" )\n",
|
||||
"movies_entity_type_lro = admin_client.create_entity_type(\n",
|
||||
" featurestore_service_pb2.CreateEntityTypeRequest(\n",
|
||||
" parent=admin_client.featurestore_path(PROJECT_ID, REGION, FEATURESTORE_ID),\n",
|
||||
" entity_type_id=\"movies\",\n",
|
||||
" entity_type=entity_type_pb2.EntityType(description=\"Movies entity\"),\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
" # Similarly, wait for EntityType creation operation.\n",
|
||||
" print(movies_entity_type_lro.result())\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
"# Similarly, wait for EntityType creation operation.\n",
|
||||
"print(movies_entity_type_lro.result())"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -744,37 +711,32 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create features for the 'users' entity.\n",
|
||||
"try:\n",
|
||||
" admin_client.batch_create_features(\n",
|
||||
" parent=admin_client.entity_type_path(\n",
|
||||
" PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n",
|
||||
"admin_client.batch_create_features(\n",
|
||||
" parent=admin_client.entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"),\n",
|
||||
" requests=[\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.INT64,\n",
|
||||
" description=\"User age\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"age\",\n",
|
||||
" ),\n",
|
||||
" requests=[\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.INT64,\n",
|
||||
" description=\"User age\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"age\",\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"User gender\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"User gender\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"gender\",\n",
|
||||
" feature_id=\"gender\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING_ARRAY,\n",
|
||||
" description=\"An array of genres that this user liked\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING_ARRAY,\n",
|
||||
" description=\"An array of genres that this user liked\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"liked_genres\",\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" ).result()\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
" feature_id=\"liked_genres\",\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
").result()"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -786,37 +748,32 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create features for movies type.\n",
|
||||
"try:\n",
|
||||
" admin_client.batch_create_features(\n",
|
||||
" parent=admin_client.entity_type_path(\n",
|
||||
" PROJECT_ID, REGION, FEATURESTORE_ID, \"movies\"\n",
|
||||
"admin_client.batch_create_features(\n",
|
||||
" parent=admin_client.entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, \"movies\"),\n",
|
||||
" requests=[\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"The title of the movie\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"title\",\n",
|
||||
" ),\n",
|
||||
" requests=[\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"The title of the movie\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"title\",\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"The genres of the movie\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.STRING,\n",
|
||||
" description=\"The genres of the movie\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"genres\",\n",
|
||||
" feature_id=\"genres\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.DOUBLE,\n",
|
||||
" description=\"The average rating for the movie, range is [1.0-5.0]\",\n",
|
||||
" ),\n",
|
||||
" featurestore_service_pb2.CreateFeatureRequest(\n",
|
||||
" feature=feature_pb2.Feature(\n",
|
||||
" value_type=feature_pb2.Feature.ValueType.DOUBLE,\n",
|
||||
" description=\"The average rating for the movie, range is [1.0-5.0]\",\n",
|
||||
" ),\n",
|
||||
" feature_id=\"average_rating\",\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" ).result()\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
" feature_id=\"average_rating\",\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
").result()"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -268,7 +268,7 @@
|
||||
"\n",
|
||||
"# Get your Google Cloud project ID from gcloud\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" shell_output=! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID: \", PROJECT_ID)"
|
||||
]
|
||||
@@ -497,6 +497,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"import re\n",
|
||||
"import time\n",
|
||||
|
||||
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
+2
-2
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! apt-get update && apt-get install -y python3-opencv-headless\n",
|
||||
" ! apt-get install -y libgl1-mesa-dev\n",
|
||||
" ! pip3 install --upgrade opencv-python-headless $USER_FLAG"
|
||||
@@ -183,7 +183,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
+2
-2
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! apt-get update && apt-get install -y python3-opencv-headless\n",
|
||||
" ! apt-get install -y libgl1-mesa-dev\n",
|
||||
" ! pip3 install --upgrade opencv-python-headless $USER_FLAG"
|
||||
@@ -183,7 +183,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
+1
-1
@@ -169,7 +169,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade tensorflow $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -4,7 +4,6 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
@@ -31,24 +30,24 @@
|
||||
"id": "fsv4jGuU89rX"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI Model Monitoring with Explainable AI Feature Attributions\n",
|
||||
"# Vertex AI Model Monitoring\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmaster%2Fnotebooks%2Fcommunity%2Fmodel_monitoring%2Fmodel_monitoring_feature_attribs.ipynb\">\n",
|
||||
" <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Cloud Notebook\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/model_monitoring/model_monitoring_feature_attribs.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/model_monitoring/model_monitoring_feature_attribs.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/raw/master/notebooks/official/model_monitoring/model_monitoring.ipynb\"\n",
|
||||
" <img src=\"https://cloud.google.com/images/products/ai/ai-solutions-icon.svg\" alt=\"Vertex AI Workbench notebook\"> Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
@@ -86,9 +85,7 @@
|
||||
"* How well do recent service requests match the training data used to build your model? This is called **training-serving skew**.\n",
|
||||
"* How significantly are service requests evolving over time? This is called **drift detection**.\n",
|
||||
"\n",
|
||||
"[Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview) adds another facet to model monitoring, which we call feature attribution monitoring. Explainable AI enables you to understand the relative contribution of each feature to a resulting prediction. In essence, it assesses the magnitude of each feature's influence.\n",
|
||||
"\n",
|
||||
"If production traffic differs from training data, or varies substantially over time, **either in terms of model predictions or feature attributions**, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
|
||||
"If production traffic differs from training data, or varies substantially over time, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -191,7 +188,6 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import pprint as pp\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"import IPython\n",
|
||||
@@ -199,12 +195,10 @@
|
||||
"assert sys.version_info.major == 3, \"This notebook requires Python 3.\"\n",
|
||||
"\n",
|
||||
"# Install Python package dependencies.\n",
|
||||
"print(\"Installing TensorFlow and TensorFlow Data Validation (TFDV)\")\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade tensorflow tensorflow_data_validation[visualization]\n",
|
||||
"! rm -f /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so\n",
|
||||
"print(\"Installing TensorFlow 2.4.1 and TensorFlow Data Validation (TFDV)\")\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade tensorflow==2.4.1 tensorflow_data_validation[visualization]\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade google-api-python-client google-auth-oauthlib google-auth-httplib2 oauth2client requests\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-aiplatform\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade explainable_ai_sdk\n",
|
||||
"! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-storage==1.32.0\n",
|
||||
"\n",
|
||||
"# Automatically restart kernel after installing new packages.\n",
|
||||
@@ -223,13 +217,12 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Import required packages.\n",
|
||||
"import os\n",
|
||||
"import random\n",
|
||||
"import sys\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"# Import required packages.\n",
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
@@ -272,8 +265,7 @@
|
||||
" !gcloud --quiet components install beta\n",
|
||||
" !gcloud --quiet components update\n",
|
||||
"!gcloud config set project $PROJECT_ID\n",
|
||||
"!gcloud config set ai/region $REGION\n",
|
||||
"os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID"
|
||||
"!gcloud config set ai/region $REGION"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -318,7 +310,7 @@
|
||||
"id": "btZeLzqQ7pXc"
|
||||
},
|
||||
"source": [
|
||||
"### Define some helper functions and data structures\n",
|
||||
"### Define some helper functions\n",
|
||||
"\n",
|
||||
"Run the following cell to define some utility functions used throughout this notebook. Although these functions are not critical to understand the main concepts, feel free to expand the cell if you're curious or want to dive deeper into how some of your API requests are made."
|
||||
]
|
||||
@@ -327,7 +319,6 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "yhDFSB2YDvfT"
|
||||
},
|
||||
"outputs": [],
|
||||
@@ -336,21 +327,21 @@
|
||||
"import copy\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from explainable_ai_sdk.metadata.tf.v2 import SavedModelMetadataBuilder\n",
|
||||
"from google.cloud.aiplatform_v1.services.endpoint_service import \\\n",
|
||||
"from google.cloud.aiplatform_v1beta1.services.endpoint_service import \\\n",
|
||||
" EndpointServiceClient\n",
|
||||
"from google.cloud.aiplatform_v1.services.job_service import JobServiceClient\n",
|
||||
"from google.cloud.aiplatform_v1.services.prediction_service import \\\n",
|
||||
"from google.cloud.aiplatform_v1beta1.services.job_service import \\\n",
|
||||
" JobServiceClient\n",
|
||||
"from google.cloud.aiplatform_v1beta1.services.prediction_service import \\\n",
|
||||
" PredictionServiceClient\n",
|
||||
"from google.cloud.aiplatform_v1.types.io import BigQuerySource\n",
|
||||
"from google.cloud.aiplatform_v1.types.model_deployment_monitoring_job import (\n",
|
||||
"from google.cloud.aiplatform_v1beta1.types.io import BigQuerySource\n",
|
||||
"from google.cloud.aiplatform_v1beta1.types.model_deployment_monitoring_job import (\n",
|
||||
" ModelDeploymentMonitoringJob, ModelDeploymentMonitoringObjectiveConfig,\n",
|
||||
" ModelDeploymentMonitoringScheduleConfig)\n",
|
||||
"from google.cloud.aiplatform_v1.types.model_monitoring import (\n",
|
||||
"from google.cloud.aiplatform_v1beta1.types.model_monitoring import (\n",
|
||||
" ModelMonitoringAlertConfig, ModelMonitoringObjectiveConfig,\n",
|
||||
" SamplingStrategy, ThresholdConfig)\n",
|
||||
"from google.cloud.aiplatform_v1.types.prediction_service import (\n",
|
||||
" ExplainRequest, PredictRequest)\n",
|
||||
"from google.cloud.aiplatform_v1beta1.types.prediction_service import \\\n",
|
||||
" PredictRequest\n",
|
||||
"from google.protobuf import json_format\n",
|
||||
"from google.protobuf.duration_pb2 import Duration\n",
|
||||
"from google.protobuf.struct_pb2 import Value\n",
|
||||
@@ -434,27 +425,15 @@
|
||||
" return objective_configs\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def send_predict_request(endpoint, input, type=\"predict\"):\n",
|
||||
"def send_predict_request(endpoint, input):\n",
|
||||
" client_options = {\"api_endpoint\": PREDICT_API_ENDPOINT}\n",
|
||||
" client = PredictionServiceClient(client_options=client_options)\n",
|
||||
" if type == \"predict\":\n",
|
||||
" obj = PredictRequest\n",
|
||||
" method = client.predict\n",
|
||||
" elif type == \"explain\":\n",
|
||||
" obj = ExplainRequest\n",
|
||||
" method = client.explain\n",
|
||||
" else:\n",
|
||||
" raise Exception(\"unsupported request type:\" + type)\n",
|
||||
" params = {}\n",
|
||||
" params = json_format.ParseDict(params, Value())\n",
|
||||
" request = obj(endpoint=endpoint, parameters=params)\n",
|
||||
" request = PredictRequest(endpoint=endpoint, parameters=params)\n",
|
||||
" inputs = [json_format.ParseDict(input, Value())]\n",
|
||||
" request.instances.extend(inputs)\n",
|
||||
" response = None\n",
|
||||
" try:\n",
|
||||
" response = method(request)\n",
|
||||
" except Exception as ex:\n",
|
||||
" print(ex)\n",
|
||||
" response = client.predict(request)\n",
|
||||
" return response\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -566,34 +545,6 @@
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1mhT_d_Bi-Kf"
|
||||
},
|
||||
"source": [
|
||||
"### Generate model metadata for explainable AI\n",
|
||||
"\n",
|
||||
"Run the following cell to extract metadata from the exported model, which is needed for generating the prediction explanations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "d1a984c57356"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"builder = SavedModelMetadataBuilder(\n",
|
||||
" \"gs://mco-mm/churn\", outputs_to_explain=[\"churned_probs\"]\n",
|
||||
")\n",
|
||||
"builder.save_metadata(\".\")\n",
|
||||
"md = builder.get_metadata()\n",
|
||||
"del md[\"tags\"]\n",
|
||||
"del md[\"framework\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -611,67 +562,30 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "50c05f40f1dc"
|
||||
"id": "Se5DxJoK3AMz"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"MODEL_NAME = \"churn\"\n",
|
||||
"IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-5:latest\"\n",
|
||||
"ENDPOINT = \"us-central1-aiplatform.googleapis.com\"\n",
|
||||
"churn_model_path = \"gs://mco-mm/churn\"\n",
|
||||
"\n",
|
||||
"request_data = {\n",
|
||||
" \"model\": {\n",
|
||||
" \"displayName\": \"churn\",\n",
|
||||
" \"artifactUri\": churn_model_path,\n",
|
||||
" \"containerSpec\": {\"imageUri\": IMAGE},\n",
|
||||
" \"explanationSpec\": {\n",
|
||||
" \"parameters\": {\"sampledShapleyAttribution\": {\"pathCount\": 5}},\n",
|
||||
" \"metadata\": md,\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"with open(\"request_data.json\", \"w\") as outfile:\n",
|
||||
" json.dump(request_data, outfile)\n",
|
||||
"output = !curl -X POST \\\n",
|
||||
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
|
||||
" -H \"Content-Type: application/json\" \\\n",
|
||||
" https://{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{REGION}/models:upload \\\n",
|
||||
" -d @request_data.json 2>/dev/null\n",
|
||||
"# print(output)\n",
|
||||
"MODEL_ID = output[1].split()[1].split(\"/\")[5]\n",
|
||||
"print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")\n",
|
||||
"# If auto-testing this notebook, wait for model registration\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" time.sleep(300)"
|
||||
"IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-4:latest\"\n",
|
||||
"ARTIFACT = \"gs://mco-mm/churn\"\n",
|
||||
"output = !gcloud --quiet beta ai models upload --container-image-uri=$IMAGE --artifact-uri=$ARTIFACT --display-name=$MODEL_NAME --format=\"value(model)\"\n",
|
||||
"print(\"model output: \", output)\n",
|
||||
"MODEL_ID = output[1].split(\"/\")[-1]\n",
|
||||
"print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e2030b028cef"
|
||||
},
|
||||
"source": [
|
||||
"This request will return immediately but it spawns an asynchronous task that takes several minutes. Periodically check the Vertex Models page on the Cloud Console and don't continue with this lab until you see your newly created model there. It should like something like this:\n",
|
||||
"<br>\n",
|
||||
"<br>\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm0.png\" />\n",
|
||||
"<br>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d7cbb0fb73cc"
|
||||
"id": "L1HzDTh-JRsr"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy your endpoint\n",
|
||||
"\n",
|
||||
"Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet. Your endpoint is also serverless, in the sense that Google ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
|
||||
"\n",
|
||||
"Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete.**"
|
||||
"Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete. If you've already deployed a model to an endpoint, you can reuse your endpoint by running the cell after the next one.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -684,11 +598,28 @@
|
||||
"source": [
|
||||
"ENDPOINT_NAME = \"churn\"\n",
|
||||
"output = !gcloud --quiet beta ai endpoints create --display-name=$ENDPOINT_NAME --format=\"value(name)\"\n",
|
||||
"# print(\"endpoint output: \", output)\n",
|
||||
"print(\"endpoint output: \", output)\n",
|
||||
"ENDPOINT = output[-1]\n",
|
||||
"ENDPOINT_ID = ENDPOINT.split(\"/\")[-1]\n",
|
||||
"output = !gcloud --quiet beta ai endpoints deploy-model $ENDPOINT_ID --display-name=$ENDPOINT_NAME --model=$MODEL_ID --traffic-split=\"0=100\"\n",
|
||||
"print(f\"Model deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}.\")"
|
||||
"DEPLOYED_MODEL_ID = output[1].split()[-1][:-1]\n",
|
||||
"print(\n",
|
||||
" f\"Model {MODEL_NAME}/{MODEL_ID}/{DEPLOYED_MODEL_ID} deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}/{ENDPOINT}.\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "QzZS5piBeBFI"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# @title Run this cell only if you want to reuse an existing endpoint.\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" ENDPOINT_ID = \"\" # @param {type:\"string\"}\n",
|
||||
" ENDPOINT = f\"projects/mco-mm/locations/us-central1/endpoints/{ENDPOINT_ID}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -699,7 +630,7 @@
|
||||
"source": [
|
||||
"## Run a prediction test\n",
|
||||
"\n",
|
||||
"Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON, along with a pie chart summarizing the results.\n",
|
||||
"Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON.\n",
|
||||
"\n",
|
||||
"**Try this now by running the next cell and examine the results.**"
|
||||
]
|
||||
@@ -712,26 +643,23 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# print(ENDPOINT)\n",
|
||||
"# pp.pprint(DEFAULT_INPUT)\n",
|
||||
"import pprint as pp\n",
|
||||
"\n",
|
||||
"print(ENDPOINT)\n",
|
||||
"print(\"request:\")\n",
|
||||
"pp.pprint(DEFAULT_INPUT)\n",
|
||||
"try:\n",
|
||||
" resp = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
|
||||
" for i in resp.predictions:\n",
|
||||
" vals = i[\"churned_values\"]\n",
|
||||
" probs = i[\"churned_probs\"]\n",
|
||||
" for i in range(len(vals)):\n",
|
||||
" print(vals[i], probs[i])\n",
|
||||
" plt.pie(probs, labels=vals)\n",
|
||||
" plt.show()\n",
|
||||
" print(\"response\")\n",
|
||||
" pp.pprint(resp)\n",
|
||||
"except Exception as ex:\n",
|
||||
" print(\"prediction request failed\", ex)"
|
||||
"except Exception:\n",
|
||||
" print(\"prediction request failed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a1eb4131bb5e"
|
||||
"id": "lN6ADMJqeWtR"
|
||||
},
|
||||
"source": [
|
||||
"Taking a closer look at the results, we see the following elements:\n",
|
||||
@@ -743,49 +671,6 @@
|
||||
"This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "be7830c55cdc"
|
||||
},
|
||||
"source": [
|
||||
"## Run an explanation test\n",
|
||||
"\n",
|
||||
"We can also run a test of explainable AI on this endpoint. Run the next cell to send a test explanation request. If everything works as expected, you should receive a response encoding the feature importance of this prediction in a text representation called JSON, along with a bar chart summarizing the results.\n",
|
||||
"\n",
|
||||
"**Try this now by running the next cell and examine the results.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "dcbd4e755931"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# print(ENDPOINT)\n",
|
||||
"# pp.pprint(DEFAULT_INPUT)\n",
|
||||
"try:\n",
|
||||
" features = []\n",
|
||||
" scores = []\n",
|
||||
" resp = send_predict_request(ENDPOINT, DEFAULT_INPUT, type=\"explain\")\n",
|
||||
" for i in resp.explanations:\n",
|
||||
" for j in i.attributions:\n",
|
||||
" for k in j.feature_attributions:\n",
|
||||
" features.append(k)\n",
|
||||
" scores.append(j.feature_attributions[k])\n",
|
||||
" features = [x for _, x in sorted(zip(scores, features))]\n",
|
||||
" scores = sorted(scores)\n",
|
||||
" fig, ax = plt.subplots()\n",
|
||||
" fig.set_size_inches(9, 9)\n",
|
||||
" ax.barh(features, scores)\n",
|
||||
" fig.show()\n",
|
||||
" # pp.pprint(resp)\n",
|
||||
"except Exception as ex:\n",
|
||||
" print(\"explanation request failed\", ex)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -808,12 +693,10 @@
|
||||
"### Configure the following fields:\n",
|
||||
"\n",
|
||||
"1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n",
|
||||
"1. Monitor interval - time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds)\n",
|
||||
"1. Target field - prediction target column name in training dataset\n",
|
||||
"1. Skew detection threshold - skew threshold for each feature you want to monitor\n",
|
||||
"1. Prediction drift threshold - drift threshold for each feature you want to monitor\n",
|
||||
"1. Attribution Skew detection threshold - feature importance skew threshold\n",
|
||||
"1. Attribution Prediction drift threshold - feature importance drift threshold"
|
||||
"1. Monitor interval - the time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds).\n",
|
||||
"1. Target field - the prediction target column name in training dataset.\n",
|
||||
"1. Skew detection threshold - the skew threshold for each feature you want to monitor.\n",
|
||||
"1. Prediction drift threshold - the drift threshold for each feature you want to monitor."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -839,20 +722,10 @@
|
||||
"TARGET = \"churned\"\n",
|
||||
"\n",
|
||||
"# Skew and drift thresholds.\n",
|
||||
"SKEW_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\" # @param {type:\"string\"}\n",
|
||||
"SKEW_CUSTOM_THRESHOLDS = \"cnt_level_start_quickplay:.01\" # @param {type:\"string\"}\n",
|
||||
"DRIFT_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\" # @param {type:\"string\"}\n",
|
||||
"DRIFT_CUSTOM_THRESHOLDS = \"cnt_level_start_quickplay:.01\" # @param {type:\"string\"}\n",
|
||||
"ATTRIB_SKEW_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\" # @param {type:\"string\"}\n",
|
||||
"ATTRIB_SKEW_CUSTOM_THRESHOLDS = (\n",
|
||||
" \"cnt_level_start_quickplay:.01\" # @param {type:\"string\"}\n",
|
||||
")\n",
|
||||
"ATTRIB_DRIFT_DEFAULT_THRESHOLDS = (\n",
|
||||
" \"country,cnt_user_engagement\" # @param {type:\"string\"}\n",
|
||||
")\n",
|
||||
"ATTRIB_DRIFT_CUSTOM_THRESHOLDS = (\n",
|
||||
" \"cnt_level_start_quickplay:.01\" # @param {type:\"string\"}\n",
|
||||
")"
|
||||
"SKEW_DEFAULT_THRESHOLDS = \"country,language\" # @param {type:\"string\"}\n",
|
||||
"SKEW_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" # @param {type:\"string\"}\n",
|
||||
"DRIFT_DEFAULT_THRESHOLDS = \"country,language\" # @param {type:\"string\"}\n",
|
||||
"DRIFT_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -876,32 +749,18 @@
|
||||
"source": [
|
||||
"skew_thresholds = get_thresholds(SKEW_DEFAULT_THRESHOLDS, SKEW_CUSTOM_THRESHOLDS)\n",
|
||||
"drift_thresholds = get_thresholds(DRIFT_DEFAULT_THRESHOLDS, DRIFT_CUSTOM_THRESHOLDS)\n",
|
||||
"attrib_skew_thresholds = get_thresholds(\n",
|
||||
" ATTRIB_SKEW_DEFAULT_THRESHOLDS, ATTRIB_SKEW_CUSTOM_THRESHOLDS\n",
|
||||
")\n",
|
||||
"attrib_drift_thresholds = get_thresholds(\n",
|
||||
" ATTRIB_DRIFT_DEFAULT_THRESHOLDS, ATTRIB_DRIFT_CUSTOM_THRESHOLDS\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"skew_config = ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(\n",
|
||||
" skew_thresholds=skew_thresholds,\n",
|
||||
" attribution_score_skew_thresholds=attrib_skew_thresholds,\n",
|
||||
" skew_thresholds=skew_thresholds\n",
|
||||
")\n",
|
||||
"drift_config = ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(\n",
|
||||
" drift_thresholds=drift_thresholds,\n",
|
||||
" attribution_score_drift_thresholds=attrib_drift_thresholds,\n",
|
||||
" drift_thresholds=drift_thresholds\n",
|
||||
")\n",
|
||||
"explanation_config = ModelMonitoringObjectiveConfig.ExplanationConfig(\n",
|
||||
" enable_feature_attributes=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"training_dataset = ModelMonitoringObjectiveConfig.TrainingDataset(target_field=TARGET)\n",
|
||||
"training_dataset.bigquery_source = BigQuerySource(input_uri=DATASET_BQ_URI)\n",
|
||||
"objective_config = ModelMonitoringObjectiveConfig(\n",
|
||||
" training_dataset=training_dataset,\n",
|
||||
" training_prediction_skew_detection_config=skew_config,\n",
|
||||
" prediction_drift_detection_config=drift_config,\n",
|
||||
" explanation_config=explanation_config,\n",
|
||||
")\n",
|
||||
"model_ids = get_deployed_model_ids(ENDPOINT_ID)\n",
|
||||
"objective_template = ModelDeploymentMonitoringObjectiveConfig(\n",
|
||||
@@ -939,7 +798,7 @@
|
||||
"<br>\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm6.png\" />\n",
|
||||
"<br>\n",
|
||||
"As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to see an example of the layout of these measurements in Cloud Storage. If you substitute the Cloud Storage URL in your job creation email, you can view the structure and content of the data files for your own monitoring job."
|
||||
"As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to take a look at your measurements in Cloud Storage.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -966,8 +825,7 @@
|
||||
"- **instance_schemas/job-../analysis** - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).\n",
|
||||
"- **instance_schemas/job-../predict** - This is the first prediction made to your model after the current monitoring job was enabled.\n",
|
||||
"- **model_monitoring/job-../serving** - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.\n",
|
||||
"- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data.\n",
|
||||
"- **model_monitoring/job-../feature_attribution_score** - This folder is used to record data relevant to feature attribution calculations. It contains an ongoing summary of feature attribution scores relative to training data."
|
||||
"- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -989,11 +847,7 @@
|
||||
"source": [
|
||||
"## Generate test data to trigger alerting\n",
|
||||
"\n",
|
||||
"Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. This cell runs two five minute tests, one minute apart, so it should take roughly eleven minutes to complete the test.\n",
|
||||
"\n",
|
||||
"The first test sends 300 fabricated requests (one per second for five minutes) while perturbing two features of interest (cnt_level_start_quickplay and country) by a factor of two. The second test does the same thing but perturbs the selected feature distributions by a factor of three. By perturbing data in two experiments, we're able to trigger both skew and drift alerts.\n",
|
||||
"\n",
|
||||
"After running this test, it takes at least an hour to assess and report skew and drift alerts so feel free to proceed with the notebook now and you'll see how to examine the resulting alerts later."
|
||||
"Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. It takes about five minutes to run this cell and at least an hour to assess and report anamolies in skew or drift so after running this cell, feel free to proceed with the notebook and you'll see how to examine the resulting alert later."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1041,24 +895,13 @@
|
||||
" print(\"Test Completed.\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"start = 2\n",
|
||||
"end = 3\n",
|
||||
"for multiplier in range(start, end + 1):\n",
|
||||
" test_time = 300\n",
|
||||
" tests_per_sec = 1\n",
|
||||
" sleep_time = 1 / tests_per_sec\n",
|
||||
" iterations = test_time * tests_per_sec\n",
|
||||
" perturb_num = {\n",
|
||||
" \"cnt_level_start_quickplay\": (\n",
|
||||
" lambda x: x * multiplier,\n",
|
||||
" lambda x: x / multiplier,\n",
|
||||
" )\n",
|
||||
" }\n",
|
||||
" perturb_cat = {\"Japan\": max(COUNTRY.values()) * multiplier}\n",
|
||||
" monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)\n",
|
||||
" if multiplier < end:\n",
|
||||
" print(\"sleeping...\")\n",
|
||||
" time.sleep(60)"
|
||||
"test_time = 300\n",
|
||||
"tests_per_sec = 1\n",
|
||||
"sleep_time = 1 / tests_per_sec\n",
|
||||
"iterations = test_time * tests_per_sec\n",
|
||||
"perturb_num = {\"cnt_user_engagement\": (lambda x: x * 3, lambda x: x / 3)}\n",
|
||||
"perturb_cat = {\"Japan\": max(COUNTRY.values()) * 2}\n",
|
||||
"monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1089,7 +932,7 @@
|
||||
"id": "HoaqsxpaRs1m"
|
||||
},
|
||||
"source": [
|
||||
"This email is warning you that the *cnt_level_start_quickplay*, *cnt_user_engagement*, and *country* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* and *country* feature attribution values are skewed relative to your training data, again, as per your threshold specification."
|
||||
"This email is warning you that the *cnt_user_engagement*, *country* and *language* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* feature value is drifting significantly over time, again, as per your threshold specification."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1111,25 +954,11 @@
|
||||
"source": [
|
||||
"#### Monitoring Status\n",
|
||||
"\n",
|
||||
"You can verify that a given endpoint has an active model monitoring job via the Endpoint summary page:\n",
|
||||
"\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm1.png\" />\n",
|
||||
"\n",
|
||||
"#### Monitoring Alerts\n",
|
||||
"\n",
|
||||
"You can examine the alert details by clicking into the endpoint of interest, and selecting the alerts panel:\n",
|
||||
"\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm2.png\" />\n",
|
||||
"\n",
|
||||
"#### Feature Value Distributions\n",
|
||||
"\n",
|
||||
"You can also examine the recorded training and production feature distributions by drilling down into a given feature, like this:\n",
|
||||
"\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm9.png\" />\n",
|
||||
"\n",
|
||||
"which yields graphical representations of the feature distrubution during both training and production, like this:\n",
|
||||
"\n",
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm8.png\" />"
|
||||
"<img src=\"https://storage.googleapis.com/mco-general/img/mm2.png\" />"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1173,16 +1002,17 @@
|
||||
"\n",
|
||||
"- [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv)\n",
|
||||
"- [Data Understanding, Validation, and Monitoring At Scale](https://blog.tensorflow.org/2018/09/introducing-tensorflow-data-validation.html)\n",
|
||||
"- [Vertex Product Documentation](https://cloud.google.com/vertex-ai)\n",
|
||||
"- [Vertex AI Model Monitoring Reference Docs](https://cloud.google.com/vertex-ai/docs/reference)\n",
|
||||
"- [Vertex AI Model Monitoring blog article](https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai)\n",
|
||||
"- [Explainable AI Whitepaper](https://storage.googleapis.com/cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf)"
|
||||
"- [Vertex Product Documentation](https://cloud.google.com/vertex)\n",
|
||||
"- [Vertex AI Model Monitoring Reference Docs](https://cloud.google.com/vertex/docs/reference)\n",
|
||||
"- [Vertex AI Model Monitoring blog article](https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"collapsed_sections": [
|
||||
"lA32H1oKGgpf"
|
||||
],
|
||||
"name": "model_monitoring.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+26
@@ -682,6 +682,32 @@
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "define_component:print_op"
|
||||
},
|
||||
"source": [
|
||||
"## Define pipeline components\n",
|
||||
"\n",
|
||||
"The following example define a custom pipeline component for this tutorial:\n",
|
||||
"\n",
|
||||
"- This component doesn't do anything (but run a print statement)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "define_component:print_op"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@component\n",
|
||||
"def print_op(input1: str):\n",
|
||||
" print(\"training task: {}\".format(input1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
|
||||
+8
-87
@@ -90,8 +90,7 @@
|
||||
"\n",
|
||||
"- Upload a pre-trained model as a `Model` resource.\n",
|
||||
"- Run a `BatchPredictionJob` on the `Model` resource with ground truth data.\n",
|
||||
"- Generate evaluation `Metrics` artifact about the `Model` resource.\n",
|
||||
"- Compare the evaluation metrics to a threshold.\n"
|
||||
"- Generate Evaluation metrics about the `Model`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -155,7 +154,7 @@
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the latest version of Vertex AI SDK for Python."
|
||||
"Install the latest version of Vertex SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -194,8 +193,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
|
||||
"! pip3 install --upgrade kfp $USER_FLAG"
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -233,7 +231,7 @@
|
||||
"id": "check_versions"
|
||||
},
|
||||
"source": [
|
||||
"Check the versions of the packages you installed. "
|
||||
"Check the versions of the packages you installed. The KFP SDK version should be >=1.6."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -542,21 +540,10 @@
|
||||
"):\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud auth list 2>/dev/null\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
|
||||
" SERVICE_ACCOUNT = shell_output[2].strip()\n",
|
||||
" print(\"Service Account:\", SERVICE_ACCOUNT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "9ca2fb92cb31"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"shell_output[2].replace(\"*\", \"\").strip()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -601,8 +588,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import google.cloud.aiplatform as aip\n",
|
||||
"from kfp.v2.dsl import Input, Metrics, component"
|
||||
"import google.cloud.aiplatform as aip"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -649,59 +635,6 @@
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d7a5cc8c7f3c"
|
||||
},
|
||||
"source": [
|
||||
"## Create component for comparing evalution metrics to a threshold\n",
|
||||
"\n",
|
||||
"First, you create your own component that will take as input the evaluation metrics artifact and make a comparison to a threshold and return a yes/no decision that could be used in a subsequent dsl.Condition() to decide whether the model should proceed to the next step -- e.g., online deployment.\n",
|
||||
"\n",
|
||||
"The component takes the following parameters:\n",
|
||||
"\n",
|
||||
"- `eval_metrics`: The evaluation metrics artifact returned from `ModelEvaluation` component.\n",
|
||||
"- `metric_name`: The key name for the metric entry to make the comparison to.\n",
|
||||
"- `threshold`: The threshold for the metric value for a yes/no decision."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "5cf109a9bcca"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@component()\n",
|
||||
"def compare(eval_metrics: Input[Metrics], metric_name: str, threshold: float) -> str:\n",
|
||||
" path = eval_metrics.path\n",
|
||||
" # print(\"PATH\", path)\n",
|
||||
"\n",
|
||||
" gs_prefix = \"gs://\"\n",
|
||||
" gcsfuse_prefix = \"/gcs/\"\n",
|
||||
" if path.startswith(gs_prefix):\n",
|
||||
" path = path.replace(gs_prefix, gcsfuse_prefix)\n",
|
||||
"\n",
|
||||
" import json\n",
|
||||
"\n",
|
||||
" with open(path, \"r\") as f:\n",
|
||||
" data = json.load(f)\n",
|
||||
"\n",
|
||||
" slices = data[\"slicedMetrics\"]\n",
|
||||
" # print(\"# slices\", len(slices))\n",
|
||||
"\n",
|
||||
" metrics = slices[0][\"metrics\"][\"classification\"]\n",
|
||||
" # print(\"METRIC KEYS\", metrics.keys())\n",
|
||||
"\n",
|
||||
" value = metrics[metric_name]\n",
|
||||
" if value > threshold:\n",
|
||||
" return \"true\"\n",
|
||||
"\n",
|
||||
" return \"false\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -751,8 +684,6 @@
|
||||
"\n",
|
||||
"@kfp.dsl.pipeline(name=\"upload-evaluate-\" + TIMESTAMP)\n",
|
||||
"def pipeline(\n",
|
||||
" metric: str,\n",
|
||||
" threshold: float,\n",
|
||||
" project: str = PROJECT_ID,\n",
|
||||
" model_display_name: str = MODEL_DISPLAY_NAME,\n",
|
||||
" batch_prediction_display_name: str = BATCH_PREDICTION_DISPLAY_NAME,\n",
|
||||
@@ -792,7 +723,7 @@
|
||||
" machine_type=\"n1-standard-32\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" eval_task = evaluation_op(\n",
|
||||
" evaluation_op(\n",
|
||||
" project=project,\n",
|
||||
" root_dir=WORKING_DIR,\n",
|
||||
" problem_type=\"classification\",\n",
|
||||
@@ -801,12 +732,6 @@
|
||||
" class_names=[\"0\", \"1\"],\n",
|
||||
" predictions_format=\"jsonl\",\n",
|
||||
" batch_prediction_job=batch_prediction_task.outputs[\"batchpredictionjob\"],\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" _ = compare(\n",
|
||||
" eval_metrics=eval_task.outputs[\"evaluation_metrics\"],\n",
|
||||
" metric_name=metric,\n",
|
||||
" threshold=threshold,\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
@@ -862,7 +787,6 @@
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" template_path=\"evaluation_demo_pipeline.json\",\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
" parameter_values={\"metric\": \"auPrc\", \"threshold\": 0.95},\n",
|
||||
" enable_caching=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
@@ -972,10 +896,7 @@
|
||||
"artifacts = print_pipeline_output(job, \"model-batch-predict\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"model-evaluation\")\n",
|
||||
"metrics = print_pipeline_output(job, \"model-evaluation\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"compare\")\n",
|
||||
"artifacts = print_pipeline_output(job, \"compare\")"
|
||||
"metrics = print_pipeline_output(job, \"model-evaluation\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
-1388
File diff suppressed because it is too large
Load Diff
-169
@@ -1,169 +0,0 @@
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
"""
|
||||
data_preprocessing.py is the module for
|
||||
|
||||
- ingest data
|
||||
- do simple preprocessing tasks
|
||||
- upload processed data to gcs
|
||||
"""
|
||||
|
||||
# Libraries --------------------------------------------------------------------------------
|
||||
import logging
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
try:
|
||||
from pyspark import SparkContext, SparkConf
|
||||
from pyspark.sql import SparkSession
|
||||
except ImportError as error:
|
||||
print('WARN: Something wrong with pyspark library. Please check configuration settings!')
|
||||
print(error)
|
||||
|
||||
from pyspark.sql.types import StructType, DoubleType, StringType
|
||||
|
||||
# Variables --------------------------------------------------------------------------------
|
||||
DATA_SCHEMA = (StructType()
|
||||
.add("label", StringType(), True)
|
||||
.add("loan_amount", StringType(), True)
|
||||
.add("loan_term", StringType(), True)
|
||||
.add("property_area", StringType(), True)
|
||||
.add("timestamp", StringType(), True)
|
||||
.add("entity_type_customer_id", StringType(), True)
|
||||
.add("feature_7", DoubleType(), True)
|
||||
.add("feature_3", DoubleType(), True)
|
||||
.add("feature_1", DoubleType(), True)
|
||||
.add("feature_9", DoubleType(), True)
|
||||
.add("feature_5", DoubleType(), True)
|
||||
.add("feature_0", DoubleType(), True)
|
||||
.add("feature_8", DoubleType(), True)
|
||||
.add("feature_4", DoubleType(), True)
|
||||
.add("feature_2", DoubleType(), True)
|
||||
.add("feature_6", DoubleType(), True)
|
||||
)
|
||||
|
||||
ENTITY_CUSTOMER_ID = 'entity_type_customer_id'
|
||||
FEATURE_STORE_IDS = ['timestamp', 'entity_type_customer_id']
|
||||
CATEGORICAL_VARIABLES = ['loan_term', 'property_area']
|
||||
IDX_CATEGORICAL_FEATURES = [f'{col}_idx' for col in CATEGORICAL_VARIABLES]
|
||||
TARGET = 'label'
|
||||
|
||||
|
||||
# Helpers ----------------------------------------------------------------------------------
|
||||
|
||||
def set_logger():
|
||||
"""
|
||||
Set logger for the module
|
||||
Returns:
|
||||
logger: logger object
|
||||
"""
|
||||
fmt_pattern = "%(asctime)s — %(name)s — %(levelname)s —" "%(funcName)s:%(lineno)d — %(message)s"
|
||||
main_logger = logging.getLogger(__name__)
|
||||
main_logger.setLevel(logging.INFO)
|
||||
main_logger.propagate = False
|
||||
stream_handler = logging.StreamHandler(sys.stdout)
|
||||
stream_handler.setLevel(logging.INFO)
|
||||
formatter = logging.Formatter(fmt_pattern)
|
||||
stream_handler.setFormatter(formatter)
|
||||
main_logger.addHandler(stream_handler)
|
||||
return main_logger
|
||||
|
||||
|
||||
def get_args():
|
||||
"""
|
||||
Get arguments from command line
|
||||
Returns:
|
||||
args: arguments from command line
|
||||
"""
|
||||
args_parser = argparse.ArgumentParser()
|
||||
args_parser.add_argument(
|
||||
'--train-data-path',
|
||||
help='The GCS path of training sample',
|
||||
type=str,
|
||||
required=True)
|
||||
args_parser.add_argument(
|
||||
'--out-process-path',
|
||||
help='''
|
||||
The path to load processed data.
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=True)
|
||||
return args_parser.parse_args()
|
||||
|
||||
|
||||
# Main -------------------------------------------------------------------------------------
|
||||
|
||||
def main(logger, args):
|
||||
"""
|
||||
Main function
|
||||
Args:
|
||||
logger: logger object
|
||||
args: arguments from command line
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
# variables
|
||||
train_data_path = args.train_data_path
|
||||
output_data_path = args.out_process_path
|
||||
|
||||
logger.info('initializing data preprocessing.')
|
||||
logger.info('start spark session.')
|
||||
|
||||
spark = (SparkSession.builder
|
||||
.master("local[*]")
|
||||
.appName("spark go live")
|
||||
.config('spark.ui.port', '4050')
|
||||
.getOrCreate())
|
||||
try:
|
||||
logger.info(f'spark version: {spark.sparkContext.version}')
|
||||
logger.info('start ingesting data.')
|
||||
|
||||
training_data_raw_df = (spark.read.option("header", True)
|
||||
.option("delimiter", ',')
|
||||
.schema(DATA_SCHEMA)
|
||||
.csv(train_data_path)
|
||||
.drop(*FEATURE_STORE_IDS))
|
||||
|
||||
training_data_raw_df = training_data_raw_df.withColumn("label",
|
||||
training_data_raw_df.label.cast('double'))
|
||||
training_data_raw_df = training_data_raw_df.withColumn("loan_amount",
|
||||
training_data_raw_df.loan_amount.cast('double'))
|
||||
training_data_raw_df.show(truncate=False)
|
||||
|
||||
logger.info(f'load prepared data to {output_data_path}.')
|
||||
if output_data_path.startswith('gs://'):
|
||||
training_data_raw_df.write.mode('overwrite').csv(str(output_data_path), header=True)
|
||||
else:
|
||||
output_file_path = Path(output_data_path)
|
||||
output_file_path.mkdir(parents=True, exist_ok=True)
|
||||
training_data_raw_df.write.mode('overwrite').csv(str(output_file_path), header=True)
|
||||
except RuntimeError as main_error:
|
||||
logger.error(main_error)
|
||||
else:
|
||||
logger.info('data preprocessing successfully completed!')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
runtime_args = get_args()
|
||||
runtime_logger = set_logger()
|
||||
main(runtime_logger, runtime_args)
|
||||
-366
@@ -1,366 +0,0 @@
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
"""
|
||||
hp_model_tuning.py is the module for hypertune the spark pipeline
|
||||
"""
|
||||
|
||||
# Libraries --------------------------------------------------------------------------------
|
||||
import logging
|
||||
import sys
|
||||
import argparse
|
||||
from os import environ
|
||||
from datetime import datetime
|
||||
from pathlib import Path as path
|
||||
import tempfile
|
||||
from urllib.parse import urlparse, urljoin
|
||||
import json
|
||||
|
||||
try:
|
||||
from pyspark import SparkContext, SparkConf
|
||||
from pyspark.sql import SparkSession
|
||||
except ImportError as e:
|
||||
print('WARN: Something wrong with pyspark library. Please check configuration settings!')
|
||||
print(e)
|
||||
|
||||
from pyspark.sql.types import StructType, DoubleType, StringType
|
||||
from pyspark.sql.functions import col, udf
|
||||
from pyspark.sql.functions import round as spark_round
|
||||
from pyspark.ml.feature import StringIndexer, StandardScaler, VectorAssembler
|
||||
from pyspark.ml.classification import RandomForestClassifier
|
||||
from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator
|
||||
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
|
||||
from pyspark.ml import Pipeline
|
||||
|
||||
from google.cloud import storage
|
||||
|
||||
# Variables --------------------------------------------------------------------------------
|
||||
|
||||
# Data schema
|
||||
DATA_SCHEMA = (StructType()
|
||||
.add("label", DoubleType(), True)
|
||||
.add("loan_amount", DoubleType(), True)
|
||||
.add("loan_term", StringType(), True)
|
||||
.add("property_area", StringType(), True)
|
||||
.add("feature_7", DoubleType(), True)
|
||||
.add("feature_3", DoubleType(), True)
|
||||
.add("feature_1", DoubleType(), True)
|
||||
.add("feature_9", DoubleType(), True)
|
||||
.add("feature_5", DoubleType(), True)
|
||||
.add("feature_0", DoubleType(), True)
|
||||
.add("feature_8", DoubleType(), True)
|
||||
.add("feature_4", DoubleType(), True)
|
||||
.add("feature_2", DoubleType(), True)
|
||||
.add("feature_6", DoubleType(), True)
|
||||
)
|
||||
|
||||
# Training
|
||||
TARGET = 'label'
|
||||
CATEGORICAL_VARIABLES = ['loan_term', 'property_area']
|
||||
IDX_CATEGORICAL_FEATURES = [f'{col}_idx' for col in CATEGORICAL_VARIABLES]
|
||||
REAL_TIME_FEATURES_VECTOR = 'real_time_features_vector'
|
||||
REAL_TIME_FEATURES = 'real_time_features'
|
||||
FEATURES_SELECTED = ['feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5',
|
||||
'feature_6', 'feature_7', 'feature_8', 'feature_9', 'real_time_features']
|
||||
FEATURES = 'features'
|
||||
RANDOM_SEED = 8
|
||||
RANDOM_QUOTAS = [0.8, 0.2]
|
||||
MAX_DEPTH = [5, 10, 15]
|
||||
MAX_BINS = [24, 32, 40]
|
||||
N_TREES = [25, 30, 35]
|
||||
N_FOLDS = 5
|
||||
|
||||
|
||||
# Helpers ----------------------------------------------------------------------------------
|
||||
def set_logger():
|
||||
"""
|
||||
Set logger for the module
|
||||
Returns:
|
||||
logger: logger object
|
||||
"""
|
||||
fmt_pattern = "%(asctime)s — %(name)s — %(levelname)s —" "%(funcName)s:%(lineno)d — %(message)s"
|
||||
main_logger = logging.getLogger(__name__)
|
||||
main_logger.setLevel(logging.INFO)
|
||||
main_logger.propagate = False
|
||||
stream_handler = logging.StreamHandler(sys.stdout)
|
||||
stream_handler.setLevel(logging.INFO)
|
||||
formatter = logging.Formatter(fmt_pattern)
|
||||
stream_handler.setFormatter(formatter)
|
||||
main_logger.addHandler(stream_handler)
|
||||
return main_logger
|
||||
|
||||
|
||||
def get_args():
|
||||
"""
|
||||
Get arguments from command line
|
||||
Returns:
|
||||
args: arguments from command line
|
||||
"""
|
||||
args_parser = argparse.ArgumentParser()
|
||||
args_parser.add_argument(
|
||||
'--train-path',
|
||||
help='''
|
||||
The GCS path of training data'
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=False)
|
||||
args_parser.add_argument(
|
||||
'--model-path',
|
||||
help='''
|
||||
The GCS path to store the trained model.
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=False)
|
||||
args_parser.add_argument(
|
||||
'--metrics-path',
|
||||
help='''
|
||||
The GCS path to store the metrics of model.
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=True)
|
||||
return args_parser.parse_args()
|
||||
|
||||
|
||||
def build_preprocessing_components():
|
||||
"""
|
||||
Build preprocessing components
|
||||
Returns:
|
||||
preprocessing_components: preprocessing components
|
||||
"""
|
||||
loan_term_indexer = StringIndexer(inputCol=CATEGORICAL_VARIABLES[0], outputCol=IDX_CATEGORICAL_FEATURES[0],
|
||||
stringOrderType='frequencyDesc', handleInvalid='keep')
|
||||
property_area_indexer = StringIndexer(inputCol=CATEGORICAL_VARIABLES[1], outputCol=IDX_CATEGORICAL_FEATURES[1],
|
||||
stringOrderType='frequencyDesc', handleInvalid='keep')
|
||||
data_preprocessing_stages = [loan_term_indexer, property_area_indexer]
|
||||
return data_preprocessing_stages
|
||||
|
||||
|
||||
def build_feature_engineering_components():
|
||||
"""
|
||||
Build feature engineering components
|
||||
Returns:
|
||||
feature_engineering_components: feature engineering components
|
||||
"""
|
||||
feature_engineering_stages = []
|
||||
realtime_vector_assembler = VectorAssembler(inputCols=IDX_CATEGORICAL_FEATURES, outputCol=REAL_TIME_FEATURES_VECTOR)
|
||||
realtime_scaler = StandardScaler(inputCol=REAL_TIME_FEATURES_VECTOR, outputCol=REAL_TIME_FEATURES)
|
||||
features_vector_assembler = VectorAssembler(inputCols=FEATURES_SELECTED, outputCol=FEATURES)
|
||||
feature_engineering_stages.extend((realtime_vector_assembler,
|
||||
realtime_scaler,
|
||||
features_vector_assembler))
|
||||
return feature_engineering_stages
|
||||
|
||||
|
||||
def build_training_model_component():
|
||||
"""
|
||||
Build training model component
|
||||
Returns:
|
||||
training_model_component: training model component
|
||||
"""
|
||||
model_training_stage = []
|
||||
rfor = RandomForestClassifier(featuresCol=FEATURES, labelCol=TARGET, seed=RANDOM_SEED)
|
||||
model_training_stage.append(rfor)
|
||||
return model_training_stage
|
||||
|
||||
|
||||
def build_hp_pipeline(data_preprocessing_stages, feature_engineering_stages, model_training_stage):
|
||||
"""
|
||||
Build hyperparameter pipeline
|
||||
Args:
|
||||
data_preprocessing_stages: preprocessing components
|
||||
feature_engineering_stages: feature engineering components
|
||||
model_training_stage: training model component
|
||||
Returns:
|
||||
hp_pipeline: hyperparameter pipeline
|
||||
"""
|
||||
pipeline = Pipeline(stages=data_preprocessing_stages + feature_engineering_stages + model_training_stage)
|
||||
params_grid = (ParamGridBuilder()
|
||||
.addGrid(model_training_stage[0].maxDepth, MAX_DEPTH)
|
||||
.addGrid(model_training_stage[0].maxBins, MAX_BINS)
|
||||
.addGrid(model_training_stage[0].numTrees, N_TREES)
|
||||
.build())
|
||||
evaluator = BinaryClassificationEvaluator(labelCol=TARGET)
|
||||
cross_validator = CrossValidator(estimator=pipeline,
|
||||
estimatorParamMaps=params_grid,
|
||||
evaluator=evaluator,
|
||||
numFolds=N_FOLDS)
|
||||
return cross_validator
|
||||
|
||||
|
||||
def get_true_score_prediction(predictions, target):
|
||||
"""
|
||||
Get true score and prediction
|
||||
Args:
|
||||
predictions: predictions
|
||||
target: target column
|
||||
Returns:
|
||||
roc_dict: a dict of roc values for each class
|
||||
"""
|
||||
|
||||
split1_udf = udf(lambda value: value[1].item(), DoubleType())
|
||||
roc_dataset = predictions.select(col(target).alias('true'),
|
||||
spark_round(split1_udf('probability'), 5).alias('score'),
|
||||
'prediction')
|
||||
roc_df = roc_dataset.toPandas()
|
||||
roc_dict = roc_df.to_dict(orient='list')
|
||||
return roc_dict
|
||||
|
||||
|
||||
def get_metrics(predictions, target, mode):
|
||||
"""
|
||||
Get metrics
|
||||
Args:
|
||||
predictions: predictions
|
||||
target: target column
|
||||
mode: train or test
|
||||
Returns:
|
||||
metrics: metrics
|
||||
"""
|
||||
metric_labels = ['area_roc', 'area_prc', 'accuracy', 'f1', 'precision', 'recall']
|
||||
metric_cols = ['true', 'score', 'prediction']
|
||||
metric_keys = [f'{mode}_{ml}' for ml in metric_labels] + metric_cols
|
||||
|
||||
bc_evaluator = BinaryClassificationEvaluator(labelCol=target)
|
||||
mc_evaluator = MulticlassClassificationEvaluator(labelCol=target)
|
||||
|
||||
# areas, acc, f1, prec, rec
|
||||
metric_values = []
|
||||
area_roc = round(bc_evaluator.evaluate(predictions, {bc_evaluator.metricName: 'areaUnderROC'}), 5)
|
||||
area_prc = round(bc_evaluator.evaluate(predictions, {bc_evaluator.metricName: 'areaUnderPR'}), 5)
|
||||
acc = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "accuracy"}), 5)
|
||||
f1 = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "f1"}), 5)
|
||||
prec = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "weightedPrecision"}), 5)
|
||||
rec = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "weightedRecall"}), 5)
|
||||
|
||||
# true, score, prediction
|
||||
roc_dict = get_true_score_prediction(predictions, target)
|
||||
true = roc_dict['true']
|
||||
score = roc_dict['score']
|
||||
pred = roc_dict['prediction']
|
||||
|
||||
metric_values.extend((area_roc, area_prc, acc, f1, prec, rec, true, score, pred))
|
||||
metrics = dict(zip(metric_keys, metric_values))
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
def upload_file(bucket_name, source_file_name, destination_blob_name):
|
||||
storage_client = storage.Client()
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(destination_blob_name)
|
||||
blob.upload_from_filename(source_file_name)
|
||||
|
||||
|
||||
def write_metrics(bucket_name, metrics, destination, dir='/tmp'):
|
||||
temp_dir = tempfile.TemporaryDirectory(dir=dir)
|
||||
temp_metrics_file_path = str(path(temp_dir.name) / path(destination).name)
|
||||
with open(temp_metrics_file_path, 'w') as temp_file:
|
||||
json.dump(metrics, temp_file)
|
||||
upload_file(bucket_name, temp_metrics_file_path, destination)
|
||||
temp_dir.cleanup()
|
||||
|
||||
|
||||
# Main -------------------------------------------------------------------------------------
|
||||
|
||||
def main(logger, args):
|
||||
"""
|
||||
Main function
|
||||
Args:
|
||||
logger: logger
|
||||
args: args
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
train_path = args.train_path
|
||||
model_path = args.model_path
|
||||
metrics_path = args.metrics_path
|
||||
|
||||
try:
|
||||
logger.info('initializing pipeline training.')
|
||||
logger.info('start spark session.')
|
||||
spark = (SparkSession.builder
|
||||
.master("local[*]")
|
||||
.appName("spark go live")
|
||||
.config('spark.ui.port', '4050')
|
||||
.config('spark.jars.packages', 'ml.combust.mleap:mleap-runtime_2.12:0.19.0')
|
||||
.config('spark.jars.packages', 'ml.combust.mleap:mleap-base_2.12:0.19.0')
|
||||
.config('spark.jars.packages', 'ml.combust.mleap:mleap-spark_2.12:0.19.0')
|
||||
.config('spark.jars.packages', 'ml.combust.mleap:mleap-spark-extension_2.12:0.19.0')
|
||||
.getOrCreate())
|
||||
logger.info(f'spark version: {spark.sparkContext.version}')
|
||||
logger.info('start building pipeline.')
|
||||
preprocessing_stages = build_preprocessing_components()
|
||||
feature_engineering_stages = build_feature_engineering_components()
|
||||
model_training_stage = build_training_model_component()
|
||||
pipeline_cross_validator = build_hp_pipeline(preprocessing_stages, feature_engineering_stages,
|
||||
model_training_stage)
|
||||
logger.info(f'load train data from {train_path}.')
|
||||
if train_path.startswith('bq://'):
|
||||
raw_data = spark.read.format('bigquery') \
|
||||
.option('table', train_path.replace('bq://', '')) \
|
||||
.load()
|
||||
else:
|
||||
raw_data = (spark.read.format('csv')
|
||||
.option("header", "true")
|
||||
.schema(DATA_SCHEMA)
|
||||
.load(train_path))
|
||||
logger.info(f'fit model pipeline.')
|
||||
train, test = raw_data.randomSplit(RANDOM_QUOTAS, seed=RANDOM_SEED)
|
||||
pipeline_model = pipeline_cross_validator.fit(train)
|
||||
predictions = pipeline_model.transform(test)
|
||||
metrics = get_metrics(predictions, TARGET, 'test')
|
||||
for m, v in metrics.items():
|
||||
print(f'{m}: {v}')
|
||||
|
||||
logger.info(f'load model pipeline in {model_path}.')
|
||||
if model_path.startswith('gs://'):
|
||||
pipeline_model.write().overwrite().save(model_path)
|
||||
else:
|
||||
path(model_path).mkdir(parents=True, exist_ok=True)
|
||||
pipeline_model.write().overwrite().save(model_path)
|
||||
|
||||
logger.info(f'Upload metrics under {metrics_path}.')
|
||||
if metrics_path.startswith('gs://'):
|
||||
bucket = urlparse(model_path).netloc
|
||||
metrics_file_path = urlparse(metrics_path).path.strip('/')
|
||||
write_metrics(bucket, metrics, metrics_file_path)
|
||||
else:
|
||||
metrics_version_path = path(metrics_path).parents[0]
|
||||
metrics_version_path.mkdir(parents=True, exist_ok=True)
|
||||
with open(metrics_path, 'w') as json_file:
|
||||
json.dump(metrics, json_file)
|
||||
json_file.close()
|
||||
except RuntimeError as main_error:
|
||||
logger.error(main_error)
|
||||
else:
|
||||
logger.info('model pipeline training successfully completed!')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
runtime_args = get_args()
|
||||
runtime_logger = set_logger()
|
||||
main(runtime_logger, runtime_args)
|
||||
-359
@@ -1,359 +0,0 @@
|
||||
# Copyright 2022 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
"""
|
||||
model_training.py is the module for training spark pipeline
|
||||
"""
|
||||
|
||||
# Libraries --------------------------------------------------------------------------------
|
||||
import logging
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path as path
|
||||
import tempfile
|
||||
import json
|
||||
from urllib.parse import urlparse
|
||||
|
||||
try:
|
||||
from pyspark import SparkContext, SparkConf
|
||||
from pyspark.sql import SparkSession
|
||||
except ImportError as e:
|
||||
print('WARN: Something wrong with pyspark library. Please check configuration settings!')
|
||||
print(e)
|
||||
|
||||
from pyspark.sql.types import StructType, DoubleType, StringType
|
||||
from pyspark.sql.functions import col, udf
|
||||
from pyspark.sql.functions import round as spark_round
|
||||
from pyspark.ml.feature import StringIndexer, StandardScaler, VectorAssembler
|
||||
from pyspark.ml.classification import RandomForestClassifier
|
||||
from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator
|
||||
from pyspark.ml import Pipeline
|
||||
|
||||
from google.cloud import storage
|
||||
|
||||
# Variables --------------------------------------------------------------------------------
|
||||
|
||||
# Data schema
|
||||
DATA_SCHEMA = (StructType()
|
||||
.add("label", DoubleType(), True)
|
||||
.add("loan_amount", DoubleType(), True)
|
||||
.add("loan_term", StringType(), True)
|
||||
.add("property_area", StringType(), True)
|
||||
.add("feature_7", DoubleType(), True)
|
||||
.add("feature_3", DoubleType(), True)
|
||||
.add("feature_1", DoubleType(), True)
|
||||
.add("feature_9", DoubleType(), True)
|
||||
.add("feature_5", DoubleType(), True)
|
||||
.add("feature_0", DoubleType(), True)
|
||||
.add("feature_8", DoubleType(), True)
|
||||
.add("feature_4", DoubleType(), True)
|
||||
.add("feature_2", DoubleType(), True)
|
||||
.add("feature_6", DoubleType(), True)
|
||||
)
|
||||
|
||||
# Training
|
||||
TARGET = 'label'
|
||||
CATEGORICAL_VARIABLES = ['loan_term', 'property_area']
|
||||
IDX_CATEGORICAL_FEATURES = [f'{col}_idx' for col in CATEGORICAL_VARIABLES]
|
||||
REAL_TIME_FEATURES_VECTOR = 'real_time_features_vector'
|
||||
REAL_TIME_FEATURES = 'real_time_features'
|
||||
FEATURES_SELECTED = ['feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5',
|
||||
'feature_6', 'feature_7', 'feature_8', 'feature_9', 'real_time_features']
|
||||
FEATURES = 'features'
|
||||
RANDOM_SEED = 8
|
||||
RANDOM_QUOTAS = [0.8, 0.2]
|
||||
|
||||
|
||||
# Helpers ----------------------------------------------------------------------------------
|
||||
def set_logger():
|
||||
"""
|
||||
Set logger
|
||||
Returns:
|
||||
logger: logger
|
||||
"""
|
||||
fmt_pattern = "%(asctime)s — %(name)s — %(levelname)s —" "%(funcName)s:%(lineno)d — %(message)s"
|
||||
main_logger = logging.getLogger(__name__)
|
||||
main_logger.setLevel(logging.INFO)
|
||||
main_logger.propagate = False
|
||||
stream_handler = logging.StreamHandler(sys.stdout)
|
||||
stream_handler.setLevel(logging.INFO)
|
||||
formatter = logging.Formatter(fmt_pattern)
|
||||
stream_handler.setFormatter(formatter)
|
||||
main_logger.addHandler(stream_handler)
|
||||
return main_logger
|
||||
|
||||
|
||||
def get_args():
|
||||
"""
|
||||
Get arguments
|
||||
Returns:
|
||||
args: arguments
|
||||
"""
|
||||
args_parser = argparse.ArgumentParser()
|
||||
args_parser.add_argument(
|
||||
'--train-path',
|
||||
help='''
|
||||
The GCS path of training data'
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=True)
|
||||
args_parser.add_argument(
|
||||
'--model-path',
|
||||
help='''
|
||||
The GCS path to store the trained model.
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=True)
|
||||
args_parser.add_argument(
|
||||
'--metrics-path',
|
||||
help='''
|
||||
The GCS path to store the metrics of model.
|
||||
Format:
|
||||
- locally: /path/to/dir
|
||||
- cloud: gs://bucket/path
|
||||
''',
|
||||
type=str,
|
||||
required=True)
|
||||
return args_parser.parse_args()
|
||||
|
||||
|
||||
def build_preprocessing_components():
|
||||
"""
|
||||
Build preprocessing components
|
||||
Returns:
|
||||
data_preprocessing_stages: data preprocessing stages
|
||||
"""
|
||||
loan_term_indexer = StringIndexer(inputCol=CATEGORICAL_VARIABLES[0], outputCol=IDX_CATEGORICAL_FEATURES[0],
|
||||
stringOrderType='frequencyDesc', handleInvalid='keep')
|
||||
property_area_indexer = StringIndexer(inputCol=CATEGORICAL_VARIABLES[1], outputCol=IDX_CATEGORICAL_FEATURES[1],
|
||||
stringOrderType='frequencyDesc', handleInvalid='keep')
|
||||
data_preprocessing_stages = [loan_term_indexer, property_area_indexer]
|
||||
return data_preprocessing_stages
|
||||
|
||||
|
||||
def build_feature_engineering_components():
|
||||
"""
|
||||
Build feature engineering components
|
||||
Returns:
|
||||
feature_engineering_stages: feature engineering stages
|
||||
"""
|
||||
feature_engineering_stages = []
|
||||
realtime_vector_assembler = VectorAssembler(inputCols=IDX_CATEGORICAL_FEATURES, outputCol=REAL_TIME_FEATURES_VECTOR)
|
||||
realtime_scaler = StandardScaler(inputCol=REAL_TIME_FEATURES_VECTOR, outputCol=REAL_TIME_FEATURES)
|
||||
features_vector_assembler = VectorAssembler(inputCols=FEATURES_SELECTED, outputCol=FEATURES)
|
||||
feature_engineering_stages.extend((realtime_vector_assembler,
|
||||
realtime_scaler,
|
||||
features_vector_assembler))
|
||||
return feature_engineering_stages
|
||||
|
||||
|
||||
def build_training_model_component():
|
||||
"""
|
||||
Build training model component
|
||||
Returns:
|
||||
model_training_stage: model_training_stage
|
||||
"""
|
||||
model_training_stage = []
|
||||
rfor = RandomForestClassifier(featuresCol=FEATURES, labelCol=TARGET, seed=RANDOM_SEED)
|
||||
model_training_stage.append(rfor)
|
||||
return model_training_stage
|
||||
|
||||
|
||||
def build_pipeline(data_preprocessing_stages, feature_engineering_stages, model_training_stage):
|
||||
"""
|
||||
Build pipeline
|
||||
Args:
|
||||
data_preprocessing_stages: data preprocessing stages
|
||||
feature_engineering_stages: feature engineering stages
|
||||
model_training_stage: model_training_stage
|
||||
Returns:
|
||||
pipeline: pipeline
|
||||
"""
|
||||
pipeline = Pipeline(stages=data_preprocessing_stages + feature_engineering_stages + model_training_stage)
|
||||
return pipeline
|
||||
|
||||
|
||||
def get_true_score_prediction(predictions, target):
|
||||
"""
|
||||
Get true score prediction
|
||||
Args:
|
||||
predictions: predictions
|
||||
target: target
|
||||
Returns:
|
||||
roc_dict: a dict of roc values for each class
|
||||
"""
|
||||
split1_udf = udf(lambda value: value[1].item(), DoubleType())
|
||||
roc_dataset = predictions.select(col(target).alias('true'),
|
||||
spark_round(split1_udf('probability'), 5).alias('score'),
|
||||
'prediction')
|
||||
roc_df = roc_dataset.toPandas()
|
||||
roc_dict = roc_df.to_dict(orient='list')
|
||||
return roc_dict
|
||||
|
||||
|
||||
def get_metrics(predictions, target, mode):
|
||||
"""
|
||||
Get metrics
|
||||
Args:
|
||||
predictions: predictions
|
||||
target: target column name
|
||||
mode: train or test
|
||||
Returns:
|
||||
metrics: metrics
|
||||
"""
|
||||
metric_labels = ['area_roc', 'area_prc', 'accuracy', 'f1', 'precision', 'recall']
|
||||
metric_cols = ['true', 'score', 'prediction']
|
||||
metric_keys = [f'{mode}_{ml}' for ml in metric_labels] + metric_cols
|
||||
bc_evaluator = BinaryClassificationEvaluator(labelCol=target)
|
||||
mc_evaluator = MulticlassClassificationEvaluator(labelCol=target)
|
||||
|
||||
# areas, acc, f1, prec, rec
|
||||
metric_values = []
|
||||
area_roc = round(bc_evaluator.evaluate(predictions, {bc_evaluator.metricName: 'areaUnderROC'}), 5)
|
||||
area_prc = round(bc_evaluator.evaluate(predictions, {bc_evaluator.metricName: 'areaUnderPR'}), 5)
|
||||
acc = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "accuracy"}), 5)
|
||||
f1 = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "f1"}), 5)
|
||||
prec = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "weightedPrecision"}), 5)
|
||||
rec = round(mc_evaluator.evaluate(predictions, {mc_evaluator.metricName: "weightedRecall"}), 5)
|
||||
|
||||
# true, score, prediction
|
||||
roc_dict = get_true_score_prediction(predictions, target)
|
||||
true = roc_dict['true']
|
||||
score = roc_dict['score']
|
||||
pred = roc_dict['prediction']
|
||||
|
||||
metric_values.extend((area_roc, area_prc, acc, f1, prec, rec, true, score, pred))
|
||||
metrics = dict(zip(metric_keys, metric_values))
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
def upload_file(bucket_name, source_file_name, destination_blob_name):
|
||||
"""
|
||||
Upload file to bucket
|
||||
Args:
|
||||
bucket_name: bucket name
|
||||
source_file_name: source file name
|
||||
destination_blob_name: destination blob name
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
storage_client = storage.Client()
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(destination_blob_name)
|
||||
blob.upload_from_filename(source_file_name)
|
||||
|
||||
|
||||
def write_metrics(bucket_name, metrics, destination, dir='/tmp'):
|
||||
"""
|
||||
Write metrics to file
|
||||
Args:
|
||||
bucket_name: bucket name
|
||||
metrics: metrics
|
||||
destination: destination
|
||||
dir: directory to write file temporarily
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
temp_dir = tempfile.TemporaryDirectory(dir=dir)
|
||||
temp_metrics_file_path = str(path(temp_dir.name) / path(destination).name)
|
||||
with open(temp_metrics_file_path, 'w') as temp_file:
|
||||
json.dump(metrics, temp_file)
|
||||
upload_file(bucket_name, temp_metrics_file_path, destination)
|
||||
temp_dir.cleanup()
|
||||
|
||||
|
||||
# Main -------------------------------------------------------------------------------------
|
||||
|
||||
def main(logger, args):
|
||||
"""
|
||||
Main function
|
||||
Args:
|
||||
logger: logger
|
||||
args: args
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
train_path = args.train_path
|
||||
model_path = args.model_path
|
||||
metrics_path = args.metrics_path
|
||||
|
||||
try:
|
||||
logger.info('initializing pipeline training.')
|
||||
logger.info('start spark session.')
|
||||
spark = (SparkSession.builder
|
||||
.master("local[*]")
|
||||
.appName("spark go live")
|
||||
.config('spark.ui.port', '4050')
|
||||
.getOrCreate())
|
||||
logger.info(f'spark version: {spark.sparkContext.version}')
|
||||
logger.info('start bulding pipeline.')
|
||||
preprocessing_stages = build_preprocessing_components()
|
||||
feature_engineering_stages = build_feature_engineering_components()
|
||||
model_training_stage = build_training_model_component()
|
||||
pipeline = build_pipeline(preprocessing_stages, feature_engineering_stages, model_training_stage)
|
||||
|
||||
logger.info(f'load train data from {train_path}.')
|
||||
raw_data = (spark.read.format('csv')
|
||||
.option("header", "true")
|
||||
.schema(DATA_SCHEMA)
|
||||
.load(train_path))
|
||||
|
||||
logger.info(f'fit model pipeline.')
|
||||
train, test = raw_data.randomSplit(RANDOM_QUOTAS, seed=RANDOM_SEED)
|
||||
pipeline_model = pipeline.fit(train)
|
||||
predictions = pipeline_model.transform(test)
|
||||
metrics = get_metrics(predictions, TARGET, 'test')
|
||||
for m, v in metrics.items():
|
||||
print(f'{m}: {v}')
|
||||
|
||||
logger.info(f'load model pipeline in {model_path}.')
|
||||
pipeline.write().overwrite().save(model_path)
|
||||
if model_path.startswith('gs://'):
|
||||
pipeline.write().overwrite().save(model_path)
|
||||
else:
|
||||
path(model_path).mkdir(parents=True, exist_ok=True)
|
||||
pipeline.write().overwrite().save(model_path)
|
||||
|
||||
logger.info(f'Upload metrics under {metrics_path}.')
|
||||
if metrics_path.startswith('gs://'):
|
||||
bucket = urlparse(model_path).netloc
|
||||
metrics_file_path = urlparse(metrics_path).path.strip('/')
|
||||
write_metrics(bucket, metrics, metrics_file_path)
|
||||
else:
|
||||
metrics_version_path = path(metrics_path).parents[0]
|
||||
metrics_version_path.mkdir(parents=True, exist_ok=True)
|
||||
with open(metrics_path, 'w') as json_file:
|
||||
json.dump(metrics, json_file)
|
||||
json_file.close()
|
||||
except RuntimeError as main_error:
|
||||
logger.error(main_error)
|
||||
else:
|
||||
logger.info('model pipeline training successfully completed!')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
runtime_args = get_args()
|
||||
runtime_logger = set_logger()
|
||||
main(runtime_logger, runtime_args)
|
||||
@@ -235,7 +235,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"if os.environ[\"IS_TESTING\"]:\n",
|
||||
" ! pip3 install --upgrade matplotlib $USER_FLAG"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -456,8 +456,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
|
||||
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -468,8 +467,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -489,7 +488,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION $BUCKET_URI"
|
||||
"! gsutil mb -l $REGION $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -509,7 +508,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil ls -al $BUCKET_URI"
|
||||
"! gsutil ls -al $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -572,9 +571,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
|
||||
"\n",
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
|
||||
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -644,7 +643,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_URI)"
|
||||
"PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -690,7 +689,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
|
||||
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -869,7 +868,9 @@
|
||||
"source": [
|
||||
"from kfp.v2 import compiler # noqa: F811\n",
|
||||
"\n",
|
||||
"compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"intro_pipeline.json\")"
|
||||
"compiler.Compiler().compile(\n",
|
||||
" pipeline_func=pipeline, package_path=\"intro_pipeline.json\".replace(\" \", \"_\")\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -895,7 +896,7 @@
|
||||
"\n",
|
||||
"job = aip.PipelineJob(\n",
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" template_path=\"intro_pipeline.json\",\n",
|
||||
" template_path=\"intro_pipeline.json\".replace(\" \", \"_\"),\n",
|
||||
" pipeline_root=PIPELINE_ROOT,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
@@ -919,28 +920,6 @@
|
||||
"<a href=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" width=\"60%\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d428cc8803e7"
|
||||
},
|
||||
"source": [
|
||||
"### Delete the pipeline job\n",
|
||||
"\n",
|
||||
"You can delete the pipeline job with the method `delete()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "97802d9432e7"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1070,28 +1049,6 @@
|
||||
"job.run()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "37e2b84a0c0d"
|
||||
},
|
||||
"source": [
|
||||
"### Delete the pipeline job\n",
|
||||
"\n",
|
||||
"You can delete the pipeline job with the method `delete()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c5926e7eaad8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1225,7 +1182,16 @@
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial."
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial -- *Note:* this is auto-generated and not all resources may be applicable for this tutorial:\n",
|
||||
"\n",
|
||||
"- Dataset\n",
|
||||
"- Pipeline\n",
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- Batch Job\n",
|
||||
"- Custom Job\n",
|
||||
"- Hyperparameter Tuning Job\n",
|
||||
"- Cloud Storage Bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1236,10 +1202,84 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_dataset = True\n",
|
||||
"delete_pipeline = True\n",
|
||||
"delete_model = True\n",
|
||||
"delete_endpoint = True\n",
|
||||
"delete_batchjob = True\n",
|
||||
"delete_customjob = True\n",
|
||||
"delete_hptjob = True\n",
|
||||
"delete_bucket = True\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" if delete_model and \"DISPLAY_NAME\" in globals():\n",
|
||||
" models = aip.Model.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" model = models[0]\n",
|
||||
" aip.Model.delete(model)\n",
|
||||
" print(\"Deleted model:\", model)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" if delete_endpoint and \"DISPLAY_NAME\" in globals():\n",
|
||||
" endpoints = aip.Endpoint.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" endpoint = endpoints[0]\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" aip.Endpoint.delete(endpoint.resource_name)\n",
|
||||
" print(\"Deleted endpoint:\", endpoint)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_dataset and \"DISPLAY_NAME\" in globals():\n",
|
||||
" if \"none\" == \"tabular\":\n",
|
||||
" try:\n",
|
||||
" datasets = aip.TabularDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" dataset = datasets[0]\n",
|
||||
" aip.TabularDataset.delete(dataset.resource_name)\n",
|
||||
" print(\"Deleted dataset:\", dataset)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"none\" == \"image\":\n",
|
||||
" try:\n",
|
||||
" datasets = aip.ImageDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" dataset = datasets[0]\n",
|
||||
" aip.ImageDataset.delete(dataset.resource_name)\n",
|
||||
" print(\"Deleted dataset:\", dataset)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"none\" == \"text\":\n",
|
||||
" try:\n",
|
||||
" datasets = aip.TextDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" dataset = datasets[0]\n",
|
||||
" aip.TextDataset.delete(dataset.resource_name)\n",
|
||||
" print(\"Deleted dataset:\", dataset)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
" if \"none\" == \"video\":\n",
|
||||
" try:\n",
|
||||
" datasets = aip.VideoDataset.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
" )\n",
|
||||
" dataset = datasets[0]\n",
|
||||
" aip.VideoDataset.delete(dataset.resource_name)\n",
|
||||
" print(\"Deleted dataset:\", dataset)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
|
||||
" pipelines = aip.PipelineJob.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
@@ -1250,8 +1290,8 @@
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"\n",
|
||||
"if delete_bucket and \"BUCKET_URI\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
"2. **Configure parameters**: Setting the appropriate parameter values for the training job.\n",
|
||||
"3. **Train on Vertex Training**: Submitting a training job using csv input.\n",
|
||||
"4. **Hyperparameter tuning**: Running a hyperparameter tuning job.\n",
|
||||
"5. **Hyperparameter on Vertex Training with BigQuery input**: Submitting a training job using BigQuery input.\n",
|
||||
"5. **Hyperparameter on Vertex Training with BigQuerry input**: Submitting a training job using BigQuerry input.\n",
|
||||
"6. **Cleaning up**: Deleting resources created by this tutorial.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -522,8 +522,6 @@
|
||||
"import time\n",
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"\n",
|
||||
"%load_ext tensorboard"
|
||||
]
|
||||
},
|
||||
@@ -592,7 +590,7 @@
|
||||
"| `training_data_path` | string | Cloud Storage pattern where training data is stored. | Yes |\n",
|
||||
"| `validation_data_path` | string | Cloud Storage pattern where eval data is stored. | No |\n",
|
||||
"| `test_data_path` | string | Cloud Storage pattern where test data is stored. | Yes |\n",
|
||||
"| `input_type` | string | “bigquery“ or “csv“ - type of the input tabular data. If csv is mentioned then the first column is treated as target. If CSV files have a header, also pass the flag “data_has_header”. If “bigquery” is used, one can either supply training/validation data paths, or supply BigQuery project, dataset, and table names for preprocessing to produce the training and validation datasets.. | No -Deafult is \"csv\" |\n",
|
||||
"| `input_type` | string | “bigquery“ or “csv“ - type of the input tabular data. If csv is mentioned then the first column is treated as target. If CSV files have a header, also pass the flag “data_has_header”. If “bigquery” is used, one can either supply training/validation data paths, or supply BigQuery project, dataset, and table names for preprocessing to produce the training and validation datasets.. | Yes |\n",
|
||||
"| `model_type` | string | The learning task such as classification or regression. | Yes |\n",
|
||||
"| `split_column` | string | The column name used to create the training, validation, and test splits. Values of the columns (a.k.a table['split_column']) should be either “TRAIN”, “VALIDATE”, or “TEST”. “TEST” is optional. Applicable to bigquery input only. | No. |\n",
|
||||
"| `train_batch_size` | int | Batch size for training. | No - Default is 1024. |\n",
|
||||
@@ -600,6 +598,7 @@
|
||||
"| `learning_rate` | float | Learning rate for training. | No - Default is the default learning rate of the specified optimizer. |\n",
|
||||
"| `eval_frequency_secs` | int | Frequency at which evaluation and checkpointing will take place.The default is 600. | No . |\n",
|
||||
"| `num_parallel_reads` | int | Number of threads used to read input files. We suggest setting it equal or slightly less than the number of CPUs of the machine for maximal performance in most cases. For example, 6 per GPU is a good default choice. | Yes . |\n",
|
||||
"| `optimizer` | string | Training optimizer. Lowercase string name of any TF2.3 Keras optimizer is supported ('sgd', 'adam', 'ftrl', etc.). See [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers). | No - Default is 'adam'. |\n",
|
||||
"| `data_cache` | string | Choose to cache data to “memory”, “disk” or “no_cache”. For large datasets, caching the data into memory would throw out-of-memory errors, therefore, we suggest choosing “disk”. You can specify the disk size in the config file (as exemplified below). Make sure to request a sufficiently large (e.g. TB size) disk to write the data, for large (B-scale) datasets. | No, The default one is “memory” . |\n",
|
||||
"| `bq_project` | string | The name of the BigQuery project. If input_type=bigquery and using the flag –preprocessing, this is required. This is an alternative to specifying train, validation, and test data paths. | No . |\n",
|
||||
"| `dataset_name` | string | The name of the BigQuery dataset. If input_type=bigquery and using the flag –preprocessing, this is required. This is an alternative to specifying train, validation, and test data paths. | No . |\n",
|
||||
@@ -627,9 +626,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# URI of the TabNet training Docker image.\n",
|
||||
"LEARNER_IMAGE_URI = (\n",
|
||||
" \"us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/tab_net_v2\"\n",
|
||||
")\n",
|
||||
"LEARNER_IMAGE_URI = 'us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/tab_net_v2'\n",
|
||||
"\n",
|
||||
"DATASET_NAME = \"petfinder\" # Change to your dataset name.\n",
|
||||
"\n",
|
||||
@@ -671,8 +668,6 @@
|
||||
" --display-name={learning_job_name} \\\n",
|
||||
" --worker-pool-spec=machine-type=n1-standard-8,replica-count=1,container-image-uri={LEARNER_IMAGE_URI} \\\n",
|
||||
" --args=--preprocess \\\n",
|
||||
" --args=--prediction_raw_inputs \\\n",
|
||||
" --args=--exclude_key \\\n",
|
||||
" --args=--data_has_header \\\n",
|
||||
" --args=--training_data_path={TRAINING_DATA_PATH} \\\n",
|
||||
" --args=--job-dir={OUTPUT_DIR} \\\n",
|
||||
@@ -723,149 +718,6 @@
|
||||
" time.sleep(60)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2ed50da51105"
|
||||
},
|
||||
"source": [
|
||||
"# Deploy on Vertex Prediction\n",
|
||||
"## Import the model\n",
|
||||
"Our training job will export two TF SavedModel under `gs://<job_dir>/model`. The exported model can be used for online or batch prediction in Vertex Prediction. First, import the model:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e9dee2e91da8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Imports the model.\n",
|
||||
"current_time = datetime.utcnow().strftime(\"%y%m%d%H%M%S\")\n",
|
||||
"MODEL_TYPE = \"classification\"\n",
|
||||
"\n",
|
||||
"DISPLAY_NAME = f\"tabnet_{DATASET_NAME}_{MODEL_TYPE}_{current_time}\" # The display name of the model.\n",
|
||||
"MODEL_NAME = f\"tabnet_{MODEL_TYPE}_model\" # Used by the deployment container.\n",
|
||||
"ARTIFACT_URI = f\"{OUTPUT_DIR}/model/\"\n",
|
||||
"\n",
|
||||
"print(\"Display_name:\", DISPLAY_NAME)\n",
|
||||
"print(\"Model name: \", MODEL_NAME)\n",
|
||||
"\n",
|
||||
"aiplatform.init(\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" location=REGION,\n",
|
||||
" staging_bucket=BUCKET_NAME,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model = aiplatform.Model.upload(\n",
|
||||
" display_name=DISPLAY_NAME,\n",
|
||||
" artifact_uri=ARTIFACT_URI,\n",
|
||||
" serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-4:latest\",\n",
|
||||
")\n",
|
||||
"print(model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6aad41003302"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy the model\n",
|
||||
"After importing the model, you must deploy it to an endpoint so that you can get online predictions. More information about this process can be found in the [official documentation](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d463a410ac44"
|
||||
},
|
||||
"source": [
|
||||
"## Create a model endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "10224e3190c3"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = aiplatform.Endpoint.create(display_name=DATASET_NAME)\n",
|
||||
"\n",
|
||||
"# If you want to use existing endpoint\n",
|
||||
"# endpoint = aiplatform.Endpoint('[location of existing endpoint]')\n",
|
||||
"print(\"Endpoint: \", endpoint)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "33fe5010c84e"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy model to the endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "d29efd150e44"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model.deploy(\n",
|
||||
" endpoint=endpoint,\n",
|
||||
" machine_type=\"n1-standard-4\",\n",
|
||||
" traffic_split={\"0\": 100},\n",
|
||||
" deployed_model_display_name=DISPLAY_NAME,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "92aaeb09cd63"
|
||||
},
|
||||
"source": [
|
||||
"## Online prediction"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2288d32ddfb8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prediction = endpoint.predict(\n",
|
||||
" [\n",
|
||||
" {\n",
|
||||
" \"Type\": \"Cat\",\n",
|
||||
" \"Age\": 3,\n",
|
||||
" \"Breed1\": \"Tabby\",\n",
|
||||
" \"Gender\": \"Male\",\n",
|
||||
" \"Color1\": \"Black\",\n",
|
||||
" \"Color2\": \"White\",\n",
|
||||
" \"MaturitySize\": \"Small\",\n",
|
||||
" \"FurLength\": \"Short\",\n",
|
||||
" \"Vaccinated\": \"No\",\n",
|
||||
" \"Sterilized\": \"No\",\n",
|
||||
" \"Health\": \"Healthy\",\n",
|
||||
" \"Fee\": 100,\n",
|
||||
" \"PhotoAmt\": 2,\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(prediction)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -920,8 +772,6 @@
|
||||
" imageUri: {LEARNER_IMAGE_URI}\n",
|
||||
" args:\n",
|
||||
" - --preprocess \n",
|
||||
" - --prediction_raw_inputs\n",
|
||||
" - --exclude_key\n",
|
||||
" - --data_has_header\n",
|
||||
" - --training_data_path={TRAINING_DATA_PATH}\n",
|
||||
" - --job-dir={OUTPUT_DIR}\n",
|
||||
@@ -1048,8 +898,6 @@
|
||||
" - --eval_frequency_secs=10800\n",
|
||||
" - --input_type=bigquery\n",
|
||||
" - --preprocess\n",
|
||||
" - --prediction_raw_inputs\n",
|
||||
" - --exclude_key\n",
|
||||
" - --model_type=classification\n",
|
||||
" - --stream_inputs\n",
|
||||
" - --bq_project={BQ_PROJECT}\n",
|
||||
@@ -1109,12 +957,6 @@
|
||||
"# Delete BQ table\n",
|
||||
"! bq rm -f {PROJECT_ID}:{DATASET_NAME}.train\n",
|
||||
"\n",
|
||||
"# Delete endpoint resource\n",
|
||||
"endpoint.delete(force=True)\n",
|
||||
"\n",
|
||||
"# Delete model resource\n",
|
||||
"model.delete()\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil -m rm -r $BUCKET_URI"
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user