Merge branch 'main' into ml_ops_7

* Added notebook

* Ran linter test

Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>

.cloud-build/test_folders.txt

@@ -1,2 +1,3 @@
 notebooks/official
 notebooks/notebook_template.ipynb
+notebooks/community/ml_ops

.github/workflows/linter/requirements.txt

@@ -2,7 +2,7 @@ git+https://github.com/tensorflow/docs
 ipython
 jupyter
 nbconvert
-black==22.1.0
+black==22.3.0
 pyupgrade==2.31.1
 isort==5.10.1
 flake8==4.0.1

community-content/CODEOWNERS

@@ -3,3 +3,4 @@
 /pytorch_text_classification_using_vertex_sdk_and_gcloud  @RajeshThallam
 /pytorch_text_classification_using_vertex_sdk_and_gcloud		@RajeshThallam @ultrons
 /sklearn_text_classification_from_script_using_vertex_sdk       @maxhardt
+/pluto_on_workbench @wkharold

community-content/pluto_on_workbench/README.md

@@ -0,0 +1,52 @@
+# Overview
+*Pluto* is a programming environment for Julia, designed to be interactive and helpful. It provides a familiar notebook interface but it is not a Jupyter notebook. The biggest difference is that Pluto notebooks are reactive, changing a variable or function in one cell causes the cells that depend on that variable or function to be reevaluated. Pluto also provides useful interaction mechanisms that allow users to dynamically interact with the notebooks computation state.
+
+The JuliaCon 2020 presentation: [Interactive notebooks ~ Pluto.jl]() provides a good introduction to Pluto. The source is at [fonsp/Pluto.jl]()
+
+# Install Pluto
+
+## Create a Vertex AI JupyterLab Instance
+
+1. From the [GCP console](https://console.cloud.google.com) "hamburger menu"
+
+   select Vertex AI > Workbench
+2. Click NEW NOTEBOOK
+   
+   * Choose Python 3 if you won't be using a GPU
+   * Choose Python 3 (CUDA Toolkit xx.y) if you do want use a GPU
+3. Give the notebook an appropriate name
+4. Edit Notebook properties if you have special requirements otherwise accept the defaults and click CREATE
+5. When the notebook instance is ready click OPEN JUPYTERLAB
+
+## Configure JupyterLab
+
+1. Open a terminal by clicking the Terminal icon.
+1. Install the plutoserver
+pip3 install git+https://github.com/fonsp/pluto-on-jupyterlab.git
+1. In a browser go to [julialang.org/downloads](https://julialang.org/downloads/)
+1. In the Current stable release right click on the `Generic Linux on x86 / 64-bit (glibc)` link
+Select copy link address
+1. Back in the terminal switch to root via
+sudo -i
+1. Download the release to /opt and install julia in /usr/local/bin
+    ```bash
+    cd /opt
+    wget <paste the release link address>
+    tar xf <name of the downloaded tar file>
+    ln -s /opt/<julia-x.y.z>/bin/julia /usr/local/bin
+    ^d
+    ```
+1. Add the Pluto package to Julia
+   ```bash
+    julia
+    julia> ]add Pluto
+    julia> bksp
+    julia> using Pluto
+    julia> ^d
+    ```
+1. From the JupyterLab menu bar select File > Shut Down
+
+# Start Pluto
+1. Click OPEN JUPYTERLAB in the Workbench
+1. In the Notebook section of the Launcher click Pluto.jl
+1. The welcome to Pluto.jl screen should appear

notebooks/community/ml_ops/stage1/README.md

@@ -76,6 +76,18 @@ The steps performed include:
     - image data
 ```
 
+[Get Started with Data Labeling](get_started_data_labeling.ipynb)
+
+```
+The steps performed include:
+
+- Create a Specialist Pool for data labelers.
+- Create a data labeling job.
+- Submit the data labeling job.
+- List data labeling jobs.
+- Cancel a data labeling job.
+```
+
 ### E2E Stage Example
 
 [Stage 1: Data Management](mlops_data_management.ipynb)

notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -39,8 +39,9 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -67,7 +68,7 @@
       "source": [
         "### Dataset\n",
         "\n",
-        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
+        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). In this version of the dataset you consider the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
       ]
     },
     {
@@ -104,7 +105,7 @@
       "source": [
         "### Recommendations\n",
         "\n",
-        "When doing E2E MLOps on Google Cloud, the following best practices with structured (tabular) data in BigQuery:\n",
+        "When doing E2E MLOps on Google Cloud, following are the best practices when dealing with structured (tabular) data in BigQuery:\n",
         "\n",
         "- For AutoML training:\n",
         "  - Create a managed dataset with Vertex AI `TabularDataset`.\n",
@@ -124,7 +125,7 @@
         "        - Within the generator (upstream)\n",
         "        - Within the model (downstream)\n",
         "    - XGBoost model training:\n",
-        "      - Use BigQuery ML builtin XGBoost training.\n",
+        "      - Use BigQuery ML built-in XGBoost training.\n",
         "      - Alternatively, create a DMatrix generator from CSV files extracted from BigQuery table.\n",
         "    - Pytorch model training:\n",
         "        - Extract the BigQuery to a pandas dataframe.\n",
@@ -132,10 +133,19 @@
         "        - Create a DataLoader generator from the pandas dataframe.\n",
         "\n",
         "\n",
-        "- Alternately:\n",
+        "- Alternatively:\n",
         "    - Extract the BigQuery table to CSV files.\n",
         "    - Preprocess the CSV files.\n",
-        "    - Create a tf.data.Dataset generator from the CSV files."
+        "    - Create a tf.data.Dataset generator from the CSV files.\n",
+        "    \n",
+        "### Costs\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "- Vertex AI\n",
+        "- Cloud Storage\n",
+        "- BigQuery\n",
+        "\n",
+        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
       ]
     },
     {
@@ -146,7 +156,7 @@
       "source": [
         "## Installations\n",
         "\n",
-        "Install *one time* the packages for executing the MLOps notebooks."
+        "Install the following packages to execute this notebook."
       ]
     },
     {
@@ -157,40 +167,22 @@
       },
       "outputs": [],
       "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "install_xgboost"
-      },
-      "source": [
-        "Install the latest GA version of *XGBoost* library as well."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "install_xgboost"
-      },
-      "outputs": [],
-      "source": [
-        "! pip3 install -U xgboost $USER_FLAG"
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\"\n",
+        "\n",
+        "# Install the packages\n",
+        "! pip3 install --upgrade pyarrow $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "! pip3 install -U xgboost $USER_FLAG\n",
+        "! pip3 install -U tensorflow-io==0.18 $USER_FLAG"
       ]
     },
     {
@@ -222,6 +214,39 @@
         "    app.kernel.do_shutdown(True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a47846030fef"
+      },
+      "source": [
+        "## Before you begin"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "84cd83853240"
+      },
+      "source": [
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
+        "\n",
+        "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
+        "\n",
+        "1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
+        "\n",
+        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -298,7 +323,10 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -335,12 +363,7 @@
         "\n",
         "**The following steps are required, regardless of your notebook environment.**\n",
         "\n",
-        "When you submit a custom training job using the Vertex SDK, you upload a Python package\n",
-        "containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
-        "the code from this package. In this tutorial, Vertex AI also saves the\n",
-        "trained model that results from your job in the same bucket. You can then\n",
-        "create an `Endpoint` resource based on this output in order to serve\n",
-        "online predictions.\n",
+        "When you create a dataset resource using the Vertex SDK, you can provide a Cloud Storage bucket that contains the data. Vertex AI creates the dataset resource from the data. In this tutorial, Vertex AI also creates a dataset resource from your data in the Cloud Storage bucket.\n",
         "\n",
         "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
       ]
@@ -353,7 +376,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -364,8 +387,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -385,7 +408,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -405,7 +428,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -414,9 +437,6 @@
         "id": "setup_vars"
       },
       "source": [
-        "### Set up variables\n",
-        "\n",
-        "Next, set up some variables used throughout the tutorial.\n",
         "### Import libraries and define constants"
       ]
     },
@@ -428,75 +448,12 @@
       },
       "outputs": [],
       "source": [
-        "import google.cloud.aiplatform as aip"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_bq"
-      },
-      "source": [
-        "#### Import BigQuery\n",
-        "\n",
-        "Import the BigQuery package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_bq"
-      },
-      "outputs": [],
-      "source": [
+        "import google.cloud.aiplatform as aiplatform\n",
+        "import pandas as pd\n",
+        "import xgboost as xgb\n",
         "from google.cloud import bigquery"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_xgboost"
-      },
-      "source": [
-        "#### Import XGBoost\n",
-        "\n",
-        "Import the XGBoost package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_xgboost"
-      },
-      "outputs": [],
-      "source": [
-        "import xgboost as xgb"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_pandas"
-      },
-      "source": [
-        "#### Import pandas\n",
-        "\n",
-        "Import the pandas package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_pandas"
-      },
-      "outputs": [],
-      "source": [
-        "import pandas as pd"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -516,7 +473,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, location=REGION)"
+        "aiplatform.init(project=PROJECT_ID, location=REGION)"
       ]
     },
     {
@@ -538,7 +495,7 @@
       },
       "outputs": [],
       "source": [
-        "bqclient = bigquery.Client()"
+        "bqclient = bigquery.Client(project=PROJECT_ID)"
       ]
     },
     {
@@ -549,7 +506,7 @@
       "source": [
         "#### Location of BigQuery training data.\n",
         "\n",
-        "Now set the variable `IMPORT_FILE` to the location of the data table in BigQuery."
+        "Now, set the variable `IMPORT_FILE` to the location of the data table in BigQuery and `BQ_TABLE` with the table id."
       ]
     },
     {
@@ -591,10 +548,10 @@
       },
       "outputs": [],
       "source": [
-        "dataset = aip.TabularDataset.create(\n",
+        "dataset = aiplatform.TabularDataset.create(\n",
         "    display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
         "    bq_source=[IMPORT_FILE],\n",
-        "    labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
+        "    labels={\"user_metadata\": BUCKET_URI[5:]},\n",
         ")\n",
         "\n",
         "label_column = \"mean_temp\"\n",
@@ -610,7 +567,7 @@
       "source": [
         "### Copy the dataset to Cloud Storage\n",
         "\n",
-        "Next, you make a copy of the BigQuery dataset, as a CSV file, to Cloud Storage using the BigQuery extract command.\n",
+        "Next, you make a copy of the BigQuery table as a CSV file, to Cloud Storage using the BigQuery extract command.\n",
         "\n",
         "Learn more about [BigQuery command line interface](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference)."
       ]
@@ -626,9 +583,9 @@
         "comps = BQ_TABLE.split(\".\")\n",
         "BQ_PROJECT_DATASET_TABLE = comps[0] + \":\" + comps[1] + \".\" + comps[2]\n",
         "\n",
-        "! bq --location=us extract --destination_format CSV $BQ_PROJECT_DATASET_TABLE $BUCKET_NAME/mydata*.csv\n",
+        "! bq --location=us extract --destination_format CSV $BQ_PROJECT_DATASET_TABLE $BUCKET_URI/mydata*.csv\n",
         "\n",
-        "IMPORT_FILES = ! gsutil ls $BUCKET_NAME/mydata*.csv\n",
+        "IMPORT_FILES = ! gsutil ls $BUCKET_URI/mydata*.csv\n",
         "\n",
         "print(IMPORT_FILES)\n",
         "\n",
@@ -664,15 +621,12 @@
       },
       "outputs": [],
       "source": [
-        "if \"IMPORT_FILES\" in globals():\n",
-        "    gcs_source = IMPORT_FILES\n",
-        "else:\n",
-        "    gcs_source = [IMPORT_FILE]\n",
+        "gcs_source = IMPORT_FILES\n",
         "\n",
-        "dataset = aip.TabularDataset.create(\n",
+        "dataset = aiplatform.TabularDataset.create(\n",
         "    display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
         "    gcs_source=gcs_source,\n",
-        "    labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
+        "    labels={\"user_metadata\": BUCKET_URI[5:]},\n",
         ")\n",
         "\n",
         "\n",
@@ -694,6 +648,30 @@
         "Learn more about [Creating BigQuery views](https://cloud.google.com/bigquery/docs/views)."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7dc142433e50"
+      },
+      "outputs": [],
+      "source": [
+        "# Set dataset name and view name in BigQuery\n",
+        "BQ_MY_DATASET = \"[your-dataset-name]\"\n",
+        "BQ_MY_TABLE = \"[your-view-name]\"\n",
+        "\n",
+        "# Otherwise, use the default names\n",
+        "if (\n",
+        "    BQ_MY_DATASET == \"\"\n",
+        "    or BQ_MY_DATASET is None\n",
+        "    or BQ_MY_DATASET == \"[your-dataset-name]\"\n",
+        "):\n",
+        "    BQ_MY_DATASET = \"mlops_dataset_\" + TIMESTAMP\n",
+        "\n",
+        "if BQ_MY_TABLE == \"\" or BQ_MY_TABLE is None or BQ_MY_TABLE == \"[your-view-name]\":\n",
+        "    BQ_MY_TABLE = \"mlops_view_\" + TIMESTAMP"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -702,8 +680,7 @@
       },
       "outputs": [],
       "source": [
-        "BQ_MY_DATASET = 'mydataset'\n",
-        "BQ_MY_TABLE = 'myview'\n",
+        "# Create the resources\n",
         "! bq --location=US mk -d \\\n",
         "$PROJECT_ID:$BQ_MY_DATASET\n",
         "\n",
@@ -744,8 +721,8 @@
       },
       "outputs": [],
       "source": [
-        "# Download a table.\n",
-        "table = bigquery.TableReference.from_string(\"bigquery-public-data.samples.gsod\")\n",
+        "# Download the table.\n",
+        "table = bigquery.TableReference.from_string(BQ_TABLE)\n",
         "\n",
         "rows = bqclient.list_rows(\n",
         "    table,\n",
@@ -1031,22 +1008,6 @@
         "TABLE_ID = \"gsod\"\n",
         "\n",
         "\n",
-        "def create_bigquery_dataset(dataset_id):\n",
-        "    dataset = bigquery.Dataset(\n",
-        "        bigquery.dataset.DatasetReference(PROJECT_ID, dataset_id)\n",
-        "    )\n",
-        "    dataset.location = \"us\"\n",
-        "\n",
-        "    try:\n",
-        "        dataset = bqclient.create_dataset(dataset)  # API request\n",
-        "        return True\n",
-        "    except Exception as err:\n",
-        "        print(err)\n",
-        "        if err.code != 409:  # http_client.CONFLICT\n",
-        "            raise\n",
-        "    return False\n",
-        "\n",
-        "\n",
         "def load_data_into_bigquery(url, dataset_id, table_id):\n",
         "    create_bigquery_dataset(dataset_id)\n",
         "    dataset = bqclient.dataset(dataset_id)\n",
@@ -1079,13 +1040,11 @@
       "source": [
         "### Read BigQuery table into XGboost DMatrix\n",
         "\n",
-        "Currently, there is no direct data feeding connector between BigQuery and the open source XGBoost.\n",
+        "Currently, there is no direct data feeding connector between BigQuery and the open source XGBoost. The BigQuery ML service has a built-in XGBoost training module.\n",
         "\n",
-        "The BigQuery ML service has XGBoost training builtin.\n",
+        "Alernatively, you extract the data either as a pandas dataframe or as CSV files. The extracted data is then given as an input to a `DMatrix` object when training the model.\n",
         "\n",
-        "Alernatively, you extract the data either as a pandas dataframe or as CSV files. The extracted data is then inputted to a `DMatrix` object when training the model.\n",
-        "\n",
-        "Learn more about [Getting started with builtin XGBoost](https://cloud.google.com/ai-platform/training/docs/algorithms/xgboost-start)"
+        "Learn more about [Getting started with built-in XGBoost](https://cloud.google.com/ai-platform/training/docs/algorithms/xgboost-start)."
       ]
     },
     {
@@ -1096,7 +1055,7 @@
       "source": [
         "### Read pandas table into XGboost DMatrix\n",
         "\n",
-        "Next, you load the pandas dataframe into a `DMatrix` object. XGBoost does not support non-numeric inputs. Any column that is categorical will need to be one-hot encoded prior to loading the dataframe."
+        "Next, you load the pandas dataframe into a `DMatrix` object. XGBoost does not support non-numeric inputs. Any column that is categorical need to be one-hot encoded prior to loading the dataframe."
       ]
     },
     {
@@ -1122,7 +1081,7 @@
       "source": [
         "### Read CSV files into XGboost DMatrix\n",
         "\n",
-        "Currently, there is no Cloud Storage support in XGBoost. If you use CSV files for input, you will need to download them locally."
+        "Currently, there is no Cloud Storage support in XGBoost. If you use CSV files for input, you need to download them locally."
       ]
     },
     {
@@ -1144,87 +1103,37 @@
         "id": "cleanup:mbsdk"
       },
       "source": [
-        "# Cleaning up\n",
+        "# Clean up\n",
         "\n",
         "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
         "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "- Vertex AI Dataset resource\n",
+        "- Cloud Storage Bucket\n",
+        "- BigQuery Dataset"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "cleanup:mbsdk"
+        "id": "47ad926d84e8"
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
+        "import os\n",
         "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the dataset using the Vertex dataset object\n",
+        "dataset.delete()\n",
         "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    # Delete the created GCS bucket\n",
+        "    ! gsutil rm -r $BUCKET_URI\n",
+        "    # Delete the created BigQuery datasets\n",
+        "    ! bq rm -r -f $PROJECT_ID:$BQ_MY_DATASET\n",
+        "    ! bq rm -r -f $PROJECT_ID:$DATASET_ID"
       ]
     }
   ],

notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb

@@ -0,0 +1,991 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 1 : formalization: get started with Vertex AI Data Labeling\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_data_labeling.ipynb\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 1 : data management: get started with Vertex AI Data Labeling service."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:flowers,icn"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use the `Vertex AI Data Labeling` service/\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI Data Labeling`\n",
+        "- `Vertex AI Dataset`\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Create a Specialist Pool for data labelers.\n",
+        "- Create a data labeling job.\n",
+        "- Submit the data labeling job.\n",
+        "- List data labeling jobs.\n",
+        "- Cancel a data labeling job.\n",
+        "\n",
+        "Learn more about [Request a Vertex AI Data Labeling job](https://cloud.google.com/vertex-ai/docs/datasets/data-labeling-job)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_aip"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the following packages to execute this notebook."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\"\n",
+        "\n",
+        "# Install the packages\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-storage"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the Kernel\n",
+        "\n",
+        "Once you've installed the Vertex SDK and Google *cloud-storage*, you need to restart the notebook kernel so it can find the packages.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GIwKc4pk_i_t"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "before_you_begin"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### GPU run-time\n",
+        "\n",
+        "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n",
+        "\n",
+        "### Set up your GCP project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a GCP project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "\n",
+        "3. [Enable the Vertex APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
+        "\n",
+        "4. [Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebooks.\n",
+        "\n",
+        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6Xtp5tvK_i_y"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append onto the name of resources which will be created in this tutorial.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "d70An-Mg_i_2"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d29f41c6619e"
+      },
+      "source": [
+        "#### Email\n",
+        "\n",
+        "You need an email address to send labeling job request to. This is the email address will be the manager of the data labeling specialist pool.\n",
+        "\n",
+        "In this tutorial, if you don't specify an email address, the email address associated with your project ID will be used."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4e443c02540c"
+      },
+      "outputs": [],
+      "source": [
+        "EMAIL = \"[your-email-address]\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ca885f17d6ac"
+      },
+      "outputs": [],
+      "source": [
+        "if EMAIL == \"[your-email-address]\":\n",
+        "    shell_output = ! gcloud auth list 2>/dev/null\n",
+        "    EMAIL = shell_output[3].replace(\"*\", \"\").strip()\n",
+        "\n",
+        "print(EMAIL)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "source": [
+        "### Authenticate your GCP account\n",
+        "\n",
+        "**If you are using Google Cloud Notebooks**, your environment is already\n",
+        "authenticated. Skip this step.\n",
+        "\n",
+        "*Note: If you are on an Vertex AI Workbench notebook and run the cell, the cell knows to skip executing the authentication steps.*\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "chybg3Ap_i_2"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your Google Cloud account. This provides access\n",
+        "# to your Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "# If on Vertex, then don't execute this code\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this tutorial in a notebook locally, replace the string\n",
+        "    # below with the path to your service account key and run this cell to\n",
+        "    # authenticate your Google Cloud account.\n",
+        "    else:\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS your_path_to_credentials.json\n",
+        "\n",
+        "    # Log in to your account on Google Cloud\n",
+        "    ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:batch_prediction"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "This tutorial is designed to use training data that is in a public Cloud Storage bucket and a local Cloud Storage bucket for your batch predictions. You may alternatively use your own training data that you have stored in a local Cloud Storage bucket.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qPPrwWpO_i_6"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fn744B7x_i_7"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_aip"
+      },
+      "source": [
+        "#### Import Vertex AI SDK\n",
+        "\n",
+        "Import the Vertex AI SDK into our Python environment.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "97-XQPkv_i_7"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import sys\n",
+        "import time\n",
+        "\n",
+        "import google.cloud.aiplatform as aip\n",
+        "from google.cloud.aiplatform import gapic\n",
+        "from google.protobuf.json_format import ParseDict\n",
+        "from google.protobuf.struct_pb2 import Value"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aip_constants"
+      },
+      "source": [
+        "#### Vertex AI constants\n",
+        "\n",
+        "Setup up the following constants for Vertex AI:\n",
+        "\n",
+        "- `API_ENDPOINT`: The Vertex AI API service endpoint for dataset, model, job, pipeline and endpoint services.\n",
+        "- `PARENT`: The Vertex AI location root path for dataset, model and endpoint resources."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "kBBYqHEd_i_8"
+      },
+      "outputs": [],
+      "source": [
+        "# API Endpoint\n",
+        "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)\n",
+        "\n",
+        "# Vertex AI location root path for your dataset, model and endpoint resources\n",
+        "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "automl_constants:automl"
+      },
+      "source": [
+        "#### Schema constants\n",
+        "\n",
+        "Next, setup constants for schemas related image classification datasets:\n",
+        "\n",
+        "- Data Labeling (Annotations) Schemas: Tells the managed dataset service how the data is labeled (annotated)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "automl_constants:automl,icn"
+      },
+      "outputs": [],
+      "source": [
+        "# Image labeling task\n",
+        "LABELING_SCHEMA_IMAGE = \"gs://google-cloud-aiplatform/schema/datalabelingjob/inputs/image_classification_1.0.0.yaml\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "clients"
+      },
+      "source": [
+        "## Create clients\n",
+        "\n",
+        "The Vertex AI SDK works as a client/server model. On your side (the Python script) you create a client that sends requests and receives responses from the server (Vertex AI).\n",
+        "\n",
+        "You use several clients in this tutorial, so set them all up upfront.\n",
+        "\n",
+        "- Specialist pool service for specialist pools\n",
+        "- Job Service for data labeling\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "i2p2VYUz_i_-"
+      },
+      "outputs": [],
+      "source": [
+        "# client options same for all services\n",
+        "client_options = {\"api_endpoint\": API_ENDPOINT}\n",
+        "\n",
+        "clients = {}\n",
+        "clients[\"job\"] = gapic.JobServiceClient(client_options=client_options)\n",
+        "\n",
+        "# add client for specialist pool\n",
+        "clients[\"specialist_pool\"] = gapic.SpecialistPoolServiceClient(\n",
+        "    client_options=client_options\n",
+        ")\n",
+        "\n",
+        "for client in clients.items():\n",
+        "    print(client)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1d4b33eb3c71"
+      },
+      "source": [
+        "### Create a CSV file for examples to label\n",
+        "\n",
+        "Next, you will create a CSV file for the examples you are requesting to be labeled. \n",
+        "\n",
+        "In this example, the examples to label are images. For each row in the CSV file, you specify the Cloud Storage location of the image to label."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:flowers,csv,icn"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "\n",
+        "LABELING_FILES = [\n",
+        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/daisy/100080576_f52e8ee070_n.jpg\",\n",
+        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/daisy/102841525_bd6628ae3c.jpg\",\n",
+        "]\n",
+        "\n",
+        "IMPORT_FILE = BUCKET_URI + \"/labeling.csv\"\n",
+        "with tf.io.gfile.GFile(IMPORT_FILE, \"w\") as f:\n",
+        "    for lf in LABELING_FILES:\n",
+        "        f.write(lf + \"\\n\")\n",
+        "\n",
+        "! gsutil cat $IMPORT_FILE"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_a_dataset:migration"
+      },
+      "source": [
+        "## Create a unlabeled dataset\n",
+        "\n",
+        "Next, you create a dataset for the data to be labeled."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9af8aed4e3ab"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = aip.ImageDataset.create(\"labeling_\" + TIMESTAMP)\n",
+        "print(dataset)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "92909927fe7e"
+      },
+      "source": [
+        "## Import the unlabeled data\n",
+        "\n",
+        "Now, import the unlabeled data to the dataset, i.e., the examples to be labeled."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c137a55b822e"
+      },
+      "outputs": [],
+      "source": [
+        "dataset.import_data(\n",
+        "    gcs_source=[IMPORT_FILE],\n",
+        "    import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "trainingpipelines_create:migration,new"
+      },
+      "source": [
+        "## Create a new data specialist pool\n",
+        "\n",
+        "You data labeling job will be sent to a data specialist pool. You may have one or more multiple specialist pools. \n",
+        "\n",
+        "In this next step, you create a new specialist pool with the method `create_specialist_pool()`. The request includes the parameters:\n",
+        "\n",
+        "- `name`: The resource name of the specialist pool.\n",
+        "- `display_name`: A human readable name for the specialist pool.\n",
+        "- `specialist_manager_emails`: A list of the email addresses of the manager(s) for the specialist pool.\n",
+        "\n",
+        "*Note:* You can use an existing specialist pool if one already existed."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "wsXOMVOMq3jO"
+      },
+      "outputs": [],
+      "source": [
+        "specialist_pool = {\n",
+        "    \"name\": \"labeling_\" + TIMESTAMP,\n",
+        "    \"display_name\": \"labeling_\" + TIMESTAMP,\n",
+        "    \"specialist_manager_emails\": [EMAIL],\n",
+        "}\n",
+        "\n",
+        "request = clients[\"specialist_pool\"].create_specialist_pool(\n",
+        "    parent=PARENT, specialist_pool=specialist_pool\n",
+        ")\n",
+        "\n",
+        "result = request.result()\n",
+        "print(result)\n",
+        "\n",
+        "specialist_name = result.name\n",
+        "\n",
+        "specialist_id = specialist_name.split(\"/\")[-1]\n",
+        "\n",
+        "print(specialist_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "train_a_model:migration"
+      },
+      "source": [
+        "## Create data labeling job\n",
+        "\n",
+        "Now that you have a specialist pool, you can send a data labeling request using the `create_data_labeling_job()` method.\n",
+        "\n",
+        "Your request will consist of the following:\n",
+        "\n",
+        "- The Vertex AI Dataset with the unlabeled data.\n",
+        "- Instructions for labeling."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5AY3SJjQq3jQ"
+      },
+      "outputs": [],
+      "source": [
+        "# create placeholder file for instructions for data labeling\n",
+        "! echo \"this is instruction\" >> instruction.txt | gsutil cp instruction.txt $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "trainingpipelines_create:migration,new,request,icn"
+      },
+      "outputs": [],
+      "source": [
+        "LABLEING_SCHEMA = LABELING_SCHEMA_IMAGE\n",
+        "INSTRUCTION_FILE = BUCKET_URI + \"/instruction.txt\"\n",
+        "\n",
+        "inputs = ParseDict({\"annotation_specs\": [\"rose\"]}, Value())\n",
+        "\n",
+        "data_labeling_job = {\n",
+        "    \"display_name\": \"labeling_\" + TIMESTAMP,\n",
+        "    \"datasets\": [dataset.resource_name],\n",
+        "    \"labeler_count\": 1,\n",
+        "    \"instruction_uri\": INSTRUCTION_FILE,\n",
+        "    \"inputs_schema_uri\": LABLEING_SCHEMA,\n",
+        "    \"inputs\": inputs,\n",
+        "    \"annotation_labels\": {\n",
+        "        \"aiplatform.googleapis.com/annotation_set_name\": \"data_labeling_job_specialist_pool\"\n",
+        "    },\n",
+        "    \"specialist_pools\": [specialist_name],\n",
+        "}\n",
+        "\n",
+        "print(data_labeling_job)\n",
+        "\n",
+        "request = clients[\"job\"].create_data_labeling_job(\n",
+        "    parent=PARENT, data_labeling_job=data_labeling_job\n",
+        ")\n",
+        "\n",
+        "print(request)\n",
+        "\n",
+        "labeling_task_name = request.name\n",
+        "\n",
+        "print(labeling_task_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "A3jRv70o_jAN"
+      },
+      "source": [
+        "### Get a data labeling job\n",
+        "\n",
+        "You can get information on your data labeling job using the `get_data_labeling_job()` method, with the following parameters:\n",
+        "\n",
+        "- `name`: The name of the labeling task."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "rPnMOftyq3jS"
+      },
+      "outputs": [],
+      "source": [
+        "request = clients[\"job\"].get_data_labeling_job(name=labeling_task_name)\n",
+        "print(request)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fQ0TVlokq3jS"
+      },
+      "source": [
+        "### Cancel a data labeling task\n",
+        "\n",
+        "You can cancel a data labeling request using the `cancel_data_labeling_job()` method, with the following parameters:\n",
+        "\n",
+        "- `name`: The name of the labeling task."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FZ3dlKJjq3jT"
+      },
+      "outputs": [],
+      "source": [
+        "request = clients[\"job\"].cancel_data_labeling_job(name=labeling_task_name)\n",
+        "print(request)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d37f8ede2327"
+      },
+      "source": [
+        "### Wait for labeling job to be canceled\n",
+        "\n",
+        "The cancel request is asyncrhonous. The code below polls on the labeling job status until the status is CANCELED."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "trainingpipelines_get:migration,new,wait"
+      },
+      "outputs": [],
+      "source": [
+        "while True:\n",
+        "    response = clients[\"job\"].get_data_labeling_job(name=labeling_task_name)\n",
+        "    if response.state == gapic.JobState.JOB_STATE_CANCELLED:\n",
+        "        print(\"Labeling job CANCELED\")\n",
+        "        break\n",
+        "    else:\n",
+        "        print(\"Canceling labeling job:\", response.state)\n",
+        "        time.sleep(60)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:migration,new"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all GCP resources used in this project, you can [delete the GCP\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aoJ18d8Y_jAy"
+      },
+      "outputs": [],
+      "source": [
+        "delete_dataset = True\n",
+        "delete_job = True\n",
+        "delete_specialist_pool = True\n",
+        "delete_bucket = True\n",
+        "\n",
+        "# Delete the dataset using the Vertex AI fully qualified identifier for the dataset\n",
+        "try:\n",
+        "    if delete_dataset:\n",
+        "        dataset.delete()\n",
+        "except Exception as e:\n",
+        "    print(e)\n",
+        "\n",
+        "# Delete the labeling job using the Vertex AI fully qualified identifier for the dataset\n",
+        "try:\n",
+        "    if delete_job:\n",
+        "        request = clients[\"job\"].delete_data_labeling_job(name=labeling_task_name)\n",
+        "except Exception as e:\n",
+        "    print(e)\n",
+        "\n",
+        "# Delete the specialist pool using the Vertex AI fully qualified identifier for the dataset\n",
+        "try:\n",
+        "    if delete_specialist_pool:\n",
+        "        clients[\"specialist_pool\"].delete_specialist_pool(name=specialist_name)\n",
+        "except Exception as e:\n",
+        "    print(e)\n",
+        "\n",
+        "\n",
+        "if delete_bucket and \"BUCKET_URI\" in globals():\n",
+        "    ! gsutil rm -r $BUCKET_URI"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [
+        "timestamp",
+        "gcp_authenticate",
+        "bucket:batch_prediction",
+        "setup_vars",
+        "import_aip",
+        "aip_constants",
+        "automl_constants:automl",
+        "datasets_create:migration,new",
+        "request:migration",
+        "call:migration",
+        "response:migration",
+        "datasets_import:migration,new",
+        "oJtL08_Q_jAF",
+        "rnhDF5vW_jAG",
+        "zQoFJ2K0_jAH",
+        "trainingpipelines_create:migration,new",
+        "m5igPySU_jAJ",
+        "xhuR86RL_jAK",
+        "S8zP7wju_jAL",
+        "trainingpipelines_get:migration,new",
+        "A3jRv70o_jAN",
+        "XC5I2xxt_jAN",
+        "models_evaluations_list:migration,new",
+        "Ngn6qqVy_jAQ",
+        "F0ryqI3F_jAQ",
+        "models_evaluations_get:migration,new",
+        "_NXujm2U_jAR",
+        "0RLTdCfj_jAS",
+        "make_batch_prediction_file:migration,new",
+        "make_batch_file:automl,image",
+        "batchpredictionjobs_create:migration,new",
+        "htIpycBi_jAX",
+        "8QO3y-36_jAY",
+        "DmClxRYK_jAY",
+        "batchpredictionjobs_get:migration,new",
+        "aSE_wqES_jAa",
+        "LUy0NIF__jAa",
+        "endpoints_create:migration,new",
+        "Ph5S0j4v_jAc",
+        "yjsSo1cM_jAd",
+        "ijvF_HGd_jAe",
+        "endpoints_deploymodel:migration,new",
+        "NFIRI0XT_jAf",
+        "c3_4BVyW_jAh",
+        "7NmySa8R_jAh",
+        "endpoints_predict:migration,new",
+        "6fb84nKh_jAk",
+        "h6IskqWe_jAo",
+        "KHf2BSMR_jAo",
+        "endpoints_undeploymodel:migration,new",
+        "KrVZz6Uw_jAp",
+        "wvFK-kir_jAq",
+        "5bwEQMKT_jAr",
+        "LOsxiKj4_jAs",
+        "PWMEUCbF_jAt",
+        "OalQ6m9P_jAu",
+        "models_export:migration,new",
+        "lqJoqYMI_jAv",
+        "v6isqzPQ_jAw",
+        "ZCyd1qAb_jAx"
+      ],
+      "name": "get_started_with_data_labeling.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/community/ml_ops/stage2/README.md

@@ -185,6 +185,21 @@ The steps performed include:
 - Train an AutoML model with CMEK encryption.
 ```
 
+[Get Started with TensorFlow Hub models](get_started_with_tfhub_models.ipynb)
+
+```
+The steps performed include:
+
+- Download a TensorFlow Hub prebuilt model.
+- Add the task component as a classifier for the CIFAR-10 dataset.
+- Fine tune locally the model with transfer learning training.
+- Construct a custom training script:
+    - Get training data from TensorFlow Datasets
+    - Get model architecture from TensorFlow Hub
+    - Train then model
+    - Save model artifacts and upload as Vertex AI Model resource.
+```
+
 
 ### E2E Stage Example
 

notebooks/community/ml_ops/stage2/get_started_automl_training.ipynb

@@ -65,9 +65,11 @@
         "id": "dataset:flowers,icn"
       },
       "source": [
-        "### Dataset\n",
+        "### Datasets\n",
         "\n",
-        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
+        "#### Image\n",
+        "\n",
+        "The image dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
       ]
     },
     {
@@ -76,9 +78,9 @@
         "id": "dataset:gsod,lrg"
       },
       "source": [
-        "### Dataset\n",
+        "#### Tabular\n",
         "\n",
-        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
+        "The tabular dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
       ]
     },
     {
@@ -87,9 +89,20 @@
         "id": "dataset:happydb,tcn"
       },
       "source": [
-        "### Dataset\n",
+        "#### Text\n",
         "\n",
-        "The dataset used for this tutorial is the [Happy Moments dataset](https://www.kaggle.com/ritresearch/happydb) from [Kaggle Datasets](https://www.kaggle.com/ritresearch/happydb). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
+        "The text dataset used for this tutorial is the [Happy Moments dataset](https://www.kaggle.com/ritresearch/happydb) from [Kaggle Datasets](https://www.kaggle.com/ritresearch/happydb). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "98eb93ec6faa"
+      },
+      "source": [
+        "#### Video\n",
+        "\n",
+        "The video dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset](https://todo) from [MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you will use in this tutorial is stored in a public #(GCS) bucket. The trained model will predict the start frame where a golf swing begins."
       ]
     },
     {
@@ -112,7 +125,8 @@
         "- Export the image model as an edge model.\n",
         "- Train a tabular model.\n",
         "- Export the tabular model as a cloud model.\n",
-        "- Train a text model."
+        "- Train a text model.\n",
+        "- Train a video model."
       ]
     },
     {
@@ -162,7 +176,11 @@
         "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
         "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
         "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
+        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
+        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
+        "    ! pip3 install --upgrade rpy2 $USER_FLAG\n",
+        "    ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
+        "    ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
       ]
     },
     {
@@ -320,7 +338,8 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
       ]
     },
     {
@@ -331,8 +350,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -352,7 +371,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -372,7 +391,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -417,7 +436,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -546,7 +565,7 @@
       "outputs": [],
       "source": [
         "dataset = aip.ImageDataset.create(\n",
-        "    display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
         "    gcs_source=[IMPORT_FILE],\n",
         "    import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
         ")\n",
@@ -594,7 +613,7 @@
       "outputs": [],
       "source": [
         "dag = aip.AutoMLImageTrainingJob(\n",
-        "    display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
         "    prediction_type=\"classification\",\n",
         "    multi_label=False,\n",
         "    model_type=\"MOBILE_TF_LOW_LATENCY_1\",\n",
@@ -637,7 +656,7 @@
       "source": [
         "model = dag.run(\n",
         "    dataset=dataset,\n",
-        "    model_display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    model_display_name=\"flowers_\" + TIMESTAMP,\n",
         "    training_fraction_split=0.8,\n",
         "    validation_fraction_split=0.1,\n",
         "    test_fraction_split=0.1,\n",
@@ -667,7 +686,7 @@
       "outputs": [],
       "source": [
         "# Get model resource ID\n",
-        "models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
+        "models = aip.Model.list(filter=\"display_name=flowers_\" + TIMESTAMP)\n",
         "\n",
         "# Get a reference to the Model Service client\n",
         "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
@@ -800,7 +819,7 @@
         "id": "undeploy_model:mbsdk"
       },
       "source": [
-        "## Undeploy the model\n",
+        "#### Undeploy the model\n",
         "\n",
         "When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
       ]
@@ -846,7 +865,7 @@
       "outputs": [],
       "source": [
         "response = model.export_model(\n",
-        "    artifact_destination=BUCKET_NAME, export_format_id=\"tflite\", sync=True\n",
+        "    artifact_destination=BUCKET_URI, export_format_id=\"tflite\", sync=True\n",
         ")\n",
         "\n",
         "model_package = response[\"artifactOutputUri\"]"
@@ -988,9 +1007,9 @@
       "outputs": [],
       "source": [
         "dataset = aip.TabularDataset.create(\n",
-        "    display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
+        "    display_name=\"gsod_\" + TIMESTAMP,\n",
         "    bq_source=[IMPORT_FILE],\n",
-        "    labels={\"user_metadata\": BUCKET_NAME[5:]},\n",
+        "    labels={\"user_metadata\": BUCKET_NAME},\n",
         ")\n",
         "\n",
         "label_column = \"mean_temp\"\n",
@@ -1060,7 +1079,7 @@
       "outputs": [],
       "source": [
         "dag = aip.AutoMLTabularTrainingJob(\n",
-        "    display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    display_name=\"gsod_\" + TIMESTAMP,\n",
         "    optimization_prediction_type=\"regression\",\n",
         "    optimization_objective=\"minimize-rmse\",\n",
         "    column_transformations=TRANSFORMATIONS,\n",
@@ -1103,7 +1122,7 @@
       "source": [
         "model = dag.run(\n",
         "    dataset=dataset,\n",
-        "    model_display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    model_display_name=\"gsod_\" + TIMESTAMP,\n",
         "    training_fraction_split=0.8,\n",
         "    validation_fraction_split=0.1,\n",
         "    test_fraction_split=0.1,\n",
@@ -1134,7 +1153,7 @@
       "outputs": [],
       "source": [
         "# Get model resource ID\n",
-        "models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
+        "models = aip.Model.list(filter=\"display_name=gsod_\" + TIMESTAMP)\n",
         "\n",
         "# Get a reference to the Model Service client\n",
         "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
@@ -1177,7 +1196,7 @@
         "id": "undeploy_model:mbsdk"
       },
       "source": [
-        "## Undeploy the model\n",
+        "#### Undeploy the model\n",
         "\n",
         "When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
       ]
@@ -1218,7 +1237,7 @@
       "outputs": [],
       "source": [
         "response = model.export_model(\n",
-        "    artifact_destination=BUCKET_NAME, export_format_id=\"tf-saved-model\", sync=True\n",
+        "    artifact_destination=BUCKET_URI, export_format_id=\"tf-saved-model\", sync=True\n",
         ")\n",
         "\n",
         "model_package = response[\"artifactOutputUri\"]"
@@ -1392,7 +1411,7 @@
       "outputs": [],
       "source": [
         "dataset = aip.TextDataset.create(\n",
-        "    display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
+        "    display_name=\"happydb_\" + TIMESTAMP,\n",
         "    gcs_source=[IMPORT_FILE],\n",
         "    import_schema_uri=aip.schema.dataset.ioformat.text.single_label_classification,\n",
         ")\n",
@@ -1542,7 +1561,7 @@
         "id": "undeploy_model:mbsdk"
       },
       "source": [
-        "## Undeploy the model\n",
+        "#### Undeploy the model\n",
         "\n",
         "When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
       ]
@@ -1727,7 +1746,7 @@
       "outputs": [],
       "source": [
         "dataset = aip.VideoDataset.create(\n",
-        "    display_name=\"Happy Moments\" + \"_\" + TIMESTAMP,\n",
+        "    display_name=\"human_motion_\" + TIMESTAMP,\n",
         "    gcs_source=[IMPORT_FILE],\n",
         "    import_schema_uri=aip.schema.dataset.ioformat.video.classification,\n",
         ")\n",
@@ -1767,7 +1786,7 @@
       "outputs": [],
       "source": [
         "dag = aip.AutoMLVideoTrainingJob(\n",
-        "    display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    display_name=\"human_motion_\" + TIMESTAMP,\n",
         "    prediction_type=\"classification\",\n",
         ")\n",
         "\n",
@@ -1804,7 +1823,7 @@
       "source": [
         "model = dag.run(\n",
         "    dataset=dataset,\n",
-        "    model_display_name=\"happydb_\" + TIMESTAMP,\n",
+        "    model_display_name=\"human_motion_\" + TIMESTAMP,\n",
         "    training_fraction_split=0.8,\n",
         "    test_fraction_split=0.2,\n",
         ")"
@@ -1831,7 +1850,7 @@
       "outputs": [],
       "source": [
         "# Get model resource ID\n",
-        "models = aip.Model.list(filter=\"display_name=happydb_\" + TIMESTAMP)\n",
+        "models = aip.Model.list(filter=\"display_name=human_motion_\" + TIMESTAMP)\n",
         "\n",
         "# Get a reference to the Model Service client\n",
         "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
@@ -1899,16 +1918,7 @@
         "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
         "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
         "\n",
-        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
-        "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "Otherwise, you can delete the individual resources you created in this tutorial.\n"
       ]
     },
     {
@@ -1919,66 +1929,10 @@
       },
       "outputs": [],
       "source": [
-        "delete_dataset = True\n",
-        "delete_pipeline = True\n",
-        "delete_model = True\n",
-        "delete_endpoint = True\n",
-        "delete_batchjob = True\n",
-        "delete_customjob = True\n",
-        "delete_hptjob = True\n",
         "delete_bucket = True\n",
         "\n",
-        "# Delete the dataset using the Vertex fully qualified identifier for the dataset\n",
-        "try:\n",
-        "    if delete_dataset and \"dataset_id\" in globals():\n",
-        "        clients[\"dataset\"].delete_dataset(name=dataset_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n",
-        "try:\n",
-        "    if delete_pipeline and \"pipeline_id\" in globals():\n",
-        "        clients[\"pipeline\"].delete_training_pipeline(name=pipeline_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the model using the Vertex fully qualified identifier for the model\n",
-        "try:\n",
-        "    if delete_model and \"model_to_deploy_id\" in globals():\n",
-        "        clients[\"model\"].delete_model(name=model_to_deploy_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n",
-        "try:\n",
-        "    if delete_endpoint and \"endpoint_id\" in globals():\n",
-        "        clients[\"endpoint\"].delete_endpoint(name=endpoint_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the batch job using the Vertex fully qualified identifier for the batch job\n",
-        "try:\n",
-        "    if delete_batchjob and \"batch_job_id\" in globals():\n",
-        "        clients[\"job\"].delete_batch_prediction_job(name=batch_job_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the custom job using the Vertex fully qualified identifier for the custom job\n",
-        "try:\n",
-        "    if delete_customjob and \"job_id\" in globals():\n",
-        "        clients[\"job\"].delete_custom_job(name=job_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n",
-        "try:\n",
-        "    if delete_hptjob and \"hpt_job_id\" in globals():\n",
-        "        clients[\"job\"].delete_hyperparameter_tuning_job(name=hpt_job_id)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
-        "    ! gsutil rm -r $BUCKET_NAME"
+        "if delete_bucket and \"BUCKET_URI\" in globals():\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage2/get_started_vertex_distributed_training.ipynb

@@ -767,6 +767,13 @@
         "    strategy = tf.distribute.MultiWorkerMirroredStrategy()\n",
         "    logging.info(\"Multi-worker Strategy distributed training\")\n",
         "    logging.info('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n",
+        "    # Single Machine, multiple TPU devices\n",
+        "elif args.distribute == 'tpu':\n",
+        "    cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=\"local\")\n",
+        "    tf.config.experimental_connect_to_cluster(cluster_resolver)\n",
+        "    tf.tpu.experimental.initialize_tpu_system(cluster_resolver)\n",
+        "    strategy = tf.distribute.TPUStrategy(cluster_resolver)\n",
+        "    print(\"All devices: \", tf.config.list_logical_devices('TPU'))\n",
         "\n",
         "logging.info('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n",
         "\n",
@@ -825,8 +832,11 @@
         "    else:\n",
         "        task_type, task_id = None, None\n",
         "\n",
+        "    if args.distribute==\"tpu\":\n",
+        "        save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n",
+        "        model.save(args.model_dir, options=save_locally)\n",
         "    # single, mirrored or primary for multiworker\n",
-        "    if _is_chief(task_type, task_id):\n",
+        "    elif _is_chief(task_type, task_id):\n",
         "        model.save(args.model_dir)\n",
         "    # non-primary workers for multi-workers\n",
         "    else:\n",
@@ -1182,7 +1192,6 @@
         "%%writefile custom/Dockerfile\n",
         "\n",
         "FROM gcr.io/deeplearning-platform-release/tf2-gpu.2-5\n",
-        "WORKDIR /root\n",
         "\n",
         "WORKDIR /\n",
         "\n",
@@ -1590,7 +1599,7 @@
         "%%writefile custom/Dockerfile\n",
         "FROM python:3.8\n",
         "\n",
-        "WORKDIR /root\n",
+        "WORKDIR /\n",
         "\n",
         "# Copies the trainer code to the docker image.\n",
         "COPY trainer /trainer\n",

notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -29,7 +29,7 @@
         "id": "title:generic,gcp"
       },
       "source": [
-        "# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Logging and Vertex Experiments\n",
+        "# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Logging and Vertex AI Experiments\n",
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
@@ -39,8 +39,9 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_experiments.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -56,7 +57,7 @@
         "## Overview\n",
         "\n",
         "\n",
-        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Logging and Vertex Experiments."
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Logging and Vertex AI Experiments."
       ]
     },
     {
@@ -93,7 +94,7 @@
       "source": [
         "### Recommendations\n",
         "\n",
-        "When doing E2E MLOps on Google Cloud, the following best practices for logging data when experimenting or formal training a model.\n",
+        "When doing E2E MLOps on Google Cloud, the following are some of the best practices for logging data when experimenting or formally training a model.\n",
         "\n",
         "#### Python Logging\n",
         "\n",
@@ -105,7 +106,14 @@
         "\n",
         "#### Experiments\n",
         "\n",
-        "Use Vertex AI Experiments in conjunction with logging when doing experiments to compare results for different experiment configurations."
+        "Use Vertex AI Experiments in conjunction with logging when performing experiments to compare results for different experiment configurations.\n",
+        "\n",
+        "### Costs\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "- Vertex AI\n",
+        "\n",
+        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
       ]
     },
     {
@@ -116,7 +124,7 @@
       "source": [
         "## Installations\n",
         "\n",
-        "Install *one time* the packages for executing the MLOps notebooks."
+        "Install the following packages for executing this notebook."
       ]
     },
     {
@@ -127,20 +135,17 @@
       },
       "outputs": [],
       "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-logging $USER_FLAG"
       ]
     },
     {
@@ -248,7 +253,10 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -284,7 +292,7 @@
         "### Set up variables\n",
         "\n",
         "Next, set up some variables used throughout the tutorial.\n",
-        "### Import libraries and define constants"
+        "### Import libraries"
       ]
     },
     {
@@ -295,29 +303,9 @@
       },
       "outputs": [],
       "source": [
-        "import google.cloud.aiplatform as aip"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_logging"
-      },
-      "source": [
-        "#### Import logging\n",
+        "import logging\n",
         "\n",
-        "Import the logging package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_logging"
-      },
-      "outputs": [],
-      "source": [
-        "import logging"
+        "import google.cloud.aiplatform as aiplatform"
       ]
     },
     {
@@ -339,7 +327,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, location=REGION)"
+        "aiplatform.init(project=PROJECT_ID, location=REGION)"
       ]
     },
     {
@@ -356,9 +344,9 @@
         "- Send log output to console.\n",
         "- Send log output to a file.\n",
         "\n",
-        "### Logging Levels\n",
+        "### Logging Levels in Python Logging\n",
         "\n",
-        "The logging levels in order (from least to highest) are, with each level inclusive of the previous level:\n",
+        "The logging levels in order (from least to highest) and each level inclusive of the previous level are :\n",
         "\n",
         "1. Informational\n",
         "2. Warnings\n",
@@ -398,7 +386,7 @@
       "source": [
         "### Setting logging level\n",
         "\n",
-        "To set the logging level, you get the logging handler using `getLogger()`. You can have multiple logging handles. When `getLogger()` is called w/o arguments it gets the default handler, named ROOT. With the handler, you set the logging level with the method 'setLevel()`."
+        "To set the logging level, you get the logging handler using `getLogger()`. You can have multiple logging handles. When `getLogger()` is called without any arguments, it gets the default handler named ROOT. With the handler, you set the logging level with the method `setLevel()`."
       ]
     },
     {
@@ -445,7 +433,7 @@
       "source": [
         "### Output to a local file\n",
         "\n",
-        "You can preserve your logging output to a file that is local to where the Python script is running with the method `BasicConfig()`, with the following paraneters:\n",
+        "You can preserve your logging output to a file that is local to where the Python script is running with the method `BasicConfig()`, that takes the following parameters:\n",
         "\n",
         "- `filename`: The file path to the local file to write the log output to.\n",
         "- `level`: Sets the level of logging that is written to the logging file.\n",
@@ -482,7 +470,7 @@
         "- Send log output to storage.\n",
         "- Retrieve log output from storage.\n",
         "\n",
-        "### Logging Levels\n",
+        "### Logging Levels in Cloud Logging\n",
         "\n",
         "The logging levels in order (from least to highest) are, with each level inclusive of the previous level:\n",
         "\n",
@@ -539,7 +527,7 @@
       "source": [
         "### Logging output\n",
         "\n",
-        "To log output at specific levels is identical in method, and method names, as in Python logging, except that you use your instance of the cloud logger in place of logging."
+        "Logging output at specific levels is identical to Python logging with respect to method and method names. The only difference is that you use your instance of the cloud logger in place of logging."
       ]
     },
     {
@@ -567,7 +555,7 @@
         "To get the logged output, you:\n",
         "\n",
         "1. Retrieve the log handle to the service.\n",
-        "2. Using the handle call the method `list_entries()`\n",
+        "2. Using the handle, call the method `list_entries()`.\n",
         "3. Iterate through the entries."
       ]
     },
@@ -594,10 +582,10 @@
       "source": [
         "## Logging with Vertex AI Experiments and Vertex AI ML Metadata\n",
         "\n",
-        "You can log results related to training experiments with `Vertex AI Experiments` and `ML Metadata`:\n",
+        "You can log results related to training experiments with `Vertex AI Experiments` and `ML Metadata` including:\n",
         "\n",
         "- Preserve results of an experiment.\n",
-        "- Track multiple runs -- i.e., training runs -- within an experiment.\n",
+        "- Track multiple runs i.e., training runs within an experiment.\n",
         "- Track parameters (configuration) and metrics (results).\n",
         "- Retrieve and display the logged output.\n",
         "\n",
@@ -612,14 +600,29 @@
       "source": [
         "### Create experiment for tracking training related metadata\n",
         "\n",
-        "Setup tracking the parameters (configuration) and metrics (results) for each experiment:\n",
+        "Setup tracking for parameters (configuration) and metrics (results) in each experiment:\n",
         "\n",
-        "- `aip.init()` - Create an experiment instance\n",
-        "- `aip.start_run()` - Track a specific run within the experiment.\n",
+        "- `aiplatform.init()` - Create an experiment instance\n",
+        "- `aiplatform.start_run()` - Track a specific run within the experiment.\n",
         "\n",
         "Learn more about [Introduction to Vertex AI ML Metadata](https://cloud.google.com/vertex-ai/docs/ml-metadata/introduction)."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bd7fb247cbae"
+      },
+      "outputs": [],
+      "source": [
+        "# Specify a name for the experiment\n",
+        "EXPERIMENT_NAME = \"[your-experiment-name]\"\n",
+        "\n",
+        "if EXPERIMENT_NAME == \"[your-experiment-name]\":\n",
+        "    EXPERIMENT_NAME = \"example-\" + TIMESTAMP"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -628,9 +631,9 @@
       },
       "outputs": [],
       "source": [
-        "EXPERIMENT_NAME = \"example-\" + TIMESTAMP\n",
-        "aip.init(experiment=EXPERIMENT_NAME)\n",
-        "aip.start_run(\"run-1\")"
+        "# Create experiment\n",
+        "aiplatform.init(experiment=EXPERIMENT_NAME)\n",
+        "aiplatform.start_run(\"run-1\")"
       ]
     },
     {
@@ -641,14 +644,14 @@
       "source": [
         "### Log parameters for the experiment\n",
         "\n",
-        "Typically, an experiment is associated with a specific dataset and model architecture. Within an experiment, you may have multiple training runs, where each run tries a different configuration. As examples:\n",
+        "Typically, an experiment is associated with a specific dataset and a model architecture. Within an experiment, you may have multiple training runs, where each run tries a different configuration. For example:\n",
         "\n",
         "- Dataset split\n",
         "- Dataset sampling and boosting\n",
         "- Depth and width of layers\n",
         "- Hyperparameters\n",
         "\n",
-        "These configuration settings are referred to as parameters, which you store their key/value pair using the method `log_params()`"
+        "These configuration settings are referred to as parameters, which you store as key-value pairs using the method `log_params()`"
       ]
     },
     {
@@ -663,7 +666,7 @@
         "hyperparams[\"epochs\"] = 100\n",
         "hyperparams[\"batch_size\"] = 32\n",
         "hyperparams[\"learning_rate\"] = 0.01\n",
-        "aip.log_params(hyperparams)"
+        "aiplatform.log_params(hyperparams)"
       ]
     },
     {
@@ -674,14 +677,14 @@
       "source": [
         "### Log metrics for the experiment\n",
         "\n",
-        "At the completion, or termination, of a run within an experiment, you can log results that you use to compare runs. As examples:\n",
+        "At the completion or termination of a run within an experiment, you can log results that you use to compare runs. For example:\n",
         "\n",
         "- Evaluation metrics\n",
         "- Hyperparameter search selection\n",
         "- Time to train the model\n",
         "- Early stop trigger\n",
         "\n",
-        "These results settings are referred to as metrics, which you store their key/value pair using the method `log_metrics()`"
+        "These results are referred to as metrics, which you store as key-value pairs using the method `log_metrics()`"
       ]
     },
     {
@@ -695,7 +698,7 @@
         "metrics = {}\n",
         "metrics[\"test_acc\"] = 98.7\n",
         "metrics[\"train_acc\"] = 99.3\n",
-        "aip.log_metrics(metrics)"
+        "aiplatform.log_metrics(metrics)"
       ]
     },
     {
@@ -717,36 +720,11 @@
       },
       "outputs": [],
       "source": [
-        "EXPERIMENT_NAME = \"example\"\n",
-        "\n",
-        "experiment_df = aip.get_experiment_df()\n",
+        "experiment_df = aiplatform.get_experiment_df()\n",
         "experiment_df = experiment_df[experiment_df.experiment_name == EXPERIMENT_NAME]\n",
         "experiment_df.T"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "delete_experiment"
-      },
-      "source": [
-        "### Delete the experiment\n",
-        "\n",
-        "Next, delete the experiment. You will need to get the context via the metadata to delete it."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "delete_experiment"
-      },
-      "outputs": [],
-      "source": [
-        "c = aiplatform.metadata._Context(EXPERIMENT_NAME)\n",
-        "c.delete()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -760,15 +738,9 @@
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "### Delete the experiment\n",
+        "\n",
+        "Next, delete the experiment. You will need to get the context via the metadata to delete it."
       ]
     },
     {
@@ -779,61 +751,8 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
-        "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "c = aiplatform.metadata._Context(EXPERIMENT_NAME)\n",
+        "c.delete()"
       ]
     }
   ],

notebooks/community/ml_ops/stage2/get_started_vertex_training_pytorch.ipynb

@@ -39,8 +39,9 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_pytorch.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -82,8 +83,9 @@
         "\n",
         "This tutorial uses the following Google Cloud ML services:\n",
         "\n",
-        "- `Vertex AI Training`\n",
-        "- `Vertex AI Model` resource\n",
+        "* `Vertex AI Training`\n",
+        "* `Vertex AI Model` resource\n",
+        "\n",
         "\n",
         "The steps performed include:\n",
         "\n",
@@ -93,6 +95,75 @@
         "- Create a `Vertex AI Model` resource."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "85ee859437ed"
+      },
+      "source": [
+        "## Costs \n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5cd61a5dd9db"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
+        "all the requirements to run this notebook. You can skip this step."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7e689ee0bc3c"
+      },
+      "source": [
+        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
+        "You need the following:\n",
+        "\n",
+        "* The Google Cloud SDK\n",
+        "* Git\n",
+        "* Python 3\n",
+        "* virtualenv\n",
+        "* Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Google Cloud guide to [Setting up a Python development\n",
+        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
+        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
+        "for meeting these requirements. The following steps provide a condensed set of\n",
+        "instructions:\n",
+        "\n",
+        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
+        "\n",
+        "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
+        "\n",
+        "1. [Install\n",
+        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
+        "   and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
+        "\n",
+        "1. To install Jupyter, run `pip3 install jupyter` on the\n",
+        "command-line in a terminal shell.\n",
+        "\n",
+        "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
+        "\n",
+        "1. Open this notebook in the Jupyter Notebook Dashboard."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -112,22 +183,10 @@
       },
       "outputs": [],
       "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
-        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
-        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
+        "! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
+        "! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
+        "! pip3 install --upgrade torchvision $USER_FLAG"
       ]
     },
     {
@@ -285,7 +344,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -296,8 +355,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -317,7 +376,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -337,7 +396,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -360,7 +419,7 @@
       },
       "outputs": [],
       "source": [
-        "import google.cloud.aiplatform as aip"
+        "import google.cloud.aiplatform as aiplatform"
       ]
     },
     {
@@ -382,7 +441,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -412,13 +471,15 @@
       },
       "outputs": [],
       "source": [
+        "import os\n",
+        "\n",
         "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
         "    TRAIN_GPU, TRAIN_NGPU = (\n",
-        "        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
+        "        aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
         "        int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
         "    )\n",
         "else:\n",
-        "    TRAIN_GPU, TRAIN_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
+        "    TRAIN_GPU, TRAIN_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
       ]
     },
     {
@@ -561,6 +622,7 @@
         "# Add package information\n",
         "! touch custom/README.md\n",
         "\n",
+        "# Instructions for installing package into environment of the docker image\n",
         "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n",
         "! echo \"$setup_cfg\" > custom/setup.cfg\n",
         "\n",
@@ -891,7 +953,7 @@
         "! rm -f custom.tar custom.tar.gz\n",
         "! tar cvf custom.tar custom\n",
         "! gzip custom.tar\n",
-        "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
+        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_cifar10.tar.gz"
       ]
     },
     {
@@ -902,7 +964,7 @@
       "source": [
         "### Make Pytorch container for prediction\n",
         "\n",
-        "Currently, Vertex AI does not have a prefined container for making predictions with a deployed Pytorch model. No problem, you can assemble your own custom container. Typically, one would base the container on the `Torch Server`. For demonstration purpose, you build a placeholder container (not complete) that includes the latest `Torch Server` image, and push it to the `Container Registry`."
+        "Currently, Vertex AI does not have a predefined container for making predictions with a deployed Pytorch model. No problem, you can assemble your own custom container. Typically, one would base the container on the `Torch Server`. For demonstration purpose, you build a placeholder container (not complete) that includes the latest `Torch Server` image, and push it to the `Container Registry`."
       ]
     },
     {
@@ -974,9 +1036,9 @@
       "source": [
         "DISPLAY_NAME = \"cifar10_\" + TIMESTAMP\n",
         "\n",
-        "job = aip.CustomPythonPackageTrainingJob(\n",
+        "job = aiplatform.CustomPythonPackageTrainingJob(\n",
         "    display_name=DISPLAY_NAME,\n",
-        "    python_package_gcs_uri=f\"{BUCKET_NAME}/trainer_cifar10.tar.gz\",\n",
+        "    python_package_gcs_uri=f\"{BUCKET_URI}/trainer_cifar10.tar.gz\",\n",
         "    python_module_name=\"trainer.task\",\n",
         "    container_uri=TRAIN_IMAGE,\n",
         "    model_serving_container_image_uri=DEPLOY_IMAGE,\n",
@@ -1009,7 +1071,7 @@
       },
       "outputs": [],
       "source": [
-        "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
+        "MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
         "\n",
         "DIRECT = False\n",
         "if DIRECT:\n",
@@ -1121,7 +1183,7 @@
       "source": [
         "### Delete a custom training job\n",
         "\n",
-        "After a training job is completed, you can delete the training job with the method `delete()`.  Prior to completion, a training job can be canceled with the method `cancel()`."
+        "After a training job is completed, you can delete the training job with the method `delete()`.  Prior to completion, a training job can be cancelled with the method `cancel()`."
       ]
     },
     {
@@ -1148,14 +1210,8 @@
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
         "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
+        "- Custom Job (Custom job deleted in previous cell)\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -1167,61 +1223,12 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
+        "# Delete the model using the Vertex model object\n",
+        "model.delete()\n",
         "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
         "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage2/get_started_vertex_training_r.ipynb

@@ -29,7 +29,7 @@
         "id": "title:generic,gcp"
       },
       "source": [
-        "# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex Training for R\n",
+        "# E2E ML on GCP: MLOps stage 2 : experimentation: get started with Vertex AI Training for R\n",
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
@@ -39,8 +39,9 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_r.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_r.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -56,7 +57,7 @@
         "## Overview\n",
         "\n",
         "\n",
-        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex Training for R."
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with Vertex AI Training for R. Please note that this notebook should be ran only in R notebook image (e.g., R4.1)."
       ]
     },
     {
@@ -98,6 +99,28 @@
         "- Train a R model using `Vertex AI Trainingh` service with the R-to-Python training package."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0c997d8d92ce"
+      },
+      "source": [
+        "### Costs \n",
+        "\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -109,6 +132,25 @@
         "Install *one time* the packages for executing the MLOps notebooks."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1fd00fa70a2a"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\""
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -117,22 +159,8 @@
       },
       "outputs": [],
       "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
-        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
-        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
+        "! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "! pip3 install --upgrade rpy2 $USER_FLAG"
       ]
     },
     {
@@ -164,6 +192,39 @@
         "    app.kernel.do_shutdown(True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0e3cab0cc491"
+      },
+      "source": [
+        "## Before you begin"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "be929e7b4d76"
+      },
+      "source": [
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
+        "\n",
+        "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
+        "\n",
+        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
+        "\n",
+        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -183,6 +244,8 @@
       },
       "outputs": [],
       "source": [
+        "import os\n",
+        "\n",
         "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
       ]
     },
@@ -240,7 +303,9 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type:\"string\"}\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -277,7 +342,7 @@
         "\n",
         "**The following steps are required, regardless of your notebook environment.**\n",
         "\n",
-        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
         "\n",
         "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
       ]
@@ -290,7 +355,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -301,8 +366,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -322,7 +387,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -342,7 +407,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -365,6 +430,8 @@
       },
       "outputs": [],
       "source": [
+        "import traceback\n",
+        "\n",
         "import google.cloud.aiplatform as aip"
       ]
     },
@@ -387,7 +454,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -964,14 +1031,17 @@
       },
       "outputs": [],
       "source": [
-        "INSTANCES = [\n",
-        "    {\"sepal_width\": 1, \"sepal_length\": 2, \"petal_width\": 3, \"petal_length\": 1},\n",
-        "    {\"sepal_width\": 4, \"sepal_length\": 2, \"petal_width\": 1, \"petal_length\": 1},\n",
-        "]\n",
+        "try:\n",
+        "    INSTANCES = [\n",
+        "        {\"sepal_width\": 1, \"sepal_length\": 2, \"petal_width\": 3, \"petal_length\": 1},\n",
+        "        {\"sepal_width\": 4, \"sepal_length\": 2, \"petal_width\": 1, \"petal_length\": 1},\n",
+        "    ]\n",
         "\n",
-        "prediction = endpoint.predict(instances=INSTANCES)\n",
+        "    prediction = endpoint.predict(instances=INSTANCES)\n",
         "\n",
-        "print(prediction)"
+        "    print(prediction)\n",
+        "except:\n",
+        "    traceback.print_exc()"
       ]
     },
     {
@@ -1242,7 +1312,7 @@
       },
       "outputs": [],
       "source": [
-        "CMDARGS = [\"--model-dir=\" + BUCKET_NAME]\n",
+        "CMDARGS = [\"--model-dir=\" + BUCKET_URI]\n",
         "\n",
         "job.run(args=CMDARGS, replica_count=1, machine_type=TRAIN_COMPUTE, sync=True)"
       ]
@@ -1282,14 +1352,9 @@
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
+        "- Model (Already deleted in previous cells)\n",
+        "- Endpoint (Already deleted in previous cells)\n",
+        "- Custom Job (Already deleted in previous cells)\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -1301,61 +1366,8 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
-        "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb

@@ -40,7 +40,11 @@
         "  </td>\n",
         "  <td>\n",
         "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_training_sklearn.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "     <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -101,62 +105,7 @@
       "source": [
         "## Installations\n",
         "\n",
-        "Install *one time* the packages for executing the MLOps notebooks."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "install_mlops"
-      },
-      "outputs": [],
-      "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
-        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
-        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "restart"
-      },
-      "source": [
-        "### Restart the kernel\n",
-        "\n",
-        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "restart"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "if not os.getenv(\"IS_TESTING\"):\n",
-        "    # Automatically restart kernel after installs\n",
-        "    import IPython\n",
-        "\n",
-        "    app = IPython.Application.instance()\n",
-        "    app.kernel.do_shutdown(True)"
+        "You will not need special packages for this notebook."
       ]
     },
     {
@@ -285,7 +234,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -296,8 +245,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -317,7 +266,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -337,7 +286,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -382,7 +331,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -415,6 +364,8 @@
       },
       "outputs": [],
       "source": [
+        "import os\n",
+        "\n",
         "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
         "    TRAIN_GPU, TRAIN_NGPU = (\n",
         "        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
@@ -783,7 +734,7 @@
         "! rm -f custom.tar custom.tar.gz\n",
         "! tar cvf custom.tar custom\n",
         "! gzip custom.tar\n",
-        "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_newsaggr.tar.gz"
+        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_newsaggr.tar.gz"
       ]
     },
     {
@@ -823,7 +774,7 @@
         "\n",
         "job = aip.CustomPythonPackageTrainingJob(\n",
         "    display_name=DISPLAY_NAME,\n",
-        "    python_package_gcs_uri=f\"{BUCKET_NAME}/trainer_newsaggr.tar.gz\",\n",
+        "    python_package_gcs_uri=f\"{BUCKET_URI}/trainer_newsaggr.tar.gz\",\n",
         "    python_module_name=\"trainer.task\",\n",
         "    container_uri=TRAIN_IMAGE,\n",
         "    model_serving_container_image_uri=DEPLOY_IMAGE,\n",
@@ -857,7 +808,7 @@
       },
       "outputs": [],
       "source": [
-        "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
+        "MODEL_DIR = \"{}/{}\".format(BUCKET_URI, TIMESTAMP)\n",
         "DATASET_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/00359/NewsAggregatorDataset.zip\"\n",
         "\n",
         "DIRECT = False\n",
@@ -1002,14 +953,9 @@
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
+        "\n",
         "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
+        "- Custom Job (already deleted in previous cell)\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -1017,65 +963,15 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "cleanup:mbsdk"
+        "id": "b413063dfdcf"
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
+        "# Delete the model using the Vertex model object\n",
+        "model.delete()\n",
         "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage3/README.md

@@ -66,6 +66,17 @@ The steps performed include:
 - Execute a Vertex AI pipeline.
 ```
 
+[Get Started with Dataproc components](get_started_with_dataproc_pipeline_components.ipynb)
+
+```
+The steps performed include:
+
+- DataprocPySparkBatchOp for PySpark batch workloads.
+- DataprocSparkBatchOp for Spark batch workloads.
+- DataprocSparkSqlBatchOp for running Spark SQL batch workloads.
+- DataprocSparkRBatchOp for running SparkR batch workloads.
+```
+
 [Get Started with Vertex AI AutoML components](get_started_with_automl_pipeline_components.ipynb)
 
 ```

notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb

@@ -40,7 +40,7 @@
         "  </td>\n",
         "  <td>\n",
         "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -291,7 +291,8 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
       ]
     },
     {
@@ -302,8 +303,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -323,7 +324,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -343,7 +344,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -406,9 +407,9 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
         "\n",
-        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
       ]
     },
     {
@@ -490,7 +491,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -612,9 +613,7 @@
       },
       "outputs": [],
       "source": [
-        "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
-        "\n",
-        "PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_NAME)\n",
+        "PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_URI)\n",
         "DEPLOY_COMPUTE = \"n1-standard-4\"\n",
         "\n",
         "\n",
@@ -629,6 +628,7 @@
         "    project: str = PROJECT_ID,\n",
         "    region: str = REGION,\n",
         "):\n",
+        "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
         "\n",
         "    dataset_op = gcc_aip.ImageDatasetCreateOp(\n",
         "        project=project,\n",
@@ -677,6 +677,7 @@
         "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
         "        automatic_resources_min_replica_count=1,\n",
         "        automatic_resources_max_replica_count=1,\n",
+        "        traffic_split={\"0\": 100},\n",
         "    )"
       ]
     },
@@ -688,7 +689,7 @@
       "source": [
         "### Get test item(s)\n",
         "\n",
-        "Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
+        "In the pipeline, you do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
       ]
     },
     {
@@ -733,11 +734,11 @@
         "file_1 = test_item_1.split(\"/\")[-1]\n",
         "file_2 = test_item_2.split(\"/\")[-1]\n",
         "\n",
-        "! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
-        "! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
+        "! gsutil cp $test_item_1 $BUCKET_URI/$file_1\n",
+        "! gsutil cp $test_item_2 $BUCKET_URI/$file_2\n",
         "\n",
-        "test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
-        "test_item_2 = BUCKET_NAME + \"/\" + file_2"
+        "test_item_1 = BUCKET_URI + \"/\" + file_1\n",
+        "test_item_2 = BUCKET_URI + \"/\" + file_2"
       ]
     },
     {
@@ -770,7 +771,7 @@
         "\n",
         "import tensorflow as tf\n",
         "\n",
-        "gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
+        "gcs_input_uri = BUCKET_URI + \"/test.jsonl\"\n",
         "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
         "    data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
         "    f.write(json.dumps(data) + \"\\n\")\n",
@@ -911,9 +912,21 @@
         "print(\"automl-image-training-job\")\n",
         "artifacts = print_pipeline_output(pipeline, \"automl-image-training-job\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
+        "print(\"\\n\")\n",
+        "print(endpoint_id)\n",
         "print(\"endpoint-create\")\n",
         "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
+        "    \"resourceName\"\n",
+        "]\n",
+        "print(\"\\n\")\n",
+        "print(endpoint_id)\n",
         "print(\"model-deploy\")\n",
         "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
         "print(\"\\n\\n\")\n",
@@ -929,7 +942,12 @@
         "    output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
         "        \"gcsOutputDirectory\"\n",
         "    ]\n",
-        ")"
+        ")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "batch_job_id = output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
+        "    \"resourceName\"\n",
+        "]"
       ]
     },
     {
@@ -954,6 +972,149 @@
         "pipeline.delete()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "endpoint_load:mbsdk"
+      },
+      "source": [
+        "#### Load an endpoint\n",
+        "\n",
+        "The 'Endpoint' initializer will load an endpoint from an endpoint identifier."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "endpoint_load:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = aip.Endpoint(endpoint_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "make_prediction"
+      },
+      "source": [
+        "## Send a online prediction request\n",
+        "\n",
+        "Send a online prediction to your deployed model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "get_test_item"
+      },
+      "source": [
+        "### Get test item\n",
+        "\n",
+        "You will use an arbitrary example out of the dataset as a test item. Don't be concerned that the example was likely used in training the model -- we just want to demonstrate how to make a prediction."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "get_test_item:automl,icn,csv"
+      },
+      "outputs": [],
+      "source": [
+        "test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
+        "if len(str(test_item[0]).split(\",\")) == 3:\n",
+        "    _, test_item, test_label = str(test_item[0]).split(\",\")\n",
+        "else:\n",
+        "    test_item, test_label = str(test_item[0]).split(\",\")\n",
+        "\n",
+        "print(test_item, test_label)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "predict_request:mbsdk,icn"
+      },
+      "source": [
+        "### Make the prediction\n",
+        "\n",
+        "Now that your `Model` resource is deployed to an `Endpoint` resource, you can do online predictions by sending prediction requests to the Endpoint resource.\n",
+        "\n",
+        "#### Request\n",
+        "\n",
+        "Since in this example your test item is in a Cloud Storage bucket, you open and read the contents of the image using `tf.io.gfile.Gfile()`. To pass the test data to the prediction service, you encode the bytes into base64 -- which makes the content safe from modification while transmitting binary data over the network.\n",
+        "\n",
+        "The format of each instance is:\n",
+        "\n",
+        "    { 'content': { 'b64': base64_encoded_bytes } }\n",
+        "\n",
+        "Since the `predict()` method can take multiple items (instances), send your single test item as a list of one test item.\n",
+        "\n",
+        "#### Response\n",
+        "\n",
+        "The response from the `predict()` call is a Python dictionary with the following entries:\n",
+        "\n",
+        "- `ids`: The internal assigned unique identifiers for each prediction request.\n",
+        "- `displayNames`: The class names for each class label.\n",
+        "- `confidences`: The predicted confidence, between 0 and 1, per class label.\n",
+        "- `deployed_model_id`: The Vertex AI identifier for the deployed Model resource which did the predictions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "predict_request:mbsdk,icn"
+      },
+      "outputs": [],
+      "source": [
+        "import base64\n",
+        "\n",
+        "import tensorflow as tf\n",
+        "\n",
+        "with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
+        "    content = f.read()\n",
+        "\n",
+        "# The format of each instance should conform to the deployed model's prediction input schema.\n",
+        "instances = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
+        "\n",
+        "prediction = endpoint.predict(instances=instances)\n",
+        "\n",
+        "print(prediction)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9d347472d5ba"
+      },
+      "source": [
+        "#### Delete the Vertex AI Model, Endpoint and BatchPredictionJob resources\n",
+        "\n",
+        "Next, undelpoy and delete the Vertex AI Model, Endpoint and BatchPredictionJob resources."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "baa3e1071f7b"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()\n",
+        "endpoint.delete()\n",
+        "\n",
+        "model = aip.Model(model_id)\n",
+        "model.delete()\n",
+        "\n",
+        "batch_job = aip.BatchPredictionJob(batch_job_id)\n",
+        "batch_job.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -965,17 +1126,7 @@
         "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
         "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
         "\n",
-        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
-        "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "Otherwise, you can delete the individual resources you created in this tutorial."
       ]
     },
     {
@@ -989,58 +1140,9 @@
         "delete_all = True\n",
         "\n",
         "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
         "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "    if \"BUCKET_URI\" in globals():\n",
+        "        ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb

@@ -294,7 +294,7 @@
       "outputs": [],
       "source": [
         "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
-        "BUCKET_URI = f\"gs://{BUCKET_NAME}"
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
       ]
     },
     {
@@ -517,7 +517,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -811,7 +811,7 @@
       },
       "outputs": [],
       "source": [
-        "MODEL_DIR = BUCKET_NAME + \"/bqmodel\"\n",
+        "MODEL_DIR = BUCKET_URI + \"/bqmodel\"\n",
         "\n",
         "compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"bqml.json\")\n",
         "\n",
@@ -936,6 +936,9 @@
         "print(\"\\n\\n\")\n",
         "print(\"model-upload\")\n",
         "artifacts = print_pipeline_output(pipeline, \"model-upload\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
         "print(\"\\n\\n\")\n",
         "print(\"endpoint-create\")\n",
         "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
@@ -1122,6 +1125,32 @@
         "job = bqclient.delete_dataset(\"bqml_tutorial\", delete_contents=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e776f9a3bdc4"
+      },
+      "source": [
+        "#### Delete the Vertex AI Model and Endpoint resources\n",
+        "\n",
+        "Next, undelpoy and delete the Vertex AI Model and Endpoint resources."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "63462e0480f0"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()\n",
+        "endpoint.delete()\n",
+        "\n",
+        "model = aip.Model(model_id)\n",
+        "model.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1147,10 +1176,8 @@
         "delete_all = True\n",
         "\n",
         "if delete_all:\n",
-        "    # (DEVELOPER TODO) Find generated resources from pipeline and delete\n",
-        "\n",
         "    if \"BUCKET_URI\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "        ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage3/get_started_with_custom_training_pipeline_components.ipynb

@@ -94,6 +94,12 @@
         "    - Training a Vertex AI custom trained model.\n",
         "    - Test the serving binary with a batch prediction job.\n",
         "    - Deploying a Vertex AI custom trained model.\n",
+        "- Execute a Vertex AI pipeline.\n",
+        "- Construct a pipeline for:\n",
+        "     - Construct a custom training component.\n",
+        "     - Convert custom training component to CustomTrainingJobOp.\n",
+        "     - Training a Vertex AI custom trained model using the converted component.\n",
+        "    - Deploying a Vertex AI custom trained model.\n",
         "- Execute a Vertex AI pipeline."
       ]
     },
@@ -291,7 +297,8 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
       ]
     },
     {
@@ -302,8 +309,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -323,7 +330,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -343,7 +350,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -490,7 +497,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -1017,7 +1024,7 @@
         "! rm -f custom.tar custom.tar.gz\n",
         "! tar cvf custom.tar custom\n",
         "! gzip custom.tar\n",
-        "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_flowers.tar.gz"
+        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_flowers.tar.gz"
       ]
     },
     {
@@ -1077,7 +1084,7 @@
       },
       "outputs": [],
       "source": [
-        "PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_training\".format(BUCKET_NAME)\n",
+        "PIPELINE_ROOT = \"{}/pipeline_root/custom_icn_training\".format(BUCKET_URI)\n",
         "DEPLOY_COMPUTE = \"n1-standard-4\"\n",
         "\n",
         "\n",
@@ -1210,11 +1217,11 @@
         "file_1 = test_item_1.split(\"/\")[-1]\n",
         "file_2 = test_item_2.split(\"/\")[-1]\n",
         "\n",
-        "! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
-        "! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
+        "! gsutil cp $test_item_1 $BUCKET_URI/$file_1\n",
+        "! gsutil cp $test_item_2 $BUCKET_URI/$file_2\n",
         "\n",
-        "test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
-        "test_item_2 = BUCKET_NAME + \"/\" + file_2"
+        "test_item_1 = BUCKET_URI + \"/\" + file_1\n",
+        "test_item_2 = BUCKET_URI + \"/\" + file_2"
       ]
     },
     {
@@ -1247,7 +1254,7 @@
         "\n",
         "import tensorflow as tf\n",
         "\n",
-        "gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
+        "gcs_input_uri = BUCKET_URI + \"/test.jsonl\"\n",
         "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
         "    data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
         "    f.write(json.dumps(data) + \"\\n\")\n",
@@ -1297,7 +1304,7 @@
         "        \"import_file\": IMPORT_FILE,\n",
         "        \"batch_files\": [gcs_input_uri],\n",
         "        \"display_name\": \"flowers\" + TIMESTAMP,\n",
-        "        \"python_package\": f\"{BUCKET_NAME}/trainer_flowers.tar.gz\",\n",
+        "        \"python_package\": f\"{BUCKET_URI}/trainer_flowers.tar.gz\",\n",
         "        \"python_module\": \"trainer.task\",\n",
         "        \"project\": PROJECT_ID,\n",
         "        \"region\": REGION,\n",
@@ -1361,14 +1368,29 @@
         "            + str(TASK_ID)\n",
         "            + \"/gcp_resources\"\n",
         "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
         "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
         "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
         "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
         "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
         "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
         "\n",
         "\n",
         "print(\"image-dataset-create\")\n",
@@ -1377,9 +1399,18 @@
         "print(\"custompythonpackagetrainingjob-run\")\n",
         "artifacts = print_pipeline_output(pipeline, \"custompythonpackagetrainingjob-run\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
+        "print(\"\\n\\n\")\n",
         "print(\"endpoint-create\")\n",
         "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
+        "    \"resourceName\"\n",
+        "]\n",
         "print(\"model-deploy\")\n",
         "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
         "print(\"\\n\\n\")\n",
@@ -1392,7 +1423,13 @@
         "    output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
         "        \"gcsOutputDirectory\"\n",
         "    ]\n",
-        ")"
+        ")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "batch_job_id = output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
+        "    \"resourceName\"\n",
+        "]\n",
+        "print(\"\\n\\n\")"
       ]
     },
     {
@@ -1417,6 +1454,42 @@
         "pipeline.delete()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d614c239d74c"
+      },
+      "source": [
+        "#### Delete the Vertex AI Model, Endpoint and BatchPredictionJob resources\n",
+        "\n",
+        "Next, undelpoy and delete the Vertex AI Model, Endpoint and BathPredictionJob resources."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "417791a1a7e2"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = aip.Endpoint(endpoint_id)\n",
+        "endpoint.undeploy_all()\n",
+        "try:\n",
+        "    endpoint.delete()\n",
+        "except:\n",
+        "    pass\n",
+        "\n",
+        "model = aip.Model(model_id)\n",
+        "try:\n",
+        "    model.delete()\n",
+        "except:\n",
+        "    pass\n",
+        "\n",
+        "batch_job = aip.BatchPredictionJob(batch_job_id)\n",
+        "batch_job.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1616,9 +1689,7 @@
       },
       "outputs": [],
       "source": [
-        "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
-        "\n",
-        "PIPELINE_ROOT = \"{}/pipeline_root/custom_cifar10_training\".format(BUCKET_NAME)\n",
+        "PIPELINE_ROOT = \"{}/pipeline_root/custom_cifar10_training\".format(BUCKET_URI)\n",
         "\n",
         "\n",
         "@dsl.pipeline(name=\"custom-model-training-sample-pipeline\")\n",
@@ -1632,6 +1703,8 @@
         "    location: str = REGION,\n",
         "    deploy_image: str = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
         "):\n",
+        "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
+        "\n",
         "    custom_job_op = custom_job_training_op(\n",
         "        model_dir=model_dir,\n",
         "        lr=lr,\n",
@@ -1735,46 +1808,6 @@
         "PROJECT_NUMBER = pipeline.gca_resource.name.split(\"/\")[1]\n",
         "print(PROJECT_NUMBER)\n",
         "\n",
-        "\n",
-        "def print_pipeline_output(job, output_task_name):\n",
-        "    JOB_ID = job.name\n",
-        "    print(JOB_ID)\n",
-        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
-        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
-        "        EXECUTE_OUTPUT = (\n",
-        "            PIPELINE_ROOT\n",
-        "            + \"/\"\n",
-        "            + PROJECT_NUMBER\n",
-        "            + \"/\"\n",
-        "            + JOB_ID\n",
-        "            + \"/\"\n",
-        "            + output_task_name\n",
-        "            + \"_\"\n",
-        "            + str(TASK_ID)\n",
-        "            + \"/executor_output.json\"\n",
-        "        )\n",
-        "        GCP_RESOURCES = (\n",
-        "            PIPELINE_ROOT\n",
-        "            + \"/\"\n",
-        "            + PROJECT_NUMBER\n",
-        "            + \"/\"\n",
-        "            + JOB_ID\n",
-        "            + \"/\"\n",
-        "            + output_task_name\n",
-        "            + \"_\"\n",
-        "            + str(TASK_ID)\n",
-        "            + \"/gcp_resources\"\n",
-        "        )\n",
-        "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
-        "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
-        "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
-        "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
-        "\n",
-        "    return EXECUTE_OUTPUT\n",
-        "\n",
-        "\n",
         "print(\"custom-train-model\")\n",
         "artifacts = print_pipeline_output(pipeline, \"custom-train-model\")\n",
         "print(\"\\n\\n\")\n",
@@ -1784,9 +1817,17 @@
         "print(\"model-upload\")\n",
         "artifacts = print_pipeline_output(pipeline, \"model-upload\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
         "print(\"endpoint-create\")\n",
         "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
         "print(\"\\n\\n\")\n",
+        "output = !gsutil cat $artifacts\n",
+        "output = json.loads(output[0])\n",
+        "endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
+        "    \"resourceName\"\n",
+        "]\n",
         "print(\"model-deploy\")\n",
         "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
         "print(\"\\n\\n\")"
@@ -1814,6 +1855,49 @@
         "pipeline.delete()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "93e69fc8b9e3"
+      },
+      "source": [
+        "#### Delete the Vertex AI Model and Endpoint resource\n",
+        "\n",
+        "Next, undelpoy and delete the Vertex AI Model and Endpoint resources."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "c9d18ae084b1"
+      },
+      "source": [
+        "#### Delete the Vertex model and endpoint\n",
+        "\n",
+        "Next, undelpoy and delete the Vertex Model and Endpoint resource."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dccf71121d0b"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()\n",
+        "try:\n",
+        "    endpoint.delete()\n",
+        "except:\n",
+        "    pass\n",
+        "\n",
+        "model = aip.Model(model_id)\n",
+        "try:\n",
+        "    model.delete()\n",
+        "except:\n",
+        "    pass"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1825,17 +1909,7 @@
         "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
         "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
         "\n",
-        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
-        "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "Otherwise, you can delete the individual resources you created in this tutorial."
       ]
     },
     {
@@ -1849,58 +1923,8 @@
         "delete_all = True\n",
         "\n",
         "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "    if \"BUCKET_URI\" in globals():\n",
+        "        ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage3/get_started_with_kubeflow_pipelines.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -78,6 +78,7 @@
         "- Building KFP lightweight Python function components.\n",
         "- Assembling and compiling KFP components into a pipeline.\n",
         "- Executing a KFP pipeline using Vertex AI Pipelines.\n",
+        "- Loading component and pipeline definitions from a source code repository.\n",
         "- Building sequential, parallel, multiple output components.\n",
         "- Building control flow into pipelines."
       ]
@@ -114,7 +115,11 @@
         "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
         "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
         "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
+        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
+        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
+        "    ! pip3 install --upgrade rpy2 $USER_FLAG\n",
+        "    ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
+        "    ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
       ]
     },
     {
@@ -364,7 +369,7 @@
         "):\n",
         "    # Get your GCP project id from gcloud\n",
         "    shell_output = !gcloud auth list 2>/dev/null\n",
-        "    SERVICE_ACCOUNT = shell_output[2].strip()\n",
+        "    SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
         "    print(\"Service Account:\", SERVICE_ACCOUNT)"
       ]
     },
@@ -694,14 +699,29 @@
         "            + str(TASK_ID)\n",
         "            + \"/gcp_resources\"\n",
         "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
         "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
         "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
         "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
         "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
         "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
         "\n",
         "\n",
         "print_pipeline_output(pipeline, \"hello-world\")"
@@ -803,6 +823,73 @@
         "pipeline.delete()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "load_component_pipeline_git:helloworld"
+      },
+      "source": [
+        "### Loading components and pipeline YAML definitions from source control\n",
+        "\n",
+        "By storing the component and pipeline definitions in a source repository, like Github, you can version control your components and pipelines, as follows:\n",
+        "\n",
+        "- Use the method `load_component_from_url()`.\n",
+        "\n",
+        "- Pull the raw file format version from the repo. For github, that will be in the form of:\n",
+        "\n",
+        "    https://raw.githubusercontent.com/\n",
+        "\n",
+        "- Specify the version of the component/pipeline. For github, that will be the branch."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "load_component_pipeline_git:helloworld"
+      },
+      "outputs": [],
+      "source": [
+        "VERSION = \"main\"\n",
+        "hello_world_op = components.load_component_from_url(\n",
+        "    f\"https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.yaml\"\n",
+        ")\n",
+        "\n",
+        "! wget https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.json -O hello_git_example.json\n",
+        "\n",
+        "pipeline = aip.PipelineJob(\n",
+        "    display_name=\"hello_world-git\",\n",
+        "    template_path=\"hello_git_example.json\",\n",
+        "    pipeline_root=PIPELINE_ROOT,\n",
+        ")\n",
+        "\n",
+        "pipeline.run()\n",
+        "\n",
+        "! rm -f hello_git_example.json"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "source": [
+        "### Delete a pipeline job\n",
+        "\n",
+        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "outputs": [],
+      "source": [
+        "pipeline.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1241,7 +1328,13 @@
       "source": [
         "### dsl.Condition\n",
         "\n",
-        "The statement `dsl.Condition()` implements an `if` statement. There is no support for an `else` or `elif` statement. You use a separate `dsl.Condition()` for each value you want to test for. For example, if the output from a task is `True` or `False`, you will have two `dsl.Condition()` statements, one for True and one for False."
+        "The statement `dsl.Condition()` implements an `if` statement. There is no support for an `else` or `elif` statement. You use a separate `dsl.Condition()` for each value you want to test for. For example, if the output from a task is `1` or `0`, you will have two `dsl.Condition()` statements, one for 1 and one for 0.\n",
+        "\n",
+        "The condition in `dsl.Condition()` is evaluated at run-time, not compile time. As such it is not Python code anymore.  The condition is of type `ConditionOperator`. This operator takes three parts:\n",
+        "\n",
+        "- PipelineParam or task output\n",
+        "- == or !=\n",
+        "- string or integer value"
       ]
     },
     {

notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb

@@ -0,0 +1,916 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title"
+      },
+      "source": [
+        "# Vertex AI SDK: AutoML training tabular regression model for online prediction using BigQuery\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_tabular_regression_online_bq.ipynb\">\n",
+        "        <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:automl"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use the Vertex AI SDK for Python to create tabular regression models and do online prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:gsod,lrg"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:automl,training,online_prediction"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you create an AutoML tabular regression model and deploy for online prediction from a Python script using the Vertex AI SDK. You can alternatively create and deploy models using the `gcloud` command-line tool or online using the Cloud Console.\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Create a Vertex `Dataset` resource.\n",
+        "- Train the model.\n",
+        "- View the model evaluation.\n",
+        "- Deploy the `Model` resource to a serving `Endpoint` resource.\n",
+        "- Make a prediction.\n",
+        "- Undeploy the `Model`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "costs"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_local"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "If you are using Colab or Google Cloud Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
+        "\n",
+        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
+        "\n",
+        "- The Cloud Storage SDK\n",
+        "- Git\n",
+        "- Python 3\n",
+        "- virtualenv\n",
+        "- Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
+        "\n",
+        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
+        "\n",
+        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
+        "\n",
+        "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.  Activate the virtual environment.\n",
+        "\n",
+        "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
+        "\n",
+        "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
+        "\n",
+        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the latest version of Vertex AI SDK for Python."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# Google Cloud Notebook\n",
+        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    USER_FLAG = \"--user\"\n",
+        "else:\n",
+        "    USER_FLAG = \"\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_storage"
+      },
+      "source": [
+        "Install the latest GA version of *google-cloud-storage* library as well."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_storage"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install -U google-cloud-storage $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "before_you_begin:nogpu"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### GPU runtime\n",
+        "\n",
+        "This tutorial does not require a GPU runtime.\n",
+        "\n",
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "\n",
+        "3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
+        "\n",
+        "4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
+        "\n",
+        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
+        "\n",
+        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
+        "\n",
+        "**Otherwise**, follow these steps:\n",
+        "\n",
+        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
+        "\n",
+        "**Click Create service account**.\n",
+        "\n",
+        "In the **Service account name** field, enter a name, and click **Create**.\n",
+        "\n",
+        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
+        "\n",
+        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "outputs": [],
+      "source": [
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If on Google Cloud Notebook, then don't execute this code\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aiplatform"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "source": [
+        "## Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tutorial_start:automl"
+      },
+      "source": [
+        "# Tutorial\n",
+        "\n",
+        "Now you are ready to start creating your own AutoML tabular regression model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_file:u_dataset,bq"
+      },
+      "source": [
+        "#### Location of BigQuery training data.\n",
+        "\n",
+        "Now set the variable `IMPORT_FILE` to the location of the data table in BigQuery."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:gsod,bq,lrg"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = \"bq://bigquery-public-data.samples.gsod\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_dataset:tabular,bq,lrg"
+      },
+      "source": [
+        "### Create the Dataset\n",
+        "\n",
+        "Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the `Dataset` resource.\n",
+        "- `bq_source`: Alternatively, import data items from a BigQuery table into the `Dataset` resource.\n",
+        "\n",
+        "This operation may take several minutes."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_dataset:tabular,bq,lrg"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = aiplatform.TabularDataset.create(\n",
+        "    display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
+        "    bq_source=[IMPORT_FILE],\n",
+        ")\n",
+        "\n",
+        "label_column = \"mean_temp\"\n",
+        "\n",
+        "print(dataset.resource_name)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_transformations:gsod"
+      },
+      "outputs": [],
+      "source": [
+        "TRANSFORMATIONS = [\n",
+        "    {\"auto\": {\"column_name\": \"year\"}},\n",
+        "    {\"auto\": {\"column_name\": \"month\"}},\n",
+        "    {\"auto\": {\"column_name\": \"day\"}},\n",
+        "]\n",
+        "\n",
+        "label_column = \"mean_temp\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_automl_pipeline:tabular,lrg,transformations"
+      },
+      "source": [
+        "### Create and run training pipeline\n",
+        "\n",
+        "To train an AutoML model, you perform two steps: 1) create a training pipeline, and 2) run the pipeline.\n",
+        "\n",
+        "#### Create training pipeline\n",
+        "\n",
+        "An AutoML training pipeline is created with the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the `TrainingJob` resource.\n",
+        "- `optimization_prediction_type`: The type task to train the model for.\n",
+        "  - `classification`: A tabuar classification model.\n",
+        "  - `regression`: A tabular regression model.\n",
+        "- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
+        "- `optimization_objective`: The optimization objective to minimize or maximize.\n",
+        "  - binary classification:\n",
+        "    - `minimize-log-loss`\n",
+        "    - `maximize-au-roc`\n",
+        "    - `maximize-au-prc`\n",
+        "    - `maximize-precision-at-recall`\n",
+        "    - `maximize-recall-at-precision`\n",
+        "  - multi-class classification:\n",
+        "    - `minimize-log-loss`\n",
+        "  - regression:\n",
+        "    - `minimize-rmse`\n",
+        "    - `minimize-mae`\n",
+        "    - `minimize-rmsle`\n",
+        "\n",
+        "The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_automl_pipeline:tabular,lrg,transformations"
+      },
+      "outputs": [],
+      "source": [
+        "job = aiplatform.AutoMLTabularTrainingJob(\n",
+        "    display_name=\"gsod_\" + TIMESTAMP,\n",
+        "    optimization_prediction_type=\"regression\",\n",
+        "    optimization_objective=\"minimize-rmse\",\n",
+        "    column_transformations=TRANSFORMATIONS,\n",
+        ")\n",
+        "\n",
+        "print(job)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "run_automl_pipeline:tabular"
+      },
+      "source": [
+        "#### Run the training pipeline\n",
+        "\n",
+        "Next, you run the DAG (object 'job') to start the training job by invoking the method `run`, with the following parameters:\n",
+        "\n",
+        "- `dataset`: The `Dataset` resource to train the model.\n",
+        "- `model_display_name`: The human readable name for the trained model.\n",
+        "- `training_fraction_split`: The percentage of the dataset to use for training.\n",
+        "- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
+        "- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
+        "- `target_column`: The name of the column to train as the label.\n",
+        "- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
+        "- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.\n",
+        "\n",
+        "The `run` method when completed returns the `Model` resource.\n",
+        "\n",
+        "The execution of the training pipeline will take upto 8 hours."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "run_automl_pipeline:tabular"
+      },
+      "outputs": [],
+      "source": [
+        "model = job.run(\n",
+        "    dataset=dataset,\n",
+        "    model_display_name=\"gsod_\" + TIMESTAMP,\n",
+        "    training_fraction_split=0.6,\n",
+        "    validation_fraction_split=0.2,\n",
+        "    test_fraction_split=0.2,\n",
+        "    budget_milli_node_hours=8000,\n",
+        "    disable_early_stopping=False,\n",
+        "    target_column=label_column,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "evaluate_the_model:mbsdk"
+      },
+      "source": [
+        "## Review model evaluation scores\n",
+        "After your model has finished training, you can review the evaluation scores for it.\n",
+        "\n",
+        "First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "evaluate_the_model:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "# Get model resource ID\n",
+        "models = aiplatform.Model.list(filter=\"display_name=gsod_\" + TIMESTAMP)\n",
+        "\n",
+        "# Get a reference to the Model Service client\n",
+        "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
+        "model_service_client = aiplatform.gapic.ModelServiceClient(\n",
+        "    client_options=client_options\n",
+        ")\n",
+        "\n",
+        "model_evaluations = model_service_client.list_model_evaluations(\n",
+        "    parent=models[0].resource_name\n",
+        ")\n",
+        "model_evaluation = list(model_evaluations)[0]\n",
+        "print(model_evaluation)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "deploy_model:mbsdk,dedicated"
+      },
+      "source": [
+        "## Deploy the model\n",
+        "\n",
+        "Next, deploy your model for online prediction. To deploy the model, you invoke the `deploy` method, with the following parameters:\n",
+        "\n",
+        "- `machine_type`: The type of compute machine."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "deploy_model:mbsdk,dedicated"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = model.deploy(machine_type=\"n1-standard-4\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "make_prediction"
+      },
+      "source": [
+        "## Send a online prediction request\n",
+        "\n",
+        "Send a online prediction to your deployed model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "make_test_item:automl,online_prediction"
+      },
+      "source": [
+        "### Make test item\n",
+        "\n",
+        "You will use synthetic data as a test data item. Don't be concerned that we are using synthetic data -- we just want to demonstrate how to make a prediction."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "make_test_item:automl,tabular,gsod"
+      },
+      "outputs": [],
+      "source": [
+        "INSTANCE = {\"year\": \"1932\", \"month\": \"11\", \"day\": \"6\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "predict_request:mbsdk,lrg"
+      },
+      "source": [
+        "### Make the prediction\n",
+        "\n",
+        "Now that your `Model` resource is deployed to an `Endpoint` resource, you can do online predictions by sending prediction requests to the `Endpoint` resource.\n",
+        "\n",
+        "#### Request\n",
+        "\n",
+        "The format of each instance is:\n",
+        "\n",
+        "    [feature_list]\n",
+        "\n",
+        "Since the predict() method can take multiple items (instances), send your single test item as a list of one test item.\n",
+        "\n",
+        "#### Response\n",
+        "\n",
+        "The response from the predict() call is a Python dictionary with the following entries:\n",
+        "\n",
+        "- `ids`: The internal assigned unique identifiers for each prediction request.\n",
+        "- `value`: The predicted value for each prediction.\n",
+        "- `deployed_model_id`: The Vertex AI identifier for the deployed `Model` resource which did the predictions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "predict_request:mbsdk,lrg"
+      },
+      "outputs": [],
+      "source": [
+        "instances_list = [INSTANCE]\n",
+        "\n",
+        "prediction = endpoint.predict(instances_list)\n",
+        "print(prediction)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "undeploy_model:mbsdk"
+      },
+      "source": [
+        "## Undeploy the model\n",
+        "\n",
+        "When you are done doing predictions, you undeploy the model from the `Endpoint` resouce. This deprovisions all compute resources and ends billing for the deployed model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "undeploy_model:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
+        "\n",
+        "- Dataset\n",
+        "- Model\n",
+        "- Endpoint\n",
+        "- AutoML Training Job"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "delete_all = True\n",
+        "\n",
+        "if delete_all:\n",
+        "    # Delete the dataset using the Vertex dataset object\n",
+        "    dataset.delete()\n",
+        "\n",
+        "    # Delete the model using the Vertex model object\n",
+        "    model.delete()\n",
+        "\n",
+        "    # Delete the endpoint using the Vertex endpoint object\n",
+        "    endpoint.delete()\n",
+        "\n",
+        "    # Delete the AutoML trainig job\n",
+        "    job.delete()\n",
+        "\n",
+        "    if os.getenv(\"IS_TESTING\"):\n",
+        "        ! gsutil rm -r $BUCKET_URI"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "sdk_automl_tabular_regression_online_bq.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/official/matching_engine/two-tower-model-introduction.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -268,7 +268,7 @@
         "\n",
         "# Get your Google Cloud project ID from gcloud\n",
         "if not os.getenv(\"IS_TESTING\"):\n",
-        "    shell_output=! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
         "    PROJECT_ID = shell_output[0]\n",
         "    print(\"Project ID: \", PROJECT_ID)"
       ]
@@ -497,7 +497,6 @@
       },
       "outputs": [],
       "source": [
-        "import json\n",
         "import os\n",
         "import re\n",
         "import time\n",

notebooks/official/migration/UJ15 Vertex SDK AutoML Object Tracking.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ2,12 Vertex SDK Custom Image Classification with pre-built training container.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! apt-get update && apt-get install -y python3-opencv-headless\n",
         "    ! apt-get install -y libgl1-mesa-dev\n",
         "    ! pip3 install --upgrade opencv-python-headless $USER_FLAG"
@@ -183,7 +183,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ3 Vertex SDK Custom Image Classification with custom training container.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! apt-get update && apt-get install -y python3-opencv-headless\n",
         "    ! apt-get install -y libgl1-mesa-dev\n",
         "    ! pip3 install --upgrade opencv-python-headless $USER_FLAG"
@@ -183,7 +183,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ5 Vertex SDK AutoML Image Object Detection.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ6 Vertex SDK AutoML Text Classification.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ7 Vertex SDK AutoML Text Entity Extraction.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ8 Vertex SDK AutoML Text Sentiment Analysis.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ9 Vertex SDK Custom XGBoost with pre-built training container.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/pipelines/metrics_viz_run_compare_kfp.ipynb

@@ -235,7 +235,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade matplotlib $USER_FLAG"
       ]
     },

notebooks/official/pipelines/pipelines_intro_kfp.ipynb

@@ -456,7 +456,8 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
       ]
     },
     {
@@ -467,8 +468,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -488,7 +489,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -508,7 +509,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -571,9 +572,9 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
         "\n",
-        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
       ]
     },
     {
@@ -643,7 +644,7 @@
       },
       "outputs": [],
       "source": [
-        "PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_NAME)"
+        "PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_URI)"
       ]
     },
     {
@@ -689,7 +690,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -868,9 +869,7 @@
       "source": [
         "from kfp.v2 import compiler  # noqa: F811\n",
         "\n",
-        "compiler.Compiler().compile(\n",
-        "    pipeline_func=pipeline, package_path=\"intro_pipeline.json\".replace(\" \", \"_\")\n",
-        ")"
+        "compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"intro_pipeline.json\")"
       ]
     },
     {
@@ -896,7 +895,7 @@
         "\n",
         "job = aip.PipelineJob(\n",
         "    display_name=DISPLAY_NAME,\n",
-        "    template_path=\"intro_pipeline.json\".replace(\" \", \"_\"),\n",
+        "    template_path=\"intro_pipeline.json\",\n",
         "    pipeline_root=PIPELINE_ROOT,\n",
         ")\n",
         "\n",
@@ -920,6 +919,28 @@
         "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" width=\"60%\"/></a>"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d428cc8803e7"
+      },
+      "source": [
+        "### Delete the pipeline job\n",
+        "\n",
+        "You can delete the pipeline job with the method `delete()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "97802d9432e7"
+      },
+      "outputs": [],
+      "source": [
+        "job.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1049,6 +1070,28 @@
         "job.run()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "37e2b84a0c0d"
+      },
+      "source": [
+        "### Delete the pipeline job\n",
+        "\n",
+        "You can delete the pipeline job with the method `delete()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c5926e7eaad8"
+      },
+      "outputs": [],
+      "source": [
+        "job.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1182,16 +1225,7 @@
         "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
         "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
         "\n",
-        "Otherwise, you can delete the individual resources you created in this tutorial -- *Note:* this is auto-generated and not all resources may be applicable for this tutorial:\n",
-        "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
-        "- Cloud Storage Bucket"
+        "Otherwise, you can delete the individual resources you created in this tutorial."
       ]
     },
     {
@@ -1202,84 +1236,10 @@
       },
       "outputs": [],
       "source": [
-        "delete_dataset = True\n",
         "delete_pipeline = True\n",
-        "delete_model = True\n",
-        "delete_endpoint = True\n",
-        "delete_batchjob = True\n",
-        "delete_customjob = True\n",
-        "delete_hptjob = True\n",
         "delete_bucket = True\n",
         "\n",
         "try:\n",
-        "    if delete_model and \"DISPLAY_NAME\" in globals():\n",
-        "        models = aip.Model.list(\n",
-        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        model = models[0]\n",
-        "        aip.Model.delete(model)\n",
-        "        print(\"Deleted model:\", model)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "try:\n",
-        "    if delete_endpoint and \"DISPLAY_NAME\" in globals():\n",
-        "        endpoints = aip.Endpoint.list(\n",
-        "            filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        endpoint = endpoints[0]\n",
-        "        endpoint.undeploy_all()\n",
-        "        aip.Endpoint.delete(endpoint.resource_name)\n",
-        "        print(\"Deleted endpoint:\", endpoint)\n",
-        "except Exception as e:\n",
-        "    print(e)\n",
-        "\n",
-        "if delete_dataset and \"DISPLAY_NAME\" in globals():\n",
-        "    if \"none\" == \"tabular\":\n",
-        "        try:\n",
-        "            datasets = aip.TabularDataset.list(\n",
-        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
-        "            )\n",
-        "            dataset = datasets[0]\n",
-        "            aip.TabularDataset.delete(dataset.resource_name)\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "        except Exception as e:\n",
-        "            print(e)\n",
-        "\n",
-        "    if \"none\" == \"image\":\n",
-        "        try:\n",
-        "            datasets = aip.ImageDataset.list(\n",
-        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
-        "            )\n",
-        "            dataset = datasets[0]\n",
-        "            aip.ImageDataset.delete(dataset.resource_name)\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "        except Exception as e:\n",
-        "            print(e)\n",
-        "\n",
-        "    if \"none\" == \"text\":\n",
-        "        try:\n",
-        "            datasets = aip.TextDataset.list(\n",
-        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
-        "            )\n",
-        "            dataset = datasets[0]\n",
-        "            aip.TextDataset.delete(dataset.resource_name)\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "        except Exception as e:\n",
-        "            print(e)\n",
-        "\n",
-        "    if \"none\" == \"video\":\n",
-        "        try:\n",
-        "            datasets = aip.VideoDataset.list(\n",
-        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
-        "            )\n",
-        "            dataset = datasets[0]\n",
-        "            aip.VideoDataset.delete(dataset.resource_name)\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "        except Exception as e:\n",
-        "            print(e)\n",
-        "\n",
-        "try:\n",
         "    if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
         "        pipelines = aip.PipelineJob.list(\n",
         "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
@@ -1290,8 +1250,8 @@
         "except Exception as e:\n",
         "    print(e)\n",
         "\n",
-        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
-        "    ! gsutil rm -r $BUCKET_NAME"
+        "if delete_bucket and \"BUCKET_URI\" in globals():\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],