Merge branch 'main' into fix_is_testing

* feat: add CPR notebook

* feat: add CPR notebook

* feat: add CPR notebook

* feat: add CPR notebook

* feat: finalize CPR notebook

* feat: finalize CPR notebook

* feat: notebook for bqml+automl

* feat: notebook for bqml+automl

* review: edits per Erwin review

* review: edits per Erwin review

.github/workflows/ci.yaml

@@ -7,9 +7,9 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - name: Set up Python
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v3
     - name: Fetch pull request branch
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
       with:
         fetch-depth: 0      
     - name: Fetch base main branch

.github/workflows/linter/requirements.txt

@@ -3,7 +3,7 @@ ipython
 jupyter
 nbconvert
 black==22.1.0
-pyupgrade==2.31.0
+pyupgrade==2.31.1
 isort==5.10.1
 flake8==4.0.1
-nbqa==1.2.3
+nbqa==1.3.1

README.md

@@ -6,7 +6,19 @@ Welcome to the Google Cloud [Vertex AI](https://cloud.google.com/vertex-ai/docs/
 
 ## Overview
 
-The repository contains [Notebooks](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks) and [Community Content](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/community-content) that demonstrate how to develop and manage ML workflows using Google Cloud Vertex AI.
+The repository contains [notebooks](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks) and [community content](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/community-content) that demonstrate how to develop and manage ML workflows using Google Cloud Vertex AI.
+
+## Repository structure
+
+```bash
+├── community-content - Sample code and tutorials contributed by the community
+├── notebooks
+│   ├── community - Notebooks contributed by the community
+│   ├── official - Notebooks demonstrating use of each Vertex AI service
+│   │   ├── automl
+│   │   ├── custom
+│   │   ├── ...
+```
 
 ## Contributing
 

community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/mlops_pipeline_tf_agents_bandits_movie_recommendation/src/logger/requirements.txt

@@ -1,4 +1,4 @@
 google-cloud-bigquery==2.20.0
 tensorflow==2.5.3
-pillow==9.0.0
+pillow==9.0.1
 tf-agents==0.8.0

community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/mlops_pipeline_tf_agents_bandits_movie_recommendation/src/prediction_container/requirements.txt

@@ -1,4 +1,4 @@
 google-cloud-pubsub==2.5.0
-pillow==9.0.0
+pillow==9.0.1
 tf-agents==0.8.0
 tensorflow==2.5.3

community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/mlops_pipeline_tf_agents_bandits_movie_recommendation/src/simulator/requirements.txt

@@ -1,5 +1,5 @@
 dataclasses==0.6
 google-cloud-aiplatform==1.8.1
 tensorflow==2.5.3
-pillow==9.0.0
+pillow==9.0.1
 tf-agents==0.8.0

community-content/tf_keras_text_classification_distributed_single_worker_gpus_with_gcloud_local_run_and_vertex_sdk/vertex_training_with_local_mode_container.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -113,8 +113,8 @@
       },
       "outputs": [],
       "source": [
-        "! gcloud beta ai custom-jobs local-run \\\n",
-        "  --base-image=$BASE_IMAGE_URI \\\n",
+        "! gcloud ai custom-jobs local-run \\\n",
+        "  --executor-image-uri=$BASE_IMAGE_URI \\\n",
         "  --script=$SCRIPT_PATH \\\n",
         "  --output-image-uri=$OUTPUT_IMAGE_NAME \\\n",
         "  -- \\\n",

notebooks/README.md

@@ -0,0 +1,5 @@
+The [official](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/official) folder contains notebooks organized by Google Cloud product.
+
+The [community](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/community) folder contains notebooks that aren't officially supported by Google.
+
+Contributions to the repo should use the [notebook template](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb) as a starting point.

notebooks/community/CODEOWNERS

@@ -3,19 +3,19 @@
 # @global-owner1 and @global-owner2 will be requested for
 # review when someone opens a pull request.
 
-/sdk/sdk_* @aferlitsch
-/gapic @aferlitsch
-/ml_ops @aferlitsch
-/model_monitoring/* @mco
+/sdk/sdk_* @andrewferlitsch
+/gapic @andrewferlitsch
+/ml_ops @andrewferlitsch
+/model_monitoring/* @mco-gh
 /structured_data/rapid_prototyping_* @rafael-carvalho
 
-/managed_notebooks/ @notebooks-team
-/sdk/SDK_FBProphet_Forecasting_Online.ipynb			@brianchunkang
+/managed_notebooks/
+/sdk/SDK_FBProphet_Forecasting_Online.ipynb @brianchunkang
 /pipelines/google_cloud_pipeline_components_TPU_model_train_upload_deploy.ipynb @brianchunkang
-/sdk/SDK_AutoML_Forecasting_Model_Training_Example.ipynb @thehardikv
-/sdk/sdk_automl_forecasting_evaluating_a_model.ipynb @thehardikv
-/matching_engine @yinghsienwu
-/neo4j @benofben @htappen
+/matching_engine/sdk_matching_engine_for_indexing.ipynb @ivanmkc
+/matching_engine/matching_engine_for_indexing.ipynb @yinghsienwu
 /sdk/pytorch_lightning_custom_container_training.ipynb @brianchunkang
-/tensorboard @yfang1 @wattli
-/feature_store @nayaknishant
+/tensorboard @yfang1
+/feature_store @nayaknishant @morgandu
+/vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb @entrpn
+/vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb @RajeshThallam

notebooks/community/feature_store/sdk-feature-store.ipynb

@@ -180,7 +180,7 @@
       },
       "outputs": [],
       "source": [
-        "! pip install {USER_FLAG} --upgrade git+https://github.com/googleapis/python-aiplatform.git@main"
+        "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
       ]
     },
     {
@@ -742,7 +742,7 @@
       "source": [
         "### Source Data Format and Layout\n",
         "\n",
-        "BigQuery table/Avro/CSV are supported as input data types. No matter what format you are using, each imported entity *must* have an ID; also, each entity can *optionally* have a timestamp, specifying when the feature values are generated. This Colab uses Avro as an input, located at this public [bucket](https://pantheon.corp.google.com/storage/browser/cloud-samples-data-us-central1/vertex-ai/feature-store/datasets). The Avro schemas are as follows:\n",
+        "BigQuery table/Avro/CSV are supported as input data types. No matter what format you are using, each imported entity *must* have an ID; also, each entity can *optionally* have a timestamp, specifying when the feature values are generated. This Colab uses Avro as an input, located at this public [bucket](https://console.cloud.google.com/storage/browser/cloud-samples-data-us-central1/vertex-ai/feature-store/datasets). The Avro schemas are as follows:\n",
         "\n",
         "**For the Users entity**:\n",
         "```\n",

notebooks/community/matching_engine/matching_engine_for_indexing.ipynb

@@ -459,7 +459,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil cp gs://cloud-samples-data/ai-platform-unified/matching_engine/glove-100-angular.hdf5 ."
+        "! gsutil cp gs://cloud-samples-data/vertex-ai/matching_engine/glove-100-angular.hdf5 ."
       ]
     },
     {

notebooks/community/ml_ops/README.md

@@ -13,6 +13,6 @@ The purpose of this set of notebooks and markdown files is to demonstrate Google
 3. [Formalization](stage3)
 4. [Evaluation](stage4)
 5. Deployment
-6. Serving
+6. [Serving](stage6)
 7. Monitoring
 8. Continuous Training

notebooks/community/ml_ops/stage1/get_started_dataflow.ipynb

@@ -676,6 +676,31 @@
         "dataframe[\"station_number\"] = pd.to_numeric(dataframe[\"station_number\"])"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_create_dataset"
+      },
+      "source": [
+        "### Create BQ dataset resource\n",
+        "\n",
+        "First, you create an empty dataset resource in your project."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bqml_create_dataset"
+      },
+      "outputs": [],
+      "source": [
+        "BQ_MY_DATASET = 'samples'\n",
+        "BQ_MY_TABLE = 'gsod'\n",
+        "! bq --location=US mk -d \\\n",
+        "$PROJECT_ID:$BQ_MY_DATASET"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,

notebooks/community/ml_ops/stage2/README.md

@@ -169,11 +169,23 @@ The steps performed include:
 - Creating a Vertex AI `Featurestore` resource.
     - Creating `EntityType` resources for the `Featurestore` resource.
     - Creating `Feature` resources for each `EntityType` resource.
-- Import feature values (entity data items) into `Featurestore` resource.
+- Import feature values (entity data items) into `Featurestore` resource from Cloud Storage.
+- Import feature values (entity data items) into `Featurestore` resource from pandas DataFrame.
 - Perform online serving from a `Featurestore` resource.
 - Perform batch serving from a `Featurestore` resource.
 ```
 
+[Get Started with Google CMEK Training](get_started_with_cmek_training.ipynb)
+
+```
+The steps performed include:
+
+- Creating a customer managed encryption key.
+- Creating an image dataset with CMEK encryption.
+- Train an AutoML model with CMEK encryption.
+```
+
+
 ### E2E Stage Example
 
 [Stage 2: Experimentation](mlops_experimentation.ipynb)

notebooks/community/ml_ops/stage2/get_started_vertex_feature_store.ipynb

@@ -92,6 +92,8 @@
         "    - Creating `EntityType` resources for the `Featurestore` resource.\n",
         "    - Creating `Feature` resources for each `EntityType` resource.\n",
         "- Import feature values (entity data items) into `Featurestore` resource.\n",
+        "    - From a Cloud Storage location.\n",
+        "    - From a pandas DataFrame.\n",
         "- Perform online serving from a `Featurestore` resource.\n",
         "- Perform batch serving from a `Featurestore` resource."
       ]
@@ -679,7 +681,7 @@
         "\n",
         "When importing, specify the following in your request:\n",
         "\n",
-        "- Data source format: BigQuery Table/Avro/CSV\n",
+        "- Data source format: BigQuery Table/Avro/CSV/Pandas Dataframe\n",
         "- Data source URL\n",
         "- Destination: featurestore/entity types/features to be imported\n",
         "\n",
@@ -747,7 +749,7 @@
         "}\n",
         "```\n",
         "\n",
-        "### Importing the feature values\n",
+        "### Importing the feature values from Cloud Storage\n",
         "\n",
         "You import the feature values for the `EntityType` resources using the `ingest_from_gcs()` method, with the following parameters:\n",
         "\n",
@@ -755,7 +757,7 @@
         "- `feature_ids`: A list of identifier names for `Feature` resources' data to add to the `EntityType` resource.\n",
         "- `feature_time`: The field corresponding to the timestamp for the features being entered.\n",
         "- `gcs_source_type`: The format of the imported data. Must be CSV or Avro.\n",
-        "- `gcs_source_uris=`: A list of one or more Cloud Storage locations of the imported data files."
+        "- `gcs_source_uris`: A list of one or more Cloud Storage locations of the imported data files."
       ]
     },
     {
@@ -787,6 +789,216 @@
         "print(response)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "featurestore_delete_entities:movies"
+      },
+      "source": [
+        "#### Delete the entity types and corresponding features and feature values\n",
+        "\n",
+        "Next, in preparation to repeat importing feature values from a dataframe, you first delete the existing entity types, and corresponding content."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "featurestore_delete_entities:movies"
+      },
+      "outputs": [],
+      "source": [
+        "entity_type = featurestore.get_entity_type(\"users\")\n",
+        "entity_type.delete(force=True)\n",
+        "entity_type = featurestore.get_entity_type(\"movies\")\n",
+        "entity_type.delete(force=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "featurestore_create:entity_type"
+      },
+      "source": [
+        "## Create entity types for your `Featurestore` resource\n",
+        "\n",
+        "Next, you create the `EntityType` resources for your `Featurestore` resource using the `create_entity_type()` method, with the following parameters:\n",
+        "\n",
+        "- `entity_type_id`: The name of the `EntityType` resource.\n",
+        "- `description`:  A description of the entity type."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "featurestore_create:entity_type"
+      },
+      "outputs": [],
+      "source": [
+        "for name, description in [(\"users\", \"Users descrip\"), (\"movies\", \"Movies descrip\")]:\n",
+        "    entity_type = featurestore.create_entity_type(\n",
+        "        entity_type_id=name, description=description\n",
+        "    )\n",
+        "    print(entity_type)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "featurestore_create:feature"
+      },
+      "source": [
+        "### Add `Feature` resources for your `EntityType` resources\n",
+        "\n",
+        "Next, you create the `Feature` resources for each of the `EntityType` resources in your `Featurestore` resource using the `create_feature()` method, with the following parameters:\n",
+        "\n",
+        "- `feature_id`: The name of the `Feature` resource.\n",
+        "- `description`: A description of the feature.\n",
+        "- `value_type`: The data type for the feature."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "featurestore_create:feature,movies"
+      },
+      "outputs": [],
+      "source": [
+        "def create_features(featurestore_name, entity_name, features):\n",
+        "    entity_type = aip.EntityType(\n",
+        "        entity_type_name=entity_name, featurestore_id=featurestore_name\n",
+        "    )\n",
+        "\n",
+        "    for feature in features:\n",
+        "        feature = entity_type.create_feature(\n",
+        "            feature_id=feature[0], description=feature[1], value_type=feature[2]\n",
+        "        )\n",
+        "        print(feature)\n",
+        "\n",
+        "\n",
+        "create_features(\n",
+        "    FEATURESTORE_NAME,\n",
+        "    \"users\",\n",
+        "    [\n",
+        "        (\"age\", \"Age descrip\", \"INT64\"),\n",
+        "        (\"gender\", \"Gender descrip\", \"STRING\"),\n",
+        "        (\"liked_genres\", \"Genres descrip\", \"STRING_ARRAY\"),\n",
+        "    ],\n",
+        ")\n",
+        "\n",
+        "create_features(\n",
+        "    FEATURESTORE_NAME,\n",
+        "    \"movies\",\n",
+        "    [\n",
+        "        (\"title\", \"Title descrip\", \"STRING\"),\n",
+        "        (\"genres\", \"Genres descrip\", \"STRING\"),\n",
+        "        (\"average_rating\", \"Ave descrip\", \"DOUBLE\"),\n",
+        "    ],\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:movies,lbn,df"
+      },
+      "outputs": [],
+      "source": [
+        "GCS_USERS_AVRO_URI = FS_ENTITIES[\"users\"]\n",
+        "GCS_MOVIES_AVRO_URI = FS_ENTITIES[\"movies\"]\n",
+        "\n",
+        "USERS_AVRO_FN = \"users.avro\"\n",
+        "MOVIES_AVRO_FN = \"movies.avro\"\n",
+        "\n",
+        "! gsutil cp $GCS_USERS_AVRO_URI $USERS_AVRO_FN\n",
+        "! gsutil cp $GCS_MOVIES_AVRO_URI $MOVIES_AVRO_FN"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "load_df_from_avro"
+      },
+      "source": [
+        "#### Load Avro Files into pandas DataFrames"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "load_df_from_avro"
+      },
+      "outputs": [],
+      "source": [
+        "from avro.datafile import DataFileReader\n",
+        "from avro.io import DatumReader\n",
+        "\n",
+        "\n",
+        "class AvroReader:\n",
+        "    def __init__(self, data_file):\n",
+        "        self.avro_reader = DataFileReader(open(data_file, \"rb\"), DatumReader())\n",
+        "\n",
+        "    def to_dataframe(self):\n",
+        "        records = [record for record in self.avro_reader]\n",
+        "        return pd.DataFrame.from_records(data=records)\n",
+        "\n",
+        "\n",
+        "import pandas as pd\n",
+        "\n",
+        "users_avro_reader = AvroReader(data_file=USERS_AVRO_FN)\n",
+        "users_source_df = users_avro_reader.to_dataframe()\n",
+        "print(users_source_df)\n",
+        "\n",
+        "movies_avro_reader = AvroReader(data_file=MOVIES_AVRO_FN)\n",
+        "movies_source_df = movies_avro_reader.to_dataframe()\n",
+        "print(movies_source_df)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "featurestore_import:movies,df"
+      },
+      "source": [
+        "### Importing the feature values from DataFrame\n",
+        "\n",
+        "You import the feature values for the `EntityType` resources using the `ingest_from_df()` method, with the following parameters:\n",
+        "\n",
+        "- `entity_id_field`: The identifier name for the parent `EntityType` resource.\n",
+        "- `feature_ids`: A list of identifier names for `Feature` resources' data to add to the `EntityType` resource.\n",
+        "- `feature_time`: The field corresponding to the timestamp for the features being entered.\n",
+        "- `df_source`: The DataFrame containing the imported feature values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "featurestore_import:movies,df"
+      },
+      "outputs": [],
+      "source": [
+        "entity_type = featurestore.get_entity_type(\"users\")\n",
+        "entity_type.ingest_from_df(\n",
+        "    feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n",
+        "    feature_time=\"update_time\",\n",
+        "    df_source=users_source_df,\n",
+        "    entity_id_field=\"user_id\",\n",
+        ")\n",
+        "\n",
+        "entity_type = featurestore.get_entity_type(\"movies\")\n",
+        "entity_type.ingest_from_df(\n",
+        "    feature_ids=[\"average_rating\", \"title\", \"genres\"],\n",
+        "    feature_time=\"update_time\",\n",
+        "    df_source=movies_source_df,\n",
+        "    entity_id_field=\"movie_id\",\n",
+        ")"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {

notebooks/community/ml_ops/stage2/get_started_vertex_vizier.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -39,10 +39,11 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_vizier.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_vertex_vizier.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
-        "  </td>\n",
+        "  </td>           \n",
         "</table>\n",
         "<br/><br/><br/>"
       ]
@@ -139,6 +140,25 @@
         "Install *one time* the packages for executing the MLOps notebooks."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1fd00fa70a2a"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\""
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -147,20 +167,8 @@
       },
       "outputs": [],
       "source": [
-        "ONCE_ONLY = False\n",
-        "if ONCE_ONLY:\n",
-        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
-        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
-        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
-        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
-        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
-        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
+        "! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "    "
       ]
     },
     {
@@ -268,7 +276,9 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type:\"string\"}\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -318,7 +328,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -329,8 +339,8 @@
       },
       "outputs": [],
       "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
       ]
     },
     {
@@ -350,7 +360,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_NAME"
+        "! gsutil mb -l $REGION $BUCKET_URI"
       ]
     },
     {
@@ -370,7 +380,7 @@
       },
       "outputs": [],
       "source": [
-        "! gsutil ls -al $BUCKET_NAME"
+        "! gsutil ls -al $BUCKET_URI"
       ]
     },
     {
@@ -415,7 +425,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
       ]
     },
     {
@@ -808,7 +818,7 @@
         "! rm -f custom.tar custom.tar.gz\n",
         "! tar cvf custom.tar custom\n",
         "! gzip custom.tar\n",
-        "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
+        "! gsutil cp custom.tar.gz $BUCKET_URI/trainer_boston.tar.gz"
       ]
     },
     {
@@ -916,7 +926,7 @@
       "outputs": [],
       "source": [
         "JOB_NAME = \"custom_job_\" + TIMESTAMP\n",
-        "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, JOB_NAME)\n",
+        "MODEL_DIR = \"{}/{}\".format(BUCKET_URI, JOB_NAME)\n",
         "\n",
         "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n",
         "    TRAIN_STRATEGY = \"single\"\n",
@@ -948,7 +958,7 @@
         "        \"disk_spec\": disk_spec,\n",
         "        \"python_package_spec\": {\n",
         "            \"executor_image_uri\": TRAIN_IMAGE,\n",
-        "            \"package_uris\": [BUCKET_NAME + \"/trainer_boston.tar.gz\"],\n",
+        "            \"package_uris\": [BUCKET_URI + \"/trainer_boston.tar.gz\"],\n",
         "            \"python_module\": \"trainer.task\",\n",
         "            \"args\": CMDARGS,\n",
         "        },\n",
@@ -1577,14 +1587,6 @@
         "\n",
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
-        "- Dataset\n",
-        "- Pipeline\n",
-        "- Model\n",
-        "- Endpoint\n",
-        "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -1596,61 +1598,8 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
-        "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.undeploy_all()\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the AutoML or Pipeline training job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom training job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_URI"
       ]
     }
   ],

notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb

@@ -0,0 +1,874 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VBOfRw7ifk8w"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 2 : AutoML Image Classfication Training with Customer Managed Encryption Keys (CMEK)\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb\">\n",
+        "      Open in Google Cloud Notebooks\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with AutoML training with a customer managed encyrption key CMEK."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:flowers,icn"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public #(GCS) bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use a customer managed encryption key (CMEK) for `Vertex AI AutoML` training.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI AutoML`\n",
+        "- Customer managed encryption key.\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Creating a customer managed encryption key.\n",
+        "- Creating an image dataset with CMEK encryption.\n",
+        "- Train an AutoML model with CMEK encryption."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "source": [
+        "## Installations\n",
+        "\n",
+        "Install the Vertex AI SDK and the KMS package for CMEK encryption."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sBfZtR4X1Dr_"
+      },
+      "outputs": [],
+      "source": [
+        "USER_FLAG = \"--user\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-kms $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "project_id"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aip\n",
+        "from google.cloud import kms"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mRk9eoTm6Pyi"
+      },
+      "source": [
+        "## Setting up Customer Managed Encryption Keys\n",
+        "\n",
+        "By default, Google Cloud automatically encrypts data when it is stored in Cloud Storage using encryption keys managed by Google. If you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) for your training jobs.\n",
+        "\n",
+        "### Enable KMS API\n",
+        "\n",
+        "First, you enble the [Cloud Key Management Service (KMS)](https://console.cloud.google.com/flows/enableapi?apiid=cloudkms.googleapis.com)\n",
+        "\n",
+        "Learn more about [Customer managed encryption keys (CMEK)](https://cloud.google.com/vertex-ai/docs/general/cmek)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RD_Pvrg584X3"
+      },
+      "source": [
+        "### Create a key ring\n",
+        "\n",
+        "After you have enabled the KMS API, you create a key ring and a key. Use the helper function `create_key_ring()` to create a key ring, with the following parameters:\n",
+        "\n",
+        "- `project_id`: Your project ID.\n",
+        "- `location`: Your region.\n",
+        "- `key_ring_id`: The unique identifier for your key ring.\n",
+        "\n",
+        "The helper function calls the KMS client method `create_key_ring()` to create your key ring.\n",
+        "\n",
+        "Learn more about [KMS: Create a key ring](https://cloud.google.com/kms/docs/samples/kms-create-key-ring)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dxRZzbvQnZC7"
+      },
+      "outputs": [],
+      "source": [
+        "KEY_RING_ID = \"your_cmek_key_ring_id\"\n",
+        "\n",
+        "\n",
+        "def create_key_ring(project_id, location, key_ring_id):\n",
+        "    \"\"\"\n",
+        "    Creates a new key ring in Cloud KMS\n",
+        "\n",
+        "    Args:\n",
+        "        project_id (string): Google Cloud project ID (e.g. 'my-project').\n",
+        "        location (string): Cloud KMS location (e.g. 'us-east1').\n",
+        "        id (string): ID of the key ring to create (e.g. 'my-key-ring').\n",
+        "\n",
+        "    Returns:\n",
+        "        KeyRing: Cloud KMS key ring.\n",
+        "\n",
+        "    \"\"\"\n",
+        "\n",
+        "    # Create the client.\n",
+        "    client = kms.KeyManagementServiceClient()\n",
+        "\n",
+        "    # Build the parent location name.\n",
+        "    location_name = f\"projects/{project_id}/locations/{location}\"\n",
+        "\n",
+        "    # Build the key ring.\n",
+        "    key_ring = {}\n",
+        "\n",
+        "    # Call the API.\n",
+        "    created_key_ring = client.create_key_ring(\n",
+        "        request={\n",
+        "            \"parent\": location_name,\n",
+        "            \"key_ring_id\": key_ring_id,\n",
+        "            \"key_ring\": key_ring,\n",
+        "        }\n",
+        "    )\n",
+        "    print(\"Created key ring: {}\".format(created_key_ring.name))\n",
+        "    return created_key_ring\n",
+        "\n",
+        "\n",
+        "key_ring = create_key_ring(\n",
+        "    project_id=PROJECT_ID, location=REGION, key_ring_id=KEY_RING_ID\n",
+        ")\n",
+        "print(key_ring)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gCL1-IfFtWXl"
+      },
+      "source": [
+        "### Create a key\n",
+        "\n",
+        "Next, you create your key. Use the helper function `create_key()` with the following parameters:\n",
+        "\n",
+        "- `project_id`: Your project ID.\n",
+        "- `location`: Your region.\n",
+        "- `key_ring_id`: The unique identifier for your key ring.\n",
+        "- `key_id`: The unique identifier for your key.\n",
+        "\n",
+        "The helper function calls the KMS client method `create_cryto_key()` to create your key.\n",
+        "\n",
+        "Learn more about [](https://cloud.google.com/kms/docs/samples/kms-create-key-symmetric-encrypt-decrypt)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "LXcagdmSnYYW"
+      },
+      "outputs": [],
+      "source": [
+        "KEY_ID = \"your_cmek_key_id\"\n",
+        "\n",
+        "\n",
+        "def create_key(project_id, location, key_ring_id, key_id):\n",
+        "    \"\"\"\n",
+        "    Creates a new symmetric encryption/decryption key in Cloud KMS.\n",
+        "\n",
+        "    Args:\n",
+        "        project_id (string): Google Cloud project ID (e.g. 'my-project').\n",
+        "        location (string): Cloud KMS location (e.g. 'us-east1').\n",
+        "        key_ring_id (string): ID of the Cloud KMS key ring (e.g. 'my-key-ring').\n",
+        "        key_id (string): ID of the key to create (e.g. 'my-symmetric-key').\n",
+        "\n",
+        "    Returns:\n",
+        "        CryptoKey: Cloud KMS key.\n",
+        "\n",
+        "    \"\"\"\n",
+        "\n",
+        "    # Create the client.\n",
+        "    client = kms.KeyManagementServiceClient()\n",
+        "\n",
+        "    # Build the parent key ring name.\n",
+        "    key_ring_name = client.key_ring_path(project_id, location, key_ring_id)\n",
+        "\n",
+        "    # Build the key.\n",
+        "    purpose = kms.CryptoKey.CryptoKeyPurpose.ENCRYPT_DECRYPT\n",
+        "    algorithm = (\n",
+        "        kms.CryptoKeyVersion.CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION\n",
+        "    )\n",
+        "    key = {\n",
+        "        \"purpose\": purpose,\n",
+        "        \"version_template\": {\n",
+        "            \"algorithm\": algorithm,\n",
+        "        },\n",
+        "    }\n",
+        "\n",
+        "    # Call the API.\n",
+        "    created_key = client.create_crypto_key(\n",
+        "        request={\"parent\": key_ring_name, \"crypto_key_id\": key_id, \"crypto_key\": key}\n",
+        "    )\n",
+        "    print(\"Created symmetric key: {}\".format(created_key.name))\n",
+        "    return created_key\n",
+        "\n",
+        "\n",
+        "key_id = create_key(\n",
+        "    project_id=PROJECT_ID, location=REGION, key_ring_id=KEY_RING_ID, key_id=KEY_ID\n",
+        ")\n",
+        "\n",
+        "print(key_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3gKDBOqC8Gl5"
+      },
+      "source": [
+        "### Set service account permissions\n",
+        "\n",
+        "Next, you set permissions for your Vertex AI service account to encrypt and decrypt resources using your key.\n",
+        "\n",
+        "Learn more about [Grant Vertex AI permissions](https://cloud.google.com/vertex-ai/docs/general/cmek#grant_permissions)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6QrRg08Vqfru"
+      },
+      "outputs": [],
+      "source": [
+        "# Reference: https://cloud.google.com/vertex-ai/docs/general/cmek#granting_permissions\n",
+        "# Get the service account\n",
+        "SERVICE_ACCOUNT = ! gcloud projects get-iam-policy {PROJECT_ID} \\\n",
+        "  --flatten=\"bindings[].members\" \\\n",
+        "  --format=\"table(bindings.members)\" \\\n",
+        "  --filter=\"bindings.role:roles/aiplatform.serviceAgent\" \\\n",
+        "  | grep -oP \"service-.+?@gcp-sa-aiplatform.iam.gserviceaccount.com\"\n",
+        "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
+        "\n",
+        "print(f\"Service account is: {SERVICE_ACCOUNT}\")\n",
+        "\n",
+        "# Give permissions\n",
+        "! gcloud kms keys add-iam-policy-binding {KEY_ID} \\\n",
+        "  --keyring={KEY_RING_ID} \\\n",
+        "  --location={REGION} \\\n",
+        "  --project={PROJECT_ID} \\\n",
+        "  --member=serviceAccount:{SERVICE_ACCOUNT} \\\n",
+        "  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1e8cd37e5f99"
+      },
+      "source": [
+        "Create the full resource identifier for the created key"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ebAHZg2vlhXL"
+      },
+      "outputs": [],
+      "source": [
+        "ENCRYPTION_SPEC_KEY_NAME = key_id.name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Aa_8wrqSkamz"
+      },
+      "source": [
+        "## Initialize Vertex SDK for Python\n",
+        "\n",
+        "Initialize the *client* for Vertex AI\n",
+        "\n",
+        "All resources created during this Notebook run will encrypted with the encryption key created above.\n",
+        "\n",
+        "You can override the encryption key at each function call."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project, bucket, and corresponding encryption key.\n",
+        "\n",
+        "All resources created during this session are encrypted with the encryption key you created.\n",
+        "\n",
+        "*Note:* You can override the encryption key at each function call."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ohdgOs69kGNU"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(\n",
+        "    project=PROJECT_ID,\n",
+        "    staging_bucket=BUCKET_NAME,\n",
+        "    location=REGION,\n",
+        "    encryption_spec_key_name=ENCRYPTION_SPEC_KEY_NAME,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_file:u_dataset,csv"
+      },
+      "source": [
+        "#### Location of Cloud Storage training data.\n",
+        "\n",
+        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:flowers,csv,icn"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = (\n",
+        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\"\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "35QVNhACqcTJ"
+      },
+      "source": [
+        "# Create `Vertex AI ImageDataset` resource\n",
+        "\n",
+        "Next, you create an `ImageDataset` resource, which will be encrypted using your encryption key."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4OfCqaYRqcTJ"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = aip.ImageDataset.create(\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    gcs_source=[IMPORT_FILE],\n",
+        "    import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
+        ")\n",
+        "\n",
+        "print(dataset.resource_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6-bBqipfqcTS"
+      },
+      "source": [
+        "# Launch a Training Job to Create a Model\n",
+        "\n",
+        "Train an AutoML Image Classification model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aA41rT_mb-rV"
+      },
+      "outputs": [],
+      "source": [
+        "job = aiplatform.AutoMLImageTrainingJob(\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    prediction_type=\"classification\",\n",
+        "    multi_label=False,\n",
+        "    model_type=\"CLOUD\",\n",
+        "    base_model=None,\n",
+        ")\n",
+        "\n",
+        "# This will take around half an hour to run\n",
+        "model = job.run(\n",
+        "    dataset=ds,\n",
+        "    model_display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    training_fraction_split=0.6,\n",
+        "    validation_fraction_split=0.2,\n",
+        "    test_fraction_split=0.2,\n",
+        "    budget_milli_node_hours=8000,\n",
+        "    disable_early_stopping=False,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5vhDsMJNqcTW"
+      },
+      "source": [
+        "# Deploy Your Model\n",
+        "\n",
+        "Deploy your model, then wait until the model FINISHES deployment before proceeding to prediction."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Y9GH72wWqcTX"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = model.deploy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nIw1ifPuqcTb"
+      },
+      "source": [
+        "# Predict on Endpoint\n",
+        "- Take one sample from the data imported to the dataset\n",
+        "- This sample will be encoded to base64 and passed to the endpoint for prediction"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "H23ISHdHVIZM"
+      },
+      "outputs": [],
+      "source": [
+        "test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
+        "test_item, test_label = str(test_item[0]).split(\",\")\n",
+        "\n",
+        "print(test_item, test_label)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TF_N0kqZU768"
+      },
+      "outputs": [],
+      "source": [
+        "import base64\n",
+        "\n",
+        "import tensorflow as tf\n",
+        "\n",
+        "with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
+        "    content = f.read()\n",
+        "\n",
+        "# The format of each instance should conform to the deployed model's prediction input schema.\n",
+        "instances_list = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
+        "\n",
+        "prediction = endpoint.predict(instances=instances_list)\n",
+        "\n",
+        "print(prediction)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nWA3qocXfk82"
+      },
+      "source": [
+        "# Undeploy Model from Endpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "V1brMaO_fk82"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "e00750837ca8"
+      },
+      "outputs": [],
+      "source": [
+        "# missing\n",
+        "endpoint.delete()\n",
+        "model.delete()\n",
+        "dataset.delete()\n",
+        "\n",
+        "! gcloud kms keys versions destroy key-version \\\n",
+        "  --key key {KEY_ID} \\\n",
+        "  --keyring={KEY_RING_ID} \\\n",
+        "  --location={REGION} "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aa95b7fff9b5"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud kms keys list --location {REGION} --keyring {KEY_RING_ID}"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "get_started_with_cmek_training.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/community/ml_ops/stage3/README.md

@@ -41,6 +41,7 @@ The steps performed include:
 - Building KFP lightweight Python function components.
 - Assembling and compiling KFP components into a pipeline.
 - Executing a KFP pipeline using Vertex AI Pipelines.
+- Loading component and pipeline definitions from a source code repository.
 - Building sequential, parallel, multiple output components.
 - Building control flow into pipelines.
 ```
@@ -91,19 +92,6 @@ The steps performed include:
 
 [Get Started with Vertex AI Hyperparameter Tuning components](get_started_with_hpt_pipeline_components.ipynb)
 
-```
-- Construct a pipeline for:
-    - Training BigQuery ML model.
-    - Evaluating the BigQuery ML model.
-    - Exporting the BigQuery ML model.
-    - Importing the BigQuery ML model to a Vertex AI model.
-    - Deploy the Vertex AI model.
-- Execute a Vertex AI pipeline.
-- Make a prediction with the deployed Vertex AI model.
-```
-
-[Get Started with BQML components](get_started_with_bqml_pipeline_components.ipynb)
-
 ```
 The steps performed include:
 
@@ -116,6 +104,34 @@ The steps performed include:
 - Execute a Vertex AI pipeline.
 ```
 
+[Get Started with BQML components](get_started_with_bqml_pipeline_components.ipynb)
+
+```
+The steps performed include:
+
+- Construct a pipeline for:
+    - Training BigQuery ML model.
+    - Evaluating the BigQuery ML model.
+    - Exporting the BigQuery ML model.
+    - Importing the BigQuery ML model to a Vertex AI model.
+    - Deploy the Vertex AI model.
+- Execute a Vertex AI pipeline.
+- Make a prediction with the deployed Vertex AI model.
+```
+
+[Get Started with rapid prototyping with BQML and AutoML components](get_started_with_rapid_prototyping_bqml_automl.ipynb)
+
+
+```
+The steps performed include:
+
+- Creating a BigQuery and Vertex AI training dataset.
+- Training a BigQuery ML and AutoML model.
+- Extracting evaluation metrics from the BigQueryML and AutoML models.
+- Selecting the best trained model.
+- Deploying the best trained model.
+- Testing the deployed model infrastructure.
+```
 
 ### E2E Stage Example
 

notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb

@@ -131,7 +131,9 @@
         "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
         "    ! pip3 install --upgrade kfp $USER_FLAG\n",
         "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
-        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
+        "    ! pip3 install --upgrade rpy2 $USER_FLAG\n",
+        "    ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
+        "    ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
       ]
     },
     {
@@ -540,11 +542,13 @@
       },
       "outputs": [],
       "source": [
-        "from kfp.v2.dsl import Artifact, Input, Model\n",
+        "from kfp.v2.dsl import Artifact, Input, Model, Output\n",
         "\n",
         "\n",
         "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
-        "def evaluateAutoMLModelOp(model: Input[Artifact], region: str) -> str:\n",
+        "def evaluateAutoMLModelOp(\n",
+        "    model: Input[Artifact], region: str, model_evaluation: Output[Artifact]\n",
+        "):\n",
         "    import logging\n",
         "\n",
         "    import google.cloud.aiplatform.gapic as gapic\n",
@@ -557,8 +561,7 @@
         "\n",
         "    model_evaluations = model_service_client.list_model_evaluations(parent=model_id)\n",
         "    model_evaluation = list(model_evaluations)[0]\n",
-        "    logging.info(model_evaluation)\n",
-        "    return str(model_evaluation)"
+        "    logging.info(model_evaluation)"
       ]
     },
     {
@@ -745,7 +748,7 @@
       "source": [
         "### Make the batch input file\n",
         "\n",
-        "Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can be either CSV or JSONL. You will use JSONL in this tutorial. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
+        "Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can only be in JSONL. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
         "\n",
         "- `content`: The Cloud Storage path to the image.\n",
         "- `mime_type`: The content type. In our example, it is a `jpeg` file.\n",
@@ -877,14 +880,29 @@
         "            + str(TASK_ID)\n",
         "            + \"/gcp_resources\"\n",
         "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
         "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
         "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
         "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
         "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
         "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
         "\n",
         "\n",
         "print(\"image-dataset-create\")\n",

notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb

@@ -1010,8 +1010,7 @@
         "- Determine if the best trial results exceed a threshold\n",
         "    - Retrieve the hyperparameter values\n",
         "    - Determine Cloud Storage location of the best model\n",
-        "    - Import the best model as an UnmanagedContainerModel artifact\n",
-        "    - Upload the UnmanagedContainerModel artifact as a Vertex AI Model resource."
+        "    - Upload the best model as a Vertex AI Model resource."
       ]
     },
     {
@@ -1043,6 +1042,7 @@
         "    study_spec_metrics: list,\n",
         "    study_spec_parameters: list,\n",
         "    threshold: float,\n",
+        "    deploy_image: str,\n",
         "    max_trial_count: int = 5,\n",
         "    parallel_trial_count: int = 1,\n",
         "    base_output_directory: str = PIPELINE_ROOT,\n",
@@ -1053,10 +1053,10 @@
         "\n",
         "    from google_cloud_pipeline_components.experimental import \\\n",
         "        hyperparameter_tuning_job\n",
+        "    from google_cloud_pipeline_components.types import artifact_types\n",
         "    from google_cloud_pipeline_components.v1.hyperparameter_tuning_job import \\\n",
         "        HyperparameterTuningJobRunOp\n",
         "    from google_cloud_pipeline_components.v1.model import ModelUploadOp\n",
-        "    from google_cloud_pipeline_components.types import artifact_types\n",
         "    from kfp.v2.components import importer_node\n",
         "\n",
         "    tuning_op = HyperparameterTuningJobRunOp(\n",
@@ -1095,22 +1095,21 @@
         "\n",
         "        model_dir_op = model_dir(base_output_directory, best_trial_op.output)\n",
         "\n",
-        "        unmanaged_model_importer = importer_node.importer(\n",
-        "          artifact_uri=model_dir_op.output,\n",
-        "          artifact_class=artifact_types.UnmanagedContainerModel,\n",
-        "          metadata={\n",
-        "              'containerSpec': {\n",
-        "                  'imageUri': DEPLOY_IMAGE\n",
-        "              }\n",
-        "          }\n",
-        "        )\n",
+        "        import_unmanaged_model_op = importer_node.importer(\n",
+        "            artifact_uri=model_dir_op.output,\n",
+        "            artifact_class=artifact_types.UnmanagedContainerModel,\n",
+        "            metadata={\n",
+        "                \"containerSpec\": {\n",
+        "                    \"imageUri\": DEPLOY_IMAGE,\n",
+        "                },\n",
+        "            },\n",
+        "        ).after(model_dir_op)\n",
         "\n",
         "        model_upload_op = ModelUploadOp(\n",
-        "            display_name=display_name,\n",
-        "            unmanaged_container_model=unmanaged_model_importer.outputs['artifact'],\n",
         "            project=project,\n",
-        "            location=region,\n",
-        "        )"
+        "            display_name=display_name,\n",
+        "            unmanaged_container_model=import_unmanaged_model_op.outputs[\"artifact\"],\n",
+        "        ).after(import_unmanaged_model_op)"
       ]
     },
     {
@@ -1224,6 +1223,7 @@
         "        \"study_spec_metrics\": metric_spec,\n",
         "        \"study_spec_parameters\": parameter_spec,\n",
         "        \"threshold\": 0.7,\n",
+        "        \"deploy_image\": DEPLOY_IMAGE,\n",
         "    },\n",
         "    enable_caching=False,\n",
         ")\n",
@@ -1283,14 +1283,29 @@
         "            + str(TASK_ID)\n",
         "            + \"/gcp_resources\"\n",
         "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
         "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
         "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
         "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
         "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
         "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
         "\n",
         "\n",
         "print(\"hyperparameter-tuning-job\")\n",

notebooks/community/ml_ops/stage3/mlops_formalization.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2022 Google LLC\n",
+        "# Copyright 2021 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -1872,14 +1872,29 @@
         "            + str(TASK_ID)\n",
         "            + \"/gcp_resources\"\n",
         "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
         "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
         "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
         "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
         "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
         "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
         "\n",
         "\n",
         "print(\"make-chicago-bq-dataset\")\n",
@@ -2120,7 +2135,9 @@
         "    region: str = REGION,\n",
         "    labels: dict = {\"base_model\": \"1\"},\n",
         "):\n",
-        "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
+        "    from google_cloud_pipeline_components.types import artifact_types\n",
+        "    from google_cloud_pipeline_components.v1.model import ModelUploadOp\n",
+        "    from kfp.v2.components import importer_node\n",
         "\n",
         "    model_build_op = build_model(\n",
         "        dataset_id=dataset_id,\n",
@@ -2130,14 +2147,21 @@
         "        project=project,\n",
         "    )\n",
         "\n",
-        "    model_upload_op = gcc_aip.ModelUploadOp(\n",
-        "        display_name=display_name,\n",
+        "    import_unmanaged_model_task = importer_node.importer(\n",
         "        artifact_uri=model_build_op.output,\n",
-        "        serving_container_image_uri=deploy_image,\n",
-        "        labels=labels,\n",
+        "        artifact_class=artifact_types.UnmanagedContainerModel,\n",
+        "        metadata={\n",
+        "            \"containerSpec\": {\n",
+        "                \"imageUri\": DEPLOY_IMAGE,\n",
+        "            },\n",
+        "        },\n",
+        "    ).after(model_build_op)\n",
+        "\n",
+        "    model_upload = ModelUploadOp(\n",
         "        project=project,\n",
-        "        location=region,\n",
-        "    )"
+        "        display_name=display_name,\n",
+        "        unmanaged_container_model=import_unmanaged_model_task.outputs[\"artifact\"],\n",
+        "    ).after(import_unmanaged_model_task)"
       ]
     },
     {
@@ -3048,6 +3072,8 @@
         "    region: str = REGION,\n",
         "):\n",
         "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
+        "    from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
+        "                                                              ModelDeployOp)\n",
         "\n",
         "    with dsl.Condition(warmup == \"True\", name=\"warmup-model\"):\n",
         "\n",
@@ -3089,13 +3115,13 @@
         "            labels=label,\n",
         "        )\n",
         "\n",
-        "        endpoint_op = gcc_aip.EndpointCreateOp(\n",
+        "        endpoint_op = EndpointCreateOp(\n",
         "            project=project,\n",
         "            location=region,\n",
         "            display_name=display_name,\n",
         "        ).after(training_op)\n",
         "\n",
-        "        deploy_op = gcc_aip.ModelDeployOp(\n",
+        "        deploy_op = ModelDeployOp(\n",
         "            model=training_op.outputs[\"model\"],\n",
         "            endpoint=endpoint_op.outputs[\"endpoint\"],\n",
         "            dedicated_resources_min_replica_count=1,\n",

notebooks/community/ml_ops/stage3/src/hello_world.json

@@ -0,0 +1,91 @@
+{
+  "pipelineSpec": {
+    "components": {
+      "comp-hello-world": {
+        "executorLabel": "exec-hello-world",
+        "inputDefinitions": {
+          "parameters": {
+            "text": {
+              "type": "STRING"
+            }
+          }
+        },
+        "outputDefinitions": {
+          "parameters": {
+            "Output": {
+              "type": "STRING"
+            }
+          }
+        }
+      }
+    },
+    "deploymentSpec": {
+      "executors": {
+        "exec-hello-world": {
+          "container": {
+            "args": [
+              "--executor_input",
+              "{{$}}",
+              "--function_to_execute",
+              "hello_world"
+            ],
+            "command": [
+              "sh",
+              "-c",
+              "\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.11' && \"$0\" \"$@\"\n",
+              "sh",
+              "-ec",
+              "program_path=$(mktemp -d)\nprintf \"%s\" \"$0\" > \"$program_path/ephemeral_component.py\"\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n",
+              "\nimport kfp\nfrom kfp.v2 import dsl\nfrom kfp.v2.dsl import *\nfrom typing import *\n\ndef hello_world(text: str) -> str:\n    print(text)\n    return text\n\n"
+            ],
+            "image": "python:3.9"
+          }
+        }
+      }
+    },
+    "pipelineInfo": {
+      "name": "hello-world"
+    },
+    "root": {
+      "dag": {
+        "tasks": {
+          "hello-world": {
+            "cachingOptions": {
+              "enableCache": true
+            },
+            "componentRef": {
+              "name": "comp-hello-world"
+            },
+            "inputs": {
+              "parameters": {
+                "text": {
+                  "componentInputParameter": "text"
+                }
+              }
+            },
+            "taskInfo": {
+              "name": "hello-world"
+            }
+          }
+        }
+      },
+      "inputDefinitions": {
+        "parameters": {
+          "text": {
+            "type": "STRING"
+          }
+        }
+      }
+    },
+    "schemaVersion": "2.0.0",
+    "sdkVersion": "kfp-1.8.11"
+  },
+  "runtimeConfig": {
+    "gcsOutputDirectory": "gs://andy-1234-221921aip-20220302183146/pipeline_root/hello_world",
+    "parameters": {
+      "text": {
+        "stringValue": "hi there"
+      }
+    }
+  }
+}

notebooks/community/ml_ops/stage3/src/hello_world.yaml

@@ -0,0 +1,40 @@
+name: Hello world
+inputs:
+- {name: text, type: String}
+outputs:
+- {name: Output, type: String}
+implementation:
+  container:
+    image: python:3.9
+    command:
+    - sh
+    - -c
+    - |2
+
+      if ! [ -x "$(command -v pip)" ]; then
+          python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip
+      fi
+
+      PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.11' && "$0" "$@"
+    - sh
+    - -ec
+    - |
+      program_path=$(mktemp -d)
+      printf "%s" "$0" > "$program_path/ephemeral_component.py"
+      python3 -m kfp.v2.components.executor_main                         --component_module_path                         "$program_path/ephemeral_component.py"                         "$@"
+    - |2+
+
+      import kfp
+      from kfp.v2 import dsl
+      from kfp.v2.dsl import *
+      from typing import *
+
+      def hello_world(text: str) -> str:
+          print(text)
+          return text
+
+    args:
+    - --executor_input
+    - {executorInput: null}
+    - --function_to_execute
+    - hello_world

notebooks/community/ml_ops/stage4/README.md

@@ -44,16 +44,66 @@ This stage may be done entirely by MLOps. We recommend:
 
 [Get started with Google Artifact Registry](get_started_with_google_artifact_registry.ipynb)
 
+```
+The steps performed include:
+
+- Creating a private Docker repository.
+- Tagging a container image, specific to the private Docker repository.
+- Pushing a container image to the private Docker repository.
+- Pulling a container image from the private Docker repository.
+- Deleting a private Docker repository.
+```
+
 Get started with Vertex Model Registry
 
 [Get started with Vertex ML Metadata](get_started_with_vertex_ml_metadata.ipynb)
 
+```
+The steps performed include:
+
+- Create a `Metadatastore` resource.
+- Create (record)/List an `Artifact`, with artifacts and metadata.
+- Create (record)/List an `Execution`.
+- Create (record)/List a `Context`.
+- Add `Artifact` to `Execution` as events.
+- Add `Execution` and `Artifact` into the `Context`
+- Delete `Artifact`, `Execution` and `Context`.
+- Create and run a `Vertex AI Pipeline` ML workflow to train and deploy a scikit-learn model.
+    - Create custom pipeline components that generate artifacts and metadata.
+    - Compare Vertex AI Pipelines runs.
+    - Trace the lineage for pipeline-generated artifacts.
+    - Query your pipeline run metadata.
+```
+
 Get started with custom model evaluation
 
 Get started with A/B Testing
 
 [Get started with Vertex Explainable AI](get_started_with_vertex_xai.ipynb)
 
+```
+The steps performed include:
+
+- Train an AutoML tabular model.
+    - Do a batch prediction with explanations.
+    - Do an online prediction with explanations.
+- Train an custom TensorFlow tabular model.
+    - Manually set configuration metadata.
+    - Do a batch prediction with explanations.
+    - Do an online prediction with explanations.
+    - Automatically set configuration metadata.
+- Train an custom TensorFlow image model.
+    - Manually set configuration metadata.
+    - Do a batch prediction with explanations.
+    - Do an online prediction with explanations.
+- Train an custom XGBoost tabular model.
+    - Manually set configuration metadata.
+    - Do an online prediction with explanations.
+- Train an custom scikit-learn tabular model.
+    - Manually set configuration metadata.
+    - Do an online prediction with explanations.
+```
+
 ### E2E Stage Example
 
 Stage 4: Evaluation

notebooks/community/ml_ops/stage6/README.md

@@ -0,0 +1,58 @@
+# Stage 6: Serving
+
+## Purpose
+
+Process prediction requests and return corresponding predictions in a timely manner consistent with the business requirement, whether online, on-demand or batch predictions
+
+## Recommendations  
+
+The sixth stage in MLOps is serving predictions from the blessed model deployed to production. The serving methods, depending on business requirements may be one or more of the following:
+
+- Batch predictions – prediction requests that are queued and handled offline. This is done entirely with Google Cloud core infrastructure.
+
+- Online predictions - prediction requests that are received externally over the Internet and processed in (near) real-time. The serving of the requests/responses is done entirely with Google Cloud core infrastructure, the requesting web application/clients may originate anywhere on the Internet. If the request originates outside of the Google Cloud core infrastructure, a proxy is needed to traverse through the firewall.
+
+- On-demand predictions - prediction requests that are received internally with Google Cloud core infrastructure, or direct via an edge device. The prediction response to the requestor must be near instantaneous. The serving of the requests/responses may be either within Google Cloud core infrastructure, or externally on an edge device. An example of the former is an emergency sensor and on the later a medical sensor.
+
+This stage may be done entirely by MLOps. We recommend:
+
+- Use Google Cloud core infrastructure for online serving and batch serving, and on-demand serving where it meets the speed requirements for how the responses are utilized.
+- Use IAM role settings for access control in cross-project when the application and the serving binaries are entirely within Google Cloud core infrastructure, but in different projects.
+- Deploy serving binaries within regions that are the closest to where the requests originate. Deploy in multiple regions, when requests span regional boundaries.
+- Use Cloud Functions as a proxy when prediction requests originate externally to Google Cloud core infrastructure, or must otherwise cross firewall boundaries that cannot not otherwise be handled by IAM role settings.
+- Features that dynamically change per example (e.g., bank balance) are stored in Vertex Feature Store.
+
+
+<img src='stage6.png'>
+
+## Notebooks
+
+### Get Started
+
+[Get started with Custom Prediction Routine](get_started_with_cpr.ipynb)
+
+```
+The steps performed include:
+
+- Write a custom data preprocessor.
+- Train the model.
+- Build a custom scikit-learn serving container with custom data preprocessing using the Custom Prediction Routine model server.
+    - Test the model serving container locally.
+    - Upload and deploy the model serving container to Vertex AI Endpoint.
+    - Make a prediction request.
+- Build a custom scikit-learn serving container with custom predictor (post-processing) using the Custom Prediction Routine model server.
+    - Implement custom predictor.
+    - Test the model serving container locally.
+    - Upload and deploy the model serving container to Vertex AI Endpoint.
+    - Make a prediction request.
+- Build a custom scikit-learn serving container with custom predictor and HTTP request handler using the Custom Prediction Routine model server.
+    - Implement a custom handler.
+    - Test the model serving container locally.
+    - Upload and deploy the model serving container to Vertex AI Endpoint.
+    - Make a prediction request.
+- Customize the Dockerfile for a custom scikit-learn serving container with custom predictor and HTTP request handler using the Custom Prediction Routine model server.
+    - Implement a custom Dockerfile.
+    - Test the model serving container locally.
+    - Upload and deploy the model serving container to Vertex AI Endpoint.
+    - Make a prediction request.
+```

notebooks/notebook_template.ipynb

@@ -32,16 +32,22 @@
         "<table align=\"left\">\n",
         "\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/notebook_template.ipynb\"\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/notebook_template.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>                                                                                               \n",
         "</table>"
       ]
     },

notebooks/official/CODEOWNERS

@@ -7,6 +7,8 @@
 # matching_engine folder
 /matching_engine @shenzhimo2
 
+/tabnet/tabnet_vertex_tutorial.ipynb @longtle
+
 /migration @aferlitsch
 /explainabl_ai @aferlitsch
 /pipelines @aferlitsch
@@ -17,4 +19,4 @@
 /ml_metadata/vertex-pipelines-ml-metadata.ipynb @sararob
 /vizier/gapic-vizier-multi-objective-optimization.ipynb @halio-g
 /feature_store/gapic-feature-store.ipynb @protorganizer @diemtvu
-/managed_notebooks @GoogleCloudPlatform/notebooks-team
+/managed_notebooks @GoogleCloudPlatform/notebooks-team

notebooks/official/automl/sdk_automl_tabular_forecasting_batch.ipynb

@@ -184,7 +184,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -29,23 +29,24 @@
         "id": "title"
       },
       "source": [
-        "# Vertex SDK: AutoML training text entity extraction model for online prediction\n",
+        "# Vertex AI SDK for Python: AutoML training text entity extraction model for online prediction\n",
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_entity_extraction_online.ipynb\">\n",
+        "     <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -61,7 +62,7 @@
         "## Overview\n",
         "\n",
         "\n",
-        "This tutorial demonstrates how to use the Vertex SDK to create text entity extraction models and do online prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
+        "This tutorial demonstrates how to use the Vertex AI SDK for Python to create text entity extraction models and do online prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
       ]
     },
     {

notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -33,18 +33,18 @@
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_text_sentiment_analysis_online.ipynb\">\n",
         "      Open in Google Cloud Notebooks\n",
         "    </a>\n",
         "  </td>\n",
@@ -72,7 +72,7 @@
       "source": [
         "### Dataset\n",
         "\n",
-        "The dataset used for this tutorial is the [Crowdflower Claritin-Twitter dataset](https://data.world/crowdflower/claritin-twitter) from [data.world Datasets](https://data.world). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket."
+        "The dataset used for this tutorial is the [Crowdflower Claritin-Twitter dataset](https://data.world/crowdflower/claritin-twitter) that consists of tweets tagged with sentiment, the author's gender, and whether or not they mention any of the top 10 adverse events reported to the FDA. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. In this tutorial, you will use the tweets' data to build an AutoML-text-sentiment-analysis model on Google Cloud platform."
       ]
     },
     {
@@ -88,7 +88,7 @@
         "The steps performed include:\n",
         "\n",
         "- Create a Vertex `Dataset` resource.\n",
-        "- Train the model.\n",
+        "- Create a training job for the model.\n",
         "- View the model evaluation.\n",
         "- Deploy the `Model` resource to a serving `Endpoint` resource.\n",
         "- Make a prediction.\n",
@@ -184,7 +184,9 @@
         "id": "install_storage"
       },
       "source": [
-        "Install the latest GA version of *google-cloud-storage* library as well."
+        "Install the latest GA version of *google-cloud-storage* library as well.\n",
+        "\n",
+        "**Note**: You may encounter a PIP dependency error during the installation of the Google Cloud Storage package. This can be ignored as it will not affect the proper running of this script."
       ]
     },
     {
@@ -198,18 +200,6 @@
         "! pip3 install -U google-cloud-storage $USER_FLAG"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "install_tensorflow"
-      },
-      "outputs": [],
-      "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
-        "    ! pip3 install --upgrade tensorflow $USER_FLAG"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -218,7 +208,7 @@
       "source": [
         "### Restart the kernel\n",
         "\n",
-        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages. The following cell will restart the kernel."
       ]
     },
     {
@@ -334,7 +324,10 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -516,7 +509,7 @@
       },
       "outputs": [],
       "source": [
-        "import google.cloud.aiplatform as aip"
+        "import google.cloud.aiplatform as aiplatform"
       ]
     },
     {
@@ -538,7 +531,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
       ]
     },
     {
@@ -596,10 +589,7 @@
       },
       "outputs": [],
       "source": [
-        "if \"IMPORT_FILES\" in globals():\n",
-        "    FILE = IMPORT_FILES[0]\n",
-        "else:\n",
-        "    FILE = IMPORT_FILE\n",
+        "FILE = IMPORT_FILE\n",
         "\n",
         "count = ! gsutil cat $FILE | wc -l\n",
         "print(\"Number of Examples\", int(count[0]))\n",
@@ -633,10 +623,10 @@
       },
       "outputs": [],
       "source": [
-        "dataset = aip.TextDataset.create(\n",
+        "dataset = aiplatform.TextDataset.create(\n",
         "    display_name=\"Crowdflower Claritin-Twitter\" + \"_\" + TIMESTAMP,\n",
         "    gcs_source=[IMPORT_FILE],\n",
-        "    import_schema_uri=aip.schema.dataset.ioformat.text.sentiment,\n",
+        "    import_schema_uri=aiplatform.schema.dataset.ioformat.text.sentiment,\n",
         ")\n",
         "\n",
         "print(dataset.resource_name)"
@@ -662,9 +652,7 @@
         "  - `sentiment`: A text sentiment analysis model.\n",
         "  - `extraction`: A text entity extraction model.\n",
         "- `multi_label`: If a classification task, whether single (False) or multi-labeled (True).\n",
-        "- `sentiment_max`: If a sentiment analysis task, the maximum sentiment value.\n",
-        "\n",
-        "The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
+        "- `sentiment_max`: If a sentiment analysis task, the maximum sentiment value."
       ]
     },
     {
@@ -675,13 +663,13 @@
       },
       "outputs": [],
       "source": [
-        "dag = aip.AutoMLTextTrainingJob(\n",
+        "job = aiplatform.AutoMLTextTrainingJob(\n",
         "    display_name=\"claritin_\" + TIMESTAMP,\n",
         "    prediction_type=\"sentiment\",\n",
         "    sentiment_max=SENTIMENT_MAX,\n",
         ")\n",
         "\n",
-        "print(dag)"
+        "print(job)"
       ]
     },
     {
@@ -692,7 +680,7 @@
       "source": [
         "#### Run the training pipeline\n",
         "\n",
-        "Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
+        "Next, you run the training job by invoking the method `run`, with the following parameters:\n",
         "\n",
         "- `dataset`: The `Dataset` resource to train the model.\n",
         "- `model_display_name`: The human readable name for the trained model.\n",
@@ -702,7 +690,7 @@
         "\n",
         "The `run` method when completed returns the `Model` resource.\n",
         "\n",
-        "The execution of the training pipeline will take upto 20 minutes."
+        "The execution of the training pipeline will take upto 180 minutes."
       ]
     },
     {
@@ -713,7 +701,7 @@
       },
       "outputs": [],
       "source": [
-        "model = dag.run(\n",
+        "model = job.run(\n",
         "    dataset=dataset,\n",
         "    model_display_name=\"claritin_\" + TIMESTAMP,\n",
         "    training_fraction_split=0.8,\n",
@@ -743,11 +731,13 @@
       "outputs": [],
       "source": [
         "# Get model resource ID\n",
-        "models = aip.Model.list(filter=\"display_name=claritin_\" + TIMESTAMP)\n",
+        "models = aiplatform.Model.list(filter=\"display_name=claritin_\" + TIMESTAMP)\n",
         "\n",
         "# Get a reference to the Model Service client\n",
         "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
-        "model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
+        "model_service_client = aiplatform.gapic.ModelServiceClient(\n",
+        "    client_options=client_options\n",
+        ")\n",
         "\n",
         "model_evaluations = model_service_client.list_model_evaluations(\n",
         "    parent=models[0].resource_name\n",
@@ -894,13 +884,9 @@
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
         "- Dataset\n",
-        "- Pipeline\n",
         "- Model\n",
         "- Endpoint\n",
         "- AutoML Training Job\n",
-        "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -912,60 +898,21 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
+        "# Delete the dataset using the Vertex dataset object\n",
+        "dataset.delete()\n",
         "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the model using the Vertex model object\n",
+        "model.delete()\n",
         "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the endpoint using the Vertex endpoint object\n",
+        "endpoint.delete()\n",
         "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the AutoML or Pipeline training job\n",
+        "job.delete()\n",
         "\n",
-        "    # Delete the AutoML or Pipeline trainig job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom trainig job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "# Delete the Cloud storage bucket\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_NAME"
       ]
     }
   ],

notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -33,18 +33,18 @@
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
         "      Open in Google Cloud Notebooks\n",
         "    </a>\n",
         "  </td>\n",
@@ -72,7 +72,7 @@
       "source": [
         "### Dataset\n",
         "\n",
-        "The dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset](https://todo) from [MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model will predict the start frame where a golf swing begins."
+        "The dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset from MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model will predict the start frame where an action of golf swing begins."
       ]
     },
     {
@@ -188,7 +188,9 @@
         "id": "install_storage"
       },
       "source": [
-        "Install the latest GA version of *google-cloud-storage* library as well."
+        "Install the latest GA version of *google-cloud-storage* library as well.\n",
+        "\n",
+        "**Note**: You may encounter a PIP dependency error during the installation of the Google Cloud Storage package. This can be ignored as it will not affect the proper running of this script."
       ]
     },
     {
@@ -202,18 +204,6 @@
         "! pip3 install -U google-cloud-storage $USER_FLAG"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "install_tensorflow"
-      },
-      "outputs": [],
-      "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
-        "    ! pip3 install --upgrade tensorflow $USER_FLAG"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -222,7 +212,7 @@
       "source": [
         "### Restart the kernel\n",
         "\n",
-        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages. The following cell will restart the kernel."
       ]
     },
     {
@@ -338,7 +328,10 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -520,7 +513,7 @@
       },
       "outputs": [],
       "source": [
-        "import google.cloud.aiplatform as aip"
+        "import google.cloud.aiplatform as aiplatform"
       ]
     },
     {
@@ -542,7 +535,7 @@
       },
       "outputs": [],
       "source": [
-        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+        "aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
       ]
     },
     {
@@ -564,7 +557,7 @@
       "source": [
         "#### Location of Cloud Storage training data.\n",
         "\n",
-        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
+        "Now set the variable `IMPORT_FILES` to the location of the CSV index files in Cloud Storage."
       ]
     },
     {
@@ -602,10 +595,7 @@
       },
       "outputs": [],
       "source": [
-        "if \"IMPORT_FILES\" in globals():\n",
-        "    FILE = IMPORT_FILES[0]\n",
-        "else:\n",
-        "    FILE = IMPORT_FILE\n",
+        "FILE = IMPORT_FILES[0]\n",
         "\n",
         "count = ! gsutil cat $FILE | wc -l\n",
         "print(\"Number of Examples\", int(count[0]))\n",
@@ -638,10 +628,10 @@
       },
       "outputs": [],
       "source": [
-        "dataset = aip.VideoDataset.create(\n",
+        "dataset = aiplatform.VideoDataset.create(\n",
         "    display_name=\"Golf Swings\" + \"_\" + TIMESTAMP,\n",
         "    gcs_source=IMPORT_FILES,\n",
-        "    import_schema_uri=aip.schema.dataset.ioformat.video.action_recognition,\n",
+        "    import_schema_uri=aiplatform.schema.dataset.ioformat.video.action_recognition,\n",
         ")\n",
         "\n",
         "print(dataset.resource_name)"
@@ -665,9 +655,7 @@
         "- `prediction_type`: The type task to train the model for.\n",
         "  - `classification`: A video classification model.\n",
         "  - `object_tracking`: A video object tracking model.\n",
-        "  - `action_recognition`: A video action recognition model.\n",
-        "\n",
-        "The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
+        "  - `action_recognition`: A video action recognition model."
       ]
     },
     {
@@ -678,12 +666,12 @@
       },
       "outputs": [],
       "source": [
-        "dag = aip.AutoMLVideoTrainingJob(\n",
+        "job = aiplatform.AutoMLVideoTrainingJob(\n",
         "    display_name=\"golf_\" + TIMESTAMP,\n",
         "    prediction_type=\"action_recognition\",\n",
         ")\n",
         "\n",
-        "print(dag)"
+        "print(job)"
       ]
     },
     {
@@ -694,7 +682,7 @@
       "source": [
         "#### Run the training pipeline\n",
         "\n",
-        "Next, you run the DAG to start the training job by invoking the method `run`, with the following parameters:\n",
+        "Next, you run the training job by invoking the method `run`, with the following parameters:\n",
         "\n",
         "- `dataset`: The `Dataset` resource to train the model.\n",
         "- `model_display_name`: The human readable name for the trained model.\n",
@@ -703,7 +691,7 @@
         "\n",
         "The `run` method when completed returns the `Model` resource.\n",
         "\n",
-        "The execution of the training pipeline will take upto 20 minutes."
+        "The execution of the training pipeline can take upto 40 minutes."
       ]
     },
     {
@@ -714,7 +702,7 @@
       },
       "outputs": [],
       "source": [
-        "model = dag.run(\n",
+        "model = job.run(\n",
         "    dataset=dataset,\n",
         "    model_display_name=\"golf_\" + TIMESTAMP,\n",
         "    training_fraction_split=0.8,\n",
@@ -743,11 +731,13 @@
       "outputs": [],
       "source": [
         "# Get model resource ID\n",
-        "models = aip.Model.list(filter=\"display_name=golf_\" + TIMESTAMP)\n",
+        "models = aiplatform.Model.list(filter=\"display_name=golf_\" + TIMESTAMP)\n",
         "\n",
         "# Get a reference to the Model Service client\n",
         "client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
-        "model_service_client = aip.gapic.ModelServiceClient(client_options=client_options)\n",
+        "model_service_client = aiplatform.gapic.ModelServiceClient(\n",
+        "    client_options=client_options\n",
+        ")\n",
         "\n",
         "model_evaluations = model_service_client.list_model_evaluations(\n",
         "    parent=models[0].resource_name\n",
@@ -775,7 +765,7 @@
       "source": [
         "### Get test item(s)\n",
         "\n",
-        "Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
+        "Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model as we just want to demonstrate how to make a prediction."
       ]
     },
     {
@@ -821,31 +811,38 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "make_batch_file:automl,video"
+        "id": "13a02e5067ab"
       },
       "outputs": [],
       "source": [
         "import json\n",
         "\n",
-        "import tensorflow as tf\n",
+        "from google.cloud import storage\n",
         "\n",
-        "gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
-        "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
-        "    data = {\n",
-        "        \"content\": test_item_1,\n",
-        "        \"mimeType\": \"video/avi\",\n",
-        "        \"timeSegmentStart\": \"0.0s\",\n",
-        "        \"timeSegmentEnd\": \"5.0s\",\n",
-        "    }\n",
-        "    f.write(json.dumps(data) + \"\\n\")\n",
-        "    data = {\n",
-        "        \"content\": test_item_2,\n",
-        "        \"mimeType\": \"video/avi\",\n",
-        "        \"timeSegmentStart\": \"0.0s\",\n",
-        "        \"timeSegmentEnd\": \"5.0s\",\n",
-        "    }\n",
-        "    f.write(json.dumps(data) + \"\\n\")\n",
+        "test_filename = \"test.jsonl\"\n",
+        "gcs_input_uri = BUCKET_NAME + \"/\" + test_filename\n",
         "\n",
+        "# Configure the test-data\n",
+        "data_1 = {\n",
+        "    \"content\": test_item_1,\n",
+        "    \"mimeType\": \"video/avi\",\n",
+        "    \"timeSegmentStart\": \"0.0s\",\n",
+        "    \"timeSegmentEnd\": \"5.0s\",\n",
+        "}\n",
+        "data_2 = {\n",
+        "    \"content\": test_item_2,\n",
+        "    \"mimeType\": \"video/avi\",\n",
+        "    \"timeSegmentStart\": \"0.0s\",\n",
+        "    \"timeSegmentEnd\": \"5.0s\",\n",
+        "}\n",
+        "\n",
+        "# Upload the test-data to Cloud storage bucket\n",
+        "bucket = storage.Client(project=PROJECT_ID).bucket(BUCKET_NAME.replace(\"gs://\", \"\"))\n",
+        "blob = bucket.blob(blob_name=test_filename)\n",
+        "data = json.dumps(data_1) + \"\\n\" + json.dumps(data_2) + \"\\n\"\n",
+        "blob.upload_from_string(data)\n",
+        "\n",
+        "# Check the Cloud storage uri\n",
         "print(gcs_input_uri)\n",
         "! gsutil cat $gcs_input_uri"
       ]
@@ -931,14 +928,10 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "get_batch_prediction:mbsdk,var"
+        "id": "76f7d62916bc"
       },
       "outputs": [],
       "source": [
-        "import json\n",
-        "\n",
-        "import tensorflow as tf\n",
-        "\n",
         "bp_iter_outputs = batch_predict_job.iter_outputs()\n",
         "\n",
         "prediction_results = list()\n",
@@ -946,14 +939,13 @@
         "    if blob.name.split(\"/\")[-1].startswith(\"prediction\"):\n",
         "        prediction_results.append(blob.name)\n",
         "\n",
-        "tags = list()\n",
         "for prediction_result in prediction_results:\n",
-        "    gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{prediction_result}\"\n",
-        "    with tf.io.gfile.GFile(name=gfile_name, mode=\"r\") as gfile:\n",
-        "        for line in gfile.readlines():\n",
-        "            line = json.loads(line)\n",
-        "            print(line)\n",
-        "            break"
+        "    gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{prediction_result}\".replace(\n",
+        "        BUCKET_NAME + \"/\", \"\"\n",
+        "    )\n",
+        "    data = bucket.get_blob(gfile_name).download_as_string()\n",
+        "    data = json.loads(data)\n",
+        "    print(data)"
       ]
     },
     {
@@ -970,13 +962,9 @@
         "Otherwise, you can delete the individual resources you created in this tutorial:\n",
         "\n",
         "- Dataset\n",
-        "- Pipeline\n",
         "- Model\n",
-        "- Endpoint\n",
         "- AutoML Training Job\n",
         "- Batch Job\n",
-        "- Custom Job\n",
-        "- Hyperparameter Tuning Job\n",
         "- Cloud Storage Bucket"
       ]
     },
@@ -988,60 +976,21 @@
       },
       "outputs": [],
       "source": [
-        "delete_all = True\n",
+        "# Delete the dataset using the Vertex dataset object\n",
+        "dataset.delete()\n",
         "\n",
-        "if delete_all:\n",
-        "    # Delete the dataset using the Vertex dataset object\n",
-        "    try:\n",
-        "        if \"dataset\" in globals():\n",
-        "            dataset.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the model using the Vertex model object\n",
+        "model.delete()\n",
         "\n",
-        "    # Delete the model using the Vertex model object\n",
-        "    try:\n",
-        "        if \"model\" in globals():\n",
-        "            model.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the AutoML or Pipeline training job\n",
+        "job.delete()\n",
         "\n",
-        "    # Delete the endpoint using the Vertex endpoint object\n",
-        "    try:\n",
-        "        if \"endpoint\" in globals():\n",
-        "            endpoint.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
+        "# Delete the batch prediction job using the Vertex batch prediction object\n",
+        "batch_predict_job.delete()\n",
         "\n",
-        "    # Delete the AutoML or Pipeline trainig job\n",
-        "    try:\n",
-        "        if \"dag\" in globals():\n",
-        "            dag.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the custom trainig job\n",
-        "    try:\n",
-        "        if \"job\" in globals():\n",
-        "            job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
-        "    try:\n",
-        "        if \"batch_predict_job\" in globals():\n",
-        "            batch_predict_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
-        "    try:\n",
-        "        if \"hpt_job\" in globals():\n",
-        "            hpt_job.delete()\n",
-        "    except Exception as e:\n",
-        "        print(e)\n",
-        "\n",
-        "    if \"BUCKET_NAME\" in globals():\n",
-        "        ! gsutil rm -r $BUCKET_NAME"
+        "# Delete the Cloud Storage bucket\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_NAME"
       ]
     }
   ],

notebooks/official/automl/sdk_automl_video_classification_batch.ipynb

@@ -33,19 +33,20 @@
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "          <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_classification_batch.ipynb\">\n",
+        "            <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"> \n",
+        "Open in Vertex AI Workbench \n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",

notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -29,23 +29,24 @@
         "id": "title"
       },
       "source": [
-        "# Vertex SDK: AutoML training video object tracking model for batch prediction\n",
+        "# Vertex AI SDK for Python: AutoML training video object tracking model for batch prediction\n",
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
-        "      Open in Google Cloud Notebooks\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_object_tracking_batch.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
@@ -61,7 +62,7 @@
         "## Overview\n",
         "\n",
         "\n",
-        "This tutorial demonstrates how to use the Vertex SDK to create video object tracking models and do batch prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
+        "This tutorial demonstrates how to use the Vertex AI SDK for Python to create video object tracking models and do batch prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
       ]
     },
     {

notebooks/official/explainable_ai/sdk_custom_tabular_regression_batch_explain.ipynb

@@ -1544,7 +1544,7 @@
       },
       "outputs": [],
       "source": [
-        "if not os.environ[\"IS_TESTING\"]:\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
         "    batch_predict_job.wait()"
       ]
     },
@@ -1572,7 +1572,7 @@
       },
       "outputs": [],
       "source": [
-        "if not os.environ[\"IS_TESTING\"]:\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
         "    import tensorflow as tf\n",
         "\n",
         "    bp_iter_outputs = batch_predict_job.iter_outputs()\n",

notebooks/official/feature_store/gapic-feature-store.ipynb

@@ -648,7 +648,10 @@
         "id": "85b1f59fbf6d"
       },
       "source": [
-        "Feature [monitoring](https://cloud.google.com/vertex-ai/docs/featurestore/monitoring) is in preview, so you need to use v1beta1 Python. The easiest way to set this for now is using [console UI](https://console.cloud.google.com/vertex-ai/features). For completeness, below is example to do this using v1beta1 SDK\n"
+        "Feature [monitoring](https://cloud.google.com/vertex-ai/docs/featurestore/monitoring) is in preview, so you need to use v1beta1 Python. The easiest way to set this for now is using [console UI](https://console.cloud.google.com/vertex-ai/features). For completeness, below is example to do this using v1beta1 SDK.\n",
+        "\n",
+        "\n",
+        "You can view monitoring statistics on [console UI](https://console.cloud.google.com/vertex-ai/features)."
       ]
     },
     {
@@ -745,7 +748,6 @@
       "outputs": [],
       "source": [
         "# Create features for movies type.\n",
-        "# 'title' Feature enables monitoring.\n",
         "admin_client.batch_create_features(\n",
         "    parent=admin_client.entity_type_path(PROJECT_ID, REGION, FEATURESTORE_ID, \"movies\"),\n",
         "    requests=[\n",

notebooks/official/migration/UJ1 Vertex SDK AutoML Image Classification.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ10 Vertex SDK Custom Scikit-Learn with pre-built training container.ipynb

@@ -171,7 +171,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/migration/UJ14 Vertex SDK AutoML Video Classification.ipynb

@@ -169,7 +169,7 @@
       },
       "outputs": [],
       "source": [
-        "if os.environ[\"IS_TESTING\"]:\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
         "    ! pip3 install --upgrade tensorflow $USER_FLAG"
       ]
     },

notebooks/official/pipelines/README.md

@@ -15,6 +15,7 @@ The following examples show how to use the components defined in [google_cloud_p
 - [automl_tabular_classification_beans.ipynb](./automl_tabular_classification_beans.ipynb) (tabular classification model)
 - [google-cloud-pipeline-components_automl_text.ipynb](.google-cloud-pipeline-components_automl_text.ipynb)
 - (Experimental) [google_cloud_pipeline_components_model_train_upload_deploy.ipynb](./google_cloud_pipeline_components_model_train_upload_deploy.ipynb): includes an experimental component to run a custom training job directly by defining its worker specs
+- (Experimental) [google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb](./google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb): includes an experimental evaluation component to generate evaluation metrics for a model given ground truth and predictions
 
 **Note**: Currently, pipelines built using `kfp.v2`, such as these examples, will work only with Vertex Pipelines.
 A 'compatibility mode', which will allow these pipelines to be run on OSS KFP as well, is coming soon.

notebooks/official/pipelines/google_cloud_pipeline_components_model_train_upload_deploy.ipynb

@@ -750,14 +750,17 @@
         "    model_display_name: str = MODEL_DISPLAY_NAME,\n",
         "    serving_container_image_uri: str = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
         "):\n",
+        "    from google_cloud_pipeline_components.types import artifact_types\n",
+        "    from google_cloud_pipeline_components.v1.custom_job import \\\n",
+        "        CustomTrainingJobOp\n",
         "    from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
         "                                                              ModelDeployOp)\n",
         "    from google_cloud_pipeline_components.v1.model import ModelUploadOp\n",
-        "    from kfp.v2.google import experimental\n",
+        "    from kfp.v2.components import importer_node\n",
         "\n",
-        "    train_task = print_op(\"model training\")\n",
-        "    experimental.run_as_aiplatform_custom_job(\n",
-        "        train_task,\n",
+        "    custom_job_task = CustomTrainingJobOp(\n",
+        "        project=project,\n",
+        "        display_name=\"model-training\",\n",
         "        worker_pool_specs=[\n",
         "            {\n",
         "                \"containerSpec\": {\n",
@@ -775,14 +778,22 @@
         "        ],\n",
         "    )\n",
         "\n",
+        "    import_unmanaged_model_task = importer_node.importer(\n",
+        "        artifact_uri=WORKING_DIR,\n",
+        "        artifact_class=artifact_types.UnmanagedContainerModel,\n",
+        "        metadata={\n",
+        "            \"containerSpec\": {\n",
+        "                \"imageUri\": \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
+        "            },\n",
+        "        },\n",
+        "    ).after(custom_job_task)\n",
+        "\n",
         "    model_upload_op = ModelUploadOp(\n",
         "        project=project,\n",
         "        display_name=model_display_name,\n",
-        "        artifact_uri=WORKING_DIR,\n",
-        "        serving_container_image_uri=serving_container_image_uri,\n",
-        "        # serving_container_environment_variables={\"NOT_USED\": \"NO_VALUE\"},\n",
+        "        unmanaged_container_model=import_unmanaged_model_task.outputs[\"artifact\"],\n",
         "    )\n",
-        "    model_upload_op.after(train_task)\n",
+        "    model_upload_op.after(import_unmanaged_model_task)\n",
         "\n",
         "    endpoint_create_op = EndpointCreateOp(\n",
         "        project=project,\n",

notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb

@@ -0,0 +1,995 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic"
+      },
+      "source": [
+        "# Vertex AI Pipelines:  model upload, predict, and evaluate using google-cloud-pipeline-components\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/notebooks/blob/master/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/notebooks/blob/master/official/pipelines/google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:pipelines,custom"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This notebook shows how to use the components defined in [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) in conjunction with an experimental `evaluation` method, to build a [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) workflow that uploads a tabular custom model as a `Model` resource, creates a `BatchPredictionJob` resource, and evaluates the `Model` resource with the `BatchPredictionJob` results to create an evaluation `system.Metrics` artifact."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:bikes_weather,lrg"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is part of the [safe driver prediction Kaggle competition](https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/overview). The model has been trained on this data, and ground truth will be used for evaluation.\n",
+        "\n",
+        "The dataset predicts the whether or not a claim was filed for the policy holder."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:pipelines,custom"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you evaluate a custom model using a pipeline with components from `google_cloud_pipeline_components` and a custom pipeline component you build.\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Upload a pre-trained model as a `Model` resource.\n",
+        "- Run a `BatchPredictionJob` on the `Model` resource with ground truth data.\n",
+        "- Generate Evaluation metrics about the `Model`.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "costs"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_local"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "If you are using Colab or Google Cloud Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
+        "\n",
+        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
+        "\n",
+        "- The Cloud Storage SDK\n",
+        "- Git\n",
+        "- Python 3\n",
+        "- virtualenv\n",
+        "- Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
+        "\n",
+        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
+        "\n",
+        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
+        "\n",
+        "3. [Install virtualenv](Ihttps://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.\n",
+        "\n",
+        "4. Activate that environment and run `pip3 install Jupyter` in a terminal shell to install Jupyter.\n",
+        "\n",
+        "5. Run `jupyter notebook` on the command line in a terminal shell to launch Jupyter.\n",
+        "\n",
+        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the latest version of Vertex SDK for Python."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QTmIIQ8QBfb6"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# Google Cloud Notebook\n",
+        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    USER_FLAG = \"--user\"\n",
+        "else:\n",
+        "    USER_FLAG = \"\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_gcpc"
+      },
+      "source": [
+        "Install the latest GA version of *google-cloud-pipeline-components* library as well."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "wJkgLcdCBfb8"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "a1WZ6PC7Bfb8"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "check_versions"
+      },
+      "source": [
+        "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "check_versions:kfp,gcpc"
+      },
+      "outputs": [],
+      "source": [
+        "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
+        "! python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "before_you_begin:nogpu"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### GPU runtime\n",
+        "\n",
+        "This tutorial does not require a GPU runtime.\n",
+        "\n",
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "\n",
+        "3. [Enable the Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
+        "\n",
+        "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n",
+        "\n",
+        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mXqsucfBBfb_"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MIe7f62gBfb_"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n",
+        "\n",
+        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
+        "\n",
+        "**Otherwise**, follow these steps:\n",
+        "\n",
+        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
+        "\n",
+        "**Click Create service account**.\n",
+        "\n",
+        "In the **Service account name** field, enter a name, and click **Create**.\n",
+        "\n",
+        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
+        "\n",
+        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2q0yKPmeBfcA"
+      },
+      "outputs": [],
+      "source": [
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If on Google Cloud Notebook, then don't execute this code\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Av2hiwHuBfcB"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "00DHnc5dBfcB"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "set_service_account"
+      },
+      "source": [
+        "#### Service Account\n",
+        "\n",
+        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_service_account"
+      },
+      "outputs": [],
+      "source": [
+        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_service_account"
+      },
+      "outputs": [],
+      "source": [
+        "if (\n",
+        "    SERVICE_ACCOUNT == \"\"\n",
+        "    or SERVICE_ACCOUNT is None\n",
+        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
+        "):\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = !gcloud auth list 2>/dev/null\n",
+        "    SERVICE_ACCOUNT = shell_output[2].strip()\n",
+        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "set_service_account:pipelines"
+      },
+      "source": [
+        "#### Set service account access for Vertex AI Pipelines\n",
+        "\n",
+        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_service_account:pipelines"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
+        "\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aip"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pipeline_constants"
+      },
+      "source": [
+        "#### Vertex AI Pipelines constants\n",
+        "\n",
+        "Setup up the following constant for Vertex AI Pipelines:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4Uw-NxCiBfcD"
+      },
+      "outputs": [],
+      "source": [
+        "PIPELINE_ROOT = \"{}/pipeline_root/safe_driver\".format(BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "source": [
+        "## Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "m3atE5jVBfcE"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME, location=REGION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "define_pipeline:gcpc,bikes_weather,lrg"
+      },
+      "source": [
+        "## Define evaluation pipeline that uses components from `google_cloud_pipeline_components`\n",
+        "\n",
+        "Next, define the pipeline.\n",
+        "\n",
+        "Then, [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) components are used to define the rest of the pipeline: upload the model, run batch prediction, and evaluate the model with the given predictions.\n",
+        "\n",
+        "View the definition of the [upload model component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/aiplatform/model/upload_model/component.yaml).\n",
+        "\n",
+        "View the definition of the [batch prediction component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/aiplatform/batch_predict_job/component.yaml).\n",
+        "\n",
+        "View the definition of the [experimental evaluation component](https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/experimental/evaluation/component.yaml)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NvTkSfOeBfcF"
+      },
+      "outputs": [],
+      "source": [
+        "import kfp\n",
+        "from google_cloud_pipeline_components.experimental.evaluation import \\\n",
+        "    ModelEvaluationOp as evaluation_op\n",
+        "from google_cloud_pipeline_components.types import artifact_types\n",
+        "from google_cloud_pipeline_components.v1.batch_predict_job import \\\n",
+        "    ModelBatchPredictOp as batch_prediction_op\n",
+        "from google_cloud_pipeline_components.v1.model import \\\n",
+        "    ModelUploadOp as model_upload_op\n",
+        "from kfp.v2.components import importer_node\n",
+        "\n",
+        "DATA_URIS = [\n",
+        "    \"gs://cloud-samples-data/vertex-ai/dataset-management/datasets/safe_driver/dataset_safe_driver_train_10k.csv\"\n",
+        "]\n",
+        "MODEL_URI = \"gs://cloud-samples-data/vertex-ai/google-cloud-aiplatform-ci-artifacts/models/safe_driver/model\"\n",
+        "# Create working dir\n",
+        "WORKING_DIR = f\"{PIPELINE_ROOT}/{TIMESTAMP}\"\n",
+        "MODEL_DISPLAY_NAME = f\"safe-driver-{TIMESTAMP}\"\n",
+        "BATCH_PREDICTION_DISPLAY_NAME = f\"batch-prediction-on-pipelines-model-{TIMESTAMP}\"\n",
+        "\n",
+        "\n",
+        "@kfp.dsl.pipeline(name=\"upload-evaluate-\" + TIMESTAMP)\n",
+        "def pipeline(\n",
+        "    project: str = PROJECT_ID,\n",
+        "    model_display_name: str = MODEL_DISPLAY_NAME,\n",
+        "    batch_prediction_display_name: str = BATCH_PREDICTION_DISPLAY_NAME,\n",
+        "    batch_prediction_data_uris: list = DATA_URIS,\n",
+        "):\n",
+        "    import_unmanaged_model_task = importer_node.importer(\n",
+        "        artifact_uri=MODEL_URI,\n",
+        "        artifact_class=artifact_types.UnmanagedContainerModel,\n",
+        "        metadata={\n",
+        "            \"artifactUri\": MODEL_URI,\n",
+        "            \"predictSchemata\": {\n",
+        "                \"predictionSchemaUri\": MODEL_URI + \"/prediction_schema.yaml\",\n",
+        "                \"instanceSchemaUri\": MODEL_URI + \"/instance.yaml\",\n",
+        "            },\n",
+        "            \"containerSpec\": {\n",
+        "                \"imageUri\": \"us-docker.pkg.dev/vertex-ai/automl-tabular/prediction-server:prod\",\n",
+        "                \"healthRoute\": \"/health\",\n",
+        "                \"predictRoute\": \"/predict\",\n",
+        "            },\n",
+        "        },\n",
+        "    )\n",
+        "\n",
+        "    model_task = model_upload_op(\n",
+        "        project=project,\n",
+        "        display_name=model_display_name,\n",
+        "        unmanaged_container_model=import_unmanaged_model_task.outputs[\"artifact\"],\n",
+        "    )\n",
+        "\n",
+        "    batch_prediction_task = batch_prediction_op(\n",
+        "        project=project,\n",
+        "        model=model_task.outputs[\"model\"],\n",
+        "        job_display_name=batch_prediction_display_name,\n",
+        "        gcs_source_uris=batch_prediction_data_uris,\n",
+        "        instances_format=\"csv\",\n",
+        "        predictions_format=\"jsonl\",\n",
+        "        gcs_destination_output_uri_prefix=WORKING_DIR,\n",
+        "        machine_type=\"n1-standard-32\",\n",
+        "    )\n",
+        "\n",
+        "    evaluation_op(\n",
+        "        project=project,\n",
+        "        root_dir=WORKING_DIR,\n",
+        "        problem_type=\"classification\",\n",
+        "        classification_type=\"multiclass\",\n",
+        "        ground_truth_column=\"target\",\n",
+        "        class_names=[\"0\", \"1\"],\n",
+        "        predictions_format=\"jsonl\",\n",
+        "        batch_prediction_job=batch_prediction_task.outputs[\"batchpredictionjob\"],\n",
+        "    )"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "compile_pipeline"
+      },
+      "source": [
+        "## Compile the pipeline\n",
+        "\n",
+        "Next, compile the pipeline."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "N7kt8MqfBfcG"
+      },
+      "outputs": [],
+      "source": [
+        "from kfp.v2 import compiler  # noqa: F811\n",
+        "\n",
+        "compiler.Compiler().compile(\n",
+        "    pipeline_func=pipeline,\n",
+        "    package_path=\"evaluation_demo_pipeline.json\",\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "run_pipeline:custom"
+      },
+      "source": [
+        "## Run the pipeline\n",
+        "\n",
+        "Next, run the pipeline."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "u5s7gMoABfcG"
+      },
+      "outputs": [],
+      "source": [
+        "DISPLAY_NAME = \"safe_driver\" + TIMESTAMP\n",
+        "\n",
+        "job = aip.PipelineJob(\n",
+        "    display_name=DISPLAY_NAME,\n",
+        "    template_path=\"evaluation_demo_pipeline.json\",\n",
+        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    enable_caching=True,\n",
+        ")\n",
+        "\n",
+        "job.run()\n",
+        "\n",
+        "! rm evaluation_demo_pipeline.json"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view_pipeline_run:custom"
+      },
+      "source": [
+        "Click on the generated link to see your run in the Cloud Console.\n",
+        "\n",
+        "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view_pipleline_results:bqml"
+      },
+      "source": [
+        "### View the pipeline results and evaluation metrics"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "view_pipleline_results:bqml"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_NUMBER = job.gca_resource.name.split(\"/\")[1]\n",
+        "print(PROJECT_NUMBER)\n",
+        "\n",
+        "\n",
+        "def print_pipeline_output(job, output_task_name):\n",
+        "    JOB_ID = job.name\n",
+        "    print(JOB_ID)\n",
+        "    artifact = \"\"\n",
+        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
+        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
+        "        EXECUTE_OUTPUT = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/executor_output.json\"\n",
+        "        )\n",
+        "        GCP_RESOURCES = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/gcp_resources\"\n",
+        "        )\n",
+        "        EVALUATION_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
+        "        # Check if file exists, 0 is success\n",
+        "        !gsutil -q stat $EXECUTE_OUTPUT\n",
+        "        if _exit_code == 0:\n",
+        "            ! gsutil cat $EXECUTE_OUTPUT\n",
+        "            artifact = EXECUTE_OUTPUT\n",
+        "            break\n",
+        "        !gsutil -q stat $GCP_RESOURCES\n",
+        "        if _exit_code == 0:\n",
+        "            ! gsutil cat $GCP_RESOURCES\n",
+        "            artifact = GCP_RESOURCES\n",
+        "            break\n",
+        "        !gsutil -q stat $EVALUATION_METRICS\n",
+        "        if _exit_code == 0:\n",
+        "            ! gsutil cat $EVALUATION_METRICS\n",
+        "            artifact = EVALUATION_METRICS\n",
+        "            break\n",
+        "\n",
+        "    return artifact\n",
+        "\n",
+        "\n",
+        "print(\"model-upload\")\n",
+        "artifacts = print_pipeline_output(job, \"model-upload\")\n",
+        "print(\"\\n\\n\")\n",
+        "print(\"model-batch-predict\")\n",
+        "artifacts = print_pipeline_output(job, \"model-batch-predict\")\n",
+        "print(\"\\n\\n\")\n",
+        "print(\"model-evaluation\")\n",
+        "metrics = print_pipeline_output(job, \"model-evaluation\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "source": [
+        "### Delete a pipeline job\n",
+        "\n",
+        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "outputs": [],
+      "source": [
+        "job.delete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:pipelines"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial -- *Note:* this is auto-generated and not all resources may be applicable for this tutorial:\n",
+        "\n",
+        "- Model\n",
+        "- Batch Job\n",
+        "- Cloud Storage Bucket"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JgCXD3X-BfcG"
+      },
+      "outputs": [],
+      "source": [
+        "delete_model = True\n",
+        "delete_batchjob = True\n",
+        "delete_bucket = True\n",
+        "\n",
+        "try:\n",
+        "    if delete_model and \"MODEL_DISPLAY_NAME\" in globals():\n",
+        "        models = aip.Model.list(\n",
+        "            filter=f\"display_name={MODEL_DISPLAY_NAME}\", order_by=\"create_time\"\n",
+        "        )\n",
+        "        model = models[0]\n",
+        "        aip.Model.delete(model)\n",
+        "        print(\"Deleted model:\", model)\n",
+        "except Exception as e:\n",
+        "    print(e)\n",
+        "\n",
+        "try:\n",
+        "    if delete_batchjob and \"BATCH_PREDICTION_DISPLAY_NAME\" in globals():\n",
+        "        batch_predictions = aip.BatchPredictionJob.list(\n",
+        "            filter=f\"display_name={BATCH_PREDICTION_DISPLAY_NAME}\",\n",
+        "            order_by=\"create_time\",\n",
+        "        )\n",
+        "        batch_prediction = batch_predictions[0]\n",
+        "        aip.BatchPredictionJob.delete(batch_prediction)\n",
+        "        print(\"Deleted batch prediction job:\", batch_prediction)\n",
+        "except Exception as e:\n",
+        "    print(e)\n",
+        "\n",
+        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
+        "    ! gsutil rm -r $BUCKET_NAME"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "google_cloud_pipeline_components_model_upload_predict_evaluate.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/official/tabnet/tabnet_vertex_tutorial.ipynb

@@ -0,0 +1,978 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ur8xi4C7S06n"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JAPoU8Sm5E6e"
+      },
+      "source": [
+        "<table align=\"left\">\n",
+        "\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/tabnet/tabnet_vertex_tutorial.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samplestree/master/notebooks/official/tabnet/tabnet_vertex_tutorial.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>                                                                                               \n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tvgnzT1CKxrO"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This tutorial demonstrates how to use the TabNet algorithm on the Vertex AI platform.\n",
+        "\n",
+        "TabNet combines the best of two worlds: it is explainable (similar to simpler tree-based models) while benefiting from high performance (similar to deep neural networks). This makes it great for retailers, finance and insurance industry applications such as predicting credit scores, fraud detection and forecasting. \n",
+        "\n",
+        "TabNet uses a machine learning technique called sequential attention to select which model features to reason from at each step in the model. This mechanism makes it possible to explain how the model arrives at its predictions and helps it learn more accurate models. Thanks to this design, TabNet not only outperforms other neural networks and decision trees but also provides interpretable feature attributions. Releasing TabNet as a First Party Trainer in Vertex AI means you'll be able to easily take advantage of TabNet's architecture and explainability and use it to train models on your own data. \n",
+        "\n",
+        "\n",
+        "### Dataset\n",
+        "\n",
+        "This tutorial uses the `petfinder` in the public bucket `gs://cloud-samples-data/ai-platform-unified/datasets/tabular/`, which was generated from the [PetFinder.my Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction). \n",
+        "\n",
+        "### Objective\n",
+        "\n",
+        "In this notebook, you will learn how to run TabNet model on Vertex AI.\n",
+        "The tutorial covers the following steps:\n",
+        "1. **Setup**: Importing the required libraries and setting your global variables.\n",
+        "2. **Configure parameters**: Setting the appropriate parameter values for the training job.\n",
+        "3. **Train on Vertex Training**: Submitting a training job using csv input.\n",
+        "4. **Hyperparameter tuning**: Running a hyperparameter tuning job.\n",
+        "5. **Hyperparameter on Vertex Training with BigQuerry input**: Submitting a training job using BigQuerry input.\n",
+        "6. **Cleaning up**: Deleting resources created by this tutorial.\n",
+        "\n",
+        "\n",
+        "### Costs \n",
+        "\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ze4-nDLfK4pw"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
+        "all the requirements to run this notebook. You can skip this step."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gCuSR8GkAgzl"
+      },
+      "source": [
+        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
+        "You need the following:\n",
+        "\n",
+        "* The Google Cloud SDK\n",
+        "* Git\n",
+        "* Python 3\n",
+        "* virtualenv\n",
+        "* Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Google Cloud guide to [Setting up a Python development\n",
+        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
+        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
+        "for meeting these requirements. The following steps provide a condensed set of\n",
+        "instructions:\n",
+        "\n",
+        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
+        "\n",
+        "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
+        "\n",
+        "1. [Install\n",
+        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
+        "   and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
+        "\n",
+        "1. To install Jupyter, run `pip3 install jupyter` on the\n",
+        "command-line in a terminal shell.\n",
+        "\n",
+        "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
+        "\n",
+        "1. Open this notebook in the Jupyter Notebook Dashboard."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "i7EUnXsZhAGF"
+      },
+      "source": [
+        "### Install additional packages\n",
+        "\n",
+        "Install additional package dependencies not installed in your notebook environment, such as {XGBoost, AdaNet, or TensorFlow Hub TODO: Replace with relevant packages for the tutorial}. Use the latest major GA version of each package."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2b4ef9b72d43"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "wyy5Lbnzg5fi"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install {USER_FLAG} --upgrade tensorflow\n",
+        "! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform tensorboard-plugin-profile\n",
+        "! gcloud components update --quiet"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hhq5zEbGg0XX"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "EzrelQZ22IZj"
+      },
+      "outputs": [],
+      "source": [
+        "# Automatically restart kernel after installs\n",
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "lWEdiXsJg0XY"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### Select a GPU runtime\n",
+        "\n",
+        "**Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select \"Runtime --> Change runtime type > GPU\"**"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BF1j6f9HApxa"
+      },
+      "source": [
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
+        "\n",
+        "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). {TODO: Update the APIs needed for your tutorial. Edit the API names, and update the link to append the API IDs, separating each one with a comma. For example, container.googleapis.com,cloudbuild.googleapis.com}\n",
+        "\n",
+        "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
+        "\n",
+        "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WReHDGG5g0XY"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oM1iC_MfAts1"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "PROJECT_ID = \"\"\n",
+        "\n",
+        "# Get your Google Cloud project ID from gcloud\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID: \", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qJYoRfYng0XZ"
+      },
+      "source": [
+        "Otherwise, set your project ID here."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "riG_qUokg0XZ"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
+        "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "\n",
+        "!gcloud config set project {PROJECT_ID}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "06571eb4063b"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "697568e92bd6"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dr--iN2kAylZ"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "**If you are using Google Cloud Notebooks**, your environment is already\n",
+        "authenticated. Skip this step."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "sBCra4QMA2wR"
+      },
+      "source": [
+        "**If you are using Colab**, run the cell below and follow the instructions\n",
+        "when prompted to authenticate your account via oAuth.\n",
+        "\n",
+        "**Otherwise**, follow these steps:\n",
+        "\n",
+        "1. In the Cloud Console, go to the [**Create service account key**\n",
+        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
+        "\n",
+        "2. Click **Create service account**.\n",
+        "\n",
+        "3. In the **Service account name** field, enter a name, and\n",
+        "   click **Create**.\n",
+        "\n",
+        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
+        "into the filter box, and select\n",
+        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
+        "local environment.\n",
+        "\n",
+        "6. Enter the path to your service account key as the\n",
+        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PyQmSRbKA8r-"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "# The Google Cloud Notebook product has specific requirements\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "# If on Google Cloud Notebooks, then don't execute this code\n",
+        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zgPO1eR3CYjk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you submit a training job using the Cloud SDK, you upload a Python package\n",
+        "containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
+        "the code from this package. In this tutorial, Vertex AI also saves the\n",
+        "trained model that results from your job in the same bucket. Using this model artifact, you can then\n",
+        "create Vertex AI model and endpoint resources in order to serve\n",
+        "online predictions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
+        "Cloud Storage buckets.\n",
+        "\n",
+        "You may also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n",
+        "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may\n",
+        "not use a Multi-Regional Storage bucket for training with Vertex AI."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MzGDU7TWdts_"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "REGION = \"us-central1\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cf221059d072"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP\n",
+        "print(BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-EcIXiGsCePi"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NIq7R4HZCfIc"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ucvCsknMCims"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vhOb7YnwClBb"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XoEqT2Y4DJmf"
+      },
+      "source": [
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pRUOFELefqf1"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import re\n",
+        "import time\n",
+        "from datetime import datetime\n",
+        "\n",
+        "%load_ext tensorboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "BoVqAMOuedPe"
+      },
+      "outputs": [],
+      "source": [
+        "# Please note that if you use csv input, the first column is the label column.\n",
+        "\n",
+        "IMPORT_FILE = \"petfinder-tabular-classification-tabnet-with-header.csv\"\n",
+        "! gsutil cp gs://cloud-samples-data/ai-platform-unified/datasets/tabular/{IMPORT_FILE} {BUCKET_NAME}/data/petfinder/train.csv\n",
+        "\n",
+        "gcs_source = f\"{BUCKET_NAME}/data/petfinder/train.csv\"\n",
+        "print(\"gcs_source: \", gcs_source)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CMCfG2ao8kLh"
+      },
+      "source": [
+        "## Configure parameters"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hxB0FTQT8uqY"
+      },
+      "source": [
+        "The following table shows parameters that are common to all Vertex Training jobs created using the `gcloud ai custom-jobs create` command. See the [official documentation](https://cloud.google.com/sdk/gcloud/reference/ai/custom-jobs/create) for all the possible arguments.\n",
+        "\n",
+        "| Parameter | Data type | Description | Required |\n",
+        "|--|--|--|--|\n",
+        "| `display-name` | string | Name of the job. | Yes |\n",
+        "| `worker-pool-spec` | string | Comma-separated list of arguments specifying a worker pool configuration (see below). | Yes |\n",
+        "| `region` | string | Region to submit the job to. | No |\n",
+        "\n",
+        "The `worker-pool-spec` flag can be specified multiple times, one for each worker pool. The following table shows the arguments used to specify a worker pool.\n",
+        "\n",
+        "| Parameter | Data type | Description | Required |\n",
+        "|--|--|--|--|\n",
+        "| `machine-type` | string | Machine type for the pool. See the [official documentation](https://cloud.google.com/vertex-ai/docs/training/configure-compute) for supported machines. | Yes |\n",
+        "| `replica-count` | int | The number of replicas of the machine in the pool. | No |\n",
+        "| `container-image-uri` | string | Docker image to run on each worker. | No |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Xc_I_XHz8z6B"
+      },
+      "source": [
+        "The following table shows the parameters for the TabNet training job:\n",
+        "\n",
+        "| Parameter | Data type | Description | Required |\n",
+        "|--|--|--|--|\n",
+        "| `preprocess` | boolean argument | Specify this to enable automatic preprocessing. | No|\n",
+        "| `job_dir` | string | Cloud Storage directory where the model output files will be stored. | Yes |\n",
+        "| `input_metadata_path` | string | The GCS path to the TabNet-specific metadata for the training dataset. Please see above on how to create the metadata. | No . |\n",
+        "| `training_data_path` | string | Cloud Storage pattern where training data is stored. | Yes |\n",
+        "| `validation_data_path` | string | Cloud Storage pattern where eval data is stored. | No |\n",
+        "| `test_data_path` | string | Cloud Storage pattern where test data is stored. | Yes |\n",
+        "| `input_type` | string | “bigquery“ or “csv“ - type of the input tabular data. If csv is mentioned then the first column is treated as target. If CSV files have a header, also pass the flag “data_has_header”. If “bigquery” is used, one can either supply training/validation data paths, or supply BigQuery project, dataset, and table names for preprocessing to produce the training and validation datasets.. | Yes |\n",
+        "| `model_type` | string | The learning task such as classification or regression. | Yes |\n",
+        "| `split_column` | string | The column name used to create the training, validation, and test splits. Values of the columns (a.k.a table['split_column']) should be either “TRAIN”, “VALIDATE”, or “TEST”. “TEST” is optional. Applicable to bigquery input only. | No. |\n",
+        "| `train_batch_size` | int | Batch size for training. | No - Default is 1024. |\n",
+        "| `eval_split` | float | Split fraction to use for the evaluation dataset, if `validation_data_path` is not provided. | No - Default is 0.2 |\n",
+        "| `learning_rate` | float | Learning rate for training. | No - Default is the default learning rate of the specified optimizer. |\n",
+        "| `eval_frequency_secs` | int | Frequency at which evaluation and checkpointing will take place.The default is 600. | No . |\n",
+        "| `num_parallel_reads` | int | Number of threads used to read input files. We suggest setting it equal or slightly less than the number of CPUs of the machine for maximal performance in most cases. For example, 6 per GPU is a good default choice. | Yes . |\n",
+        "| `optimizer` | string | Training optimizer. Lowercase string name of any TF2.3 Keras optimizer is supported ('sgd', 'adam', 'ftrl', etc.). See [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers). | No - Default is 'adam'. |\n",
+        "| `data_cache` | string | Choose to cache data to “memory”, “disk” or “no_cache”. For large datasets, caching the data into memory would throw out-of-memory errors, therefore, we suggest choosing “disk”. You can specify the disk size in the config file (as exemplified below). Make sure to request a sufficiently large (e.g. TB size) disk to write the data, for large (B-scale) datasets. | No, The default one is “memory” . |\n",
+        "| `bq_project` | string | The name of the BigQuery project. If input_type=bigquery and using the flag –preprocessing, this is required. This is an alternative to specifying train, validation, and test data paths. | No . |\n",
+        "| `dataset_name` | string | The name of the BigQuery dataset. If input_type=bigquery and using the flag –preprocessing, this is required. This is an alternative to specifying train, validation, and test data paths. | No . |\n",
+        "| `table_name` | string | The name of the BigQuery table. If input_type=bigquery and using the flag –preprocessing, this is required. This is an alternative to specifying train, validation, and test data paths. | No . |\n",
+        "| `loss_function_type` | string | There are several loss function types in TabNet. For regression: mse/mae are included. For classification: cross_entropy/weighted_cross_entropy/focal_loss are included | No . If the value is \"default\", we use mse for regression and cross_entropy for classification.|\n",
+        "| `deterministic_data` | boolean argument | Determinism of data reading from tabular data. The default is set to False. When setting to True, the experiment is deterministic. For fast training on large datasets, we suggest deterministic_data=False setting, albeit having randomness in the results (which becomes negligible at large datasets). Note that determinism is still not guaranteed with distributed training, as map-reduce causes randomness due to the ordering of algebraic operations with finite precision. However, this is negligible in practice, especially on large datasets. For the cases where 100% determinism is desired, besides deterministic_data=True setting, we suggest training with a single GPU (e.g. with MACHINE_TYPE=\"n1-highmem-8\").| No, The default is False . |\n",
+        "| `stream_inputs` | boolean argument | Stream input data from GCS instead of downloading it locally - this option is suggested for fast runtime. | No . |\n",
+        "| `large_category_dim` | int | Dimensionality of the embedding - if the number of distinct categories for a categorical column is greater than large_category_thresh we use a large_category_dim dimensional embedding, instead of 1-D embedding. The default is 1. We suggest increasing it (e.g. to ~5 in most cases and even ~10 if the number of categories is typically very large in the dataset), if pushing the accuracy is the main goal, rather than computational efficiency and explainability.  | No . |\n",
+        "| `large_category_thresh` | int | Threshold for categorical column cardinality - if the number of distinct categories for a categorical column is greater than large_category_thresh we use a large_category_dim dimensional embedding, instead of 1-D embedding. The default is 300. We suggest decreasing it (e.g. to ~10), if pushing the accuracy is the main goal, rather than computational efficiency and explainability. | No . |\n",
+        "| `yeo_johnson_transform` | boolean argument | Enables trainable Yeo-Johnson Power Transform (the default is disabled). Please see this link: https://www.stat.umn.edu/arc/yjpower.pdf for more information on Yeo-Johnson Power Transform. With our implementation, the transform parameters are learnable along with TabNet, trained in an end-to-end way. | No . |\n",
+        "| `apply_log_transform` | boolean argument | If log transform statistics are contained in the metadata and this flag is true then the input features will be log transformed. Use false to not use the transform and true (the default) to use it. Especially for datasets with skewed numerical distributions, log transformations could be very helpful.  | No . |\n",
+        "| `apply_quantile_transform` | boolean argument | If quantile statistics are contained in the metadata and this flag is true then the input features will be quantile transformed. Use false to not use the transform and true (the default) to use it. Especially for datasets with skewed numerical distributions, quantile transformations could be very helpful. Currently supported for BigQuery input_type. | No . |\n",
+        "| `replace_transformed_features` | boolean argument | If true then if a transformation is applied to a feature that feature will be replaced. If false (the default option) then the transformed feature 'will be added as a new feature to the list of feature columns. Use true to replace the feature with the transform and false to append the transformed feature as a new column. | No . |\n",
+        "| `target_column` | string | name of the label column. Note that for classification, the label needs to be of String or integer type. | No . |\n",
+        "| `prediction_raw_inputs` | boolean argument | If we set this argument, the model serving allows us to pass the features as a dictionary of tensors instead of CSV row. | No . |\n",
+        "| `exclude_key` | boolean argument | If we set this argument, we exclude a key in the input/output. The key is helpful in batch prediction processes input and saves output in an unpredictable order. The key helps match the output with input. | No . |"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FNmQS6EW86gp"
+      },
+      "outputs": [],
+      "source": [
+        "# URI of the TabNet training Docker image.\n",
+        "LEARNER_IMAGE_URI = 'us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/tab_net_v2'\n",
+        "\n",
+        "DATASET_NAME = \"petfinder\"  # Change to your dataset name.\n",
+        "\n",
+        "ALGORITHM = \"tabnet\"\n",
+        "MODEL_TYPE = \"classification\"\n",
+        "MODEL_NAME = f\"{DATASET_NAME}_{ALGORITHM}_{MODEL_TYPE}\"\n",
+        "\n",
+        "TRAINING_DATA_PATH = f\"{BUCKET_NAME}/data/{DATASET_NAME}/train.csv\"\n",
+        "\n",
+        "print(\"training path: \", TRAINING_DATA_PATH)\n",
+        "print(\"model name: \", MODEL_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "k1gLmZCPnm_s"
+      },
+      "source": [
+        "# Run TabNet training"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7W2rij5opa8R"
+      },
+      "outputs": [],
+      "source": [
+        "current_time = datetime.utcnow().strftime('%y%m%d%H%M%S')\n",
+        "OUTPUT_DIR = f\"{BUCKET_NAME}/{MODEL_NAME}_{current_time}\"\n",
+        "print('Output dir: ', OUTPUT_DIR)\n",
+        "learning_job_name = f'tab_net_{DATASET_NAME}_{current_time}'\n",
+        "print('Job name: ', learning_job_name)\n",
+        "\n",
+        "CREATION_LOG = !gcloud ai custom-jobs create \\\n",
+        "  --region={REGION} \\\n",
+        "  --display-name={learning_job_name} \\\n",
+        "  --worker-pool-spec=machine-type=n1-standard-8,replica-count=1,container-image-uri={LEARNER_IMAGE_URI} \\\n",
+        "  --args=--preprocess \\\n",
+        "  --args=--data_has_header \\\n",
+        "  --args=--training_data_path={TRAINING_DATA_PATH} \\\n",
+        "  --args=--job-dir={OUTPUT_DIR} \\\n",
+        "  --args=--model_type={MODEL_TYPE} \\\n",
+        "  --args=--max_steps=2000 \\\n",
+        "  --args=--batch_size=4096 \\\n",
+        "  --args=--learning_rate=0.01\n",
+        "\n",
+        "print(CREATION_LOG)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "y_GgAlhpGl4q"
+      },
+      "source": [
+        "After the job is submitted successfully, you can view its details and logs:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Lh1ZQ9uRGdk-"
+      },
+      "outputs": [],
+      "source": [
+        "JOB_ID = re.search(r\"(?<=/customJobs/)\\d+\", CREATION_LOG[1]).group(0)\n",
+        "print(JOB_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IShlAxz8piAg"
+      },
+      "outputs": [],
+      "source": [
+        "# View the job's configuration and state.\n",
+        "STATE = \"state: JOB_STATE_PENDING\"\n",
+        "\n",
+        "while STATE not in [\"state: JOB_STATE_SUCCEEDED\", \"state: JOB_STATE_FAILED\"]:\n",
+        "    DESCRIPTION = ! gcloud ai custom-jobs describe {JOB_ID} --region={REGION}\n",
+        "    STATE = DESCRIPTION[-2]\n",
+        "    print(STATE)\n",
+        "    time.sleep(60)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "E9EZvfSUWrxS"
+      },
+      "source": [
+        "# Hyperparameter Tuning\n",
+        "After successfully training your model, deploying it, and calling it to make predictions, you may want to optimize the hyperparameters used during training to improve your model's accuracy and performance. See the Vertex AI documentation for an overview of hyperparameter tuning and how to use it in your Vertex Training jobs.\n",
+        "\n",
+        "For this example, the following command runs a Vertex AI hyperparameter tuning job with 4 trials that attempts to maximize the validation AUC metric. The hyperparameters it optimizes are the number of max_steps and the learning rate."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DnJcD4vBeKtl"
+      },
+      "outputs": [],
+      "source": [
+        "DATASET_NAME = \"petfinder\"  # Change to your dataset name.\n",
+        "current_time = datetime.utcnow().strftime('%y%m%d%H%M%S')\n",
+        "OUTPUT_DIR = f\"{BUCKET_NAME}/hptuning/{MODEL_NAME}_{current_time}/\"\n",
+        "print('Output dir: ', OUTPUT_DIR)\n",
+        "learning_job_name = f'tab_net_{DATASET_NAME}_{current_time}'\n",
+        "print('Job name: ', learning_job_name)\n",
+        "\n",
+        "config = f\"\"\"studySpec:\n",
+        "  metrics:\n",
+        "  - metricId: auc\n",
+        "    goal: MAXIMIZE\n",
+        "  parameters:\n",
+        "  - parameterId: max_steps\n",
+        "    integerValueSpec:\n",
+        "      minValue: 2000\n",
+        "      maxValue: 3000\n",
+        "  - parameterId: learning_rate\n",
+        "    doubleValueSpec:\n",
+        "      minValue: 0.0000001\n",
+        "      maxValue: 0.1\n",
+        "trialJobSpec:\n",
+        "  workerPoolSpecs:\n",
+        "  - machineSpec:\n",
+        "      machineType: n1-highmem-8\n",
+        "      acceleratorType: NVIDIA_TESLA_V100\n",
+        "      acceleratorCount: 1\n",
+        "    replicaCount: 1\n",
+        "    diskSpec:\n",
+        "      bootDiskType: pd-ssd\n",
+        "      bootDiskSizeGb: 100\n",
+        "    containerSpec:\n",
+        "      imageUri: {LEARNER_IMAGE_URI}\n",
+        "      args:\n",
+        "      - --preprocess \n",
+        "      - --data_has_header\n",
+        "      - --training_data_path={TRAINING_DATA_PATH}\n",
+        "      - --job-dir={OUTPUT_DIR}\n",
+        "      - --batch_size=1028\n",
+        "      - --model_type=classification\n",
+        "\"\"\"\n",
+        "\n",
+        "!echo $'{config}' > ./config.yaml\n",
+        "\n",
+        "MAX_TRIAL_COUNT=4\n",
+        "PARALLEL_TRIAL_COUNT=2\n",
+        "\n",
+        "!gcloud ai hp-tuning-jobs create \\\n",
+        "  --config=config.yaml \\\n",
+        "  --max-trial-count={MAX_TRIAL_COUNT} \\\n",
+        "  --parallel-trial-count={PARALLEL_TRIAL_COUNT} \\\n",
+        "  --region=$REGION \\\n",
+        "  --display-name=$learning_job_name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mpKDxrP9yQK8"
+      },
+      "source": [
+        "# Hyperparameter Tuning with Big Query (BQ) input\n",
+        "You need to create the BQ dataset using the above csv file."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2z6gsQQiVQq_"
+      },
+      "source": [
+        "Create BQ dataset to your project\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NXu00dB7VZ8w"
+      },
+      "outputs": [],
+      "source": [
+        "! bq --location={REGION} mk --dataset {PROJECT_ID}:{DATASET_NAME}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0JfQ_Lz_DHUP"
+      },
+      "source": [
+        "Create the BQ table using the csv file."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zv5ZK67LXh_e"
+      },
+      "outputs": [],
+      "source": [
+        "!bq --location={REGION} load --source_format=CSV --autodetect {PROJECT_ID}:{DATASET_NAME}.train {BUCKET_NAME}/data/petfinder/train.csv"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Qsz_ACeWVPb0"
+      },
+      "source": [
+        "Run hp-tuning using BQ input."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pPaqOq6_UGIl"
+      },
+      "outputs": [],
+      "source": [
+        "current_time = datetime.utcnow().strftime('%y%m%d%H%M%S')\n",
+        "\n",
+        "BQ_PROJECT=f\"{PROJECT_ID}\"\n",
+        "DATASET_NAME = \"petfinder\"  # Change to your dataset name.\n",
+        "TABLE_NAME=\"train\"\n",
+        "OUTPUT_DIR = f\"{BUCKET_NAME}/hptuning/{MODEL_NAME}_{current_time}/\"\n",
+        "print('Output dir: ', OUTPUT_DIR)\n",
+        "learning_job_name = f'tab_net_{DATASET_NAME}_{current_time}'\n",
+        "print('Job name: ', learning_job_name)\n",
+        "\n",
+        "config = f\"\"\"studySpec:\n",
+        "  metrics:\n",
+        "  - metricId: auc\n",
+        "    goal: MAXIMIZE\n",
+        "  parameters:\n",
+        "  - parameterId: max_steps\n",
+        "    integerValueSpec:\n",
+        "      minValue: 2000\n",
+        "      maxValue: 3000\n",
+        "  - parameterId: learning_rate\n",
+        "    doubleValueSpec:\n",
+        "      minValue: 0.0000001\n",
+        "      maxValue: 0.1\n",
+        "trialJobSpec:\n",
+        "  workerPoolSpecs:\n",
+        "  - machineSpec:\n",
+        "      machineType: n1-highmem-8\n",
+        "      acceleratorType: NVIDIA_TESLA_V100\n",
+        "      acceleratorCount: 1\n",
+        "    replicaCount: 1\n",
+        "    diskSpec:\n",
+        "      bootDiskType: pd-ssd\n",
+        "      bootDiskSizeGb: 200\n",
+        "    containerSpec:\n",
+        "      imageUri: {LEARNER_IMAGE_URI}\n",
+        "      args:\n",
+        "      - --eval_frequency_secs=10800\n",
+        "      - --input_type=bigquery\n",
+        "      - --preprocess\n",
+        "      - --model_type=classification\n",
+        "      - --stream_inputs\n",
+        "      - --bq_project={BQ_PROJECT}\n",
+        "      - --dataset_name={DATASET_NAME}\n",
+        "      - --table_name={TABLE_NAME}\n",
+        "      - --target_column=Adopted\n",
+        "      - --num_parallel_reads=2\n",
+        "      - --optimizer_type=adam\n",
+        "      - --data_cache=disk\n",
+        "      - --deterministic_data=False\n",
+        "      - --loss_function_type=weighted_cross_entropy\n",
+        "      - --replace_transformed_features=True\n",
+        "      - --apply_quantile_transform=True\n",
+        "      - --apply_log_transform=True\n",
+        "      - --batch_size=32768\n",
+        "      - --job-dir={OUTPUT_DIR}\n",
+        "\"\"\"\n",
+        "\n",
+        "!echo $'{config}' > ./hptuning-config.yaml\n",
+        "\n",
+        "MAX_TRIAL_COUNT=2\n",
+        "PARALLEL_TRIAL_COUNT=2\n",
+        "\n",
+        "!gcloud ai hp-tuning-jobs create \\\n",
+        "  --config=hptuning-config.yaml \\\n",
+        "  --max-trial-count={MAX_TRIAL_COUNT} \\\n",
+        "  --parallel-trial-count={PARALLEL_TRIAL_COUNT} \\\n",
+        "  --region=$REGION \\\n",
+        "  --display-name=$learning_job_name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TpV-iwP9qw9c"
+      },
+      "source": [
+        "## Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sx_vKniMq9ZX"
+      },
+      "outputs": [],
+      "source": [
+        "# Delete Cloud Storage objects that were created\n",
+        "! gsutil -m rm -r $JOB_DIR\n",
+        "\n",
+        "# Delete BQ table\n",
+        "! bq rm -f {PROJECT_ID}:{DATASET_NAME}.train\n",
+        "\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil -m rm -r $BUCKET_URI"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "tabnet_vertex_tutorial.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}