feat: add BML get started

notebooks/community/ml_ops/stage2/get_started_automl_training.ipynb

@@ -100,7 +100,7 @@
       "source": [
         "### Objective\n",
         "\n",
-        "In this tutorial, you learn how to use `AutoML` for when training with `Vertex AI`.\n",
+        "In this tutorial, you learn how to use `AutoML` for training with `Vertex AI`.\n",
         "\n",
         "This tutorial uses the following Google Cloud ML services:\n",
         "\n",

notebooks/community/ml_ops/stage2/get_started_bqml_training.ipynb

@@ -0,0 +1,837 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2021 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 2 : experimentation: get started with BigQuery ML Training\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/ml_ops_stage2/get_started_bqml_training.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/ml_ops_stage2/get_started_bqml_training.ipynb\">\n",
+        "      Open in Google Cloud Notebooks\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with BigQuery ML Training."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:penguins,lcn,bq"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage2,get_started_bqml_training"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use `BigQueryML` (BQML) for training with `Vertex AI`.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `BigQueryML Training`\n",
+        "- `Vertex AI Model resource`\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Create a local BQ table in your project.\n",
+        "- Train a BQML model.\n",
+        "- Evaluate the BQML model.\n",
+        "- Export the BQML model as a cloud model.\n",
+        "- Upload the exported model as a Vertex AI Model resource."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "source": [
+        "## Installations\n",
+        "\n",
+        "Install *one time* the packages for executing the MLOps notebooks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "outputs": [],
+      "source": [
+        "ONCE_ONLY = False\n",
+        "if ONCE_ONLY:\n",
+        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
+        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
+        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
+        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
+        "    ! pip3 install --upgrade kfp $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "project_id"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aip"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_bq"
+      },
+      "source": [
+        "#### Import BigQuery\n",
+        "\n",
+        "Import the BigQuery package into your Python environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_bq"
+      },
+      "outputs": [],
+      "source": [
+        "from google.cloud import bigquery"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_bq"
+      },
+      "source": [
+        "### Create BigQuery client\n",
+        "\n",
+        "Create the BigQuery client."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_bq"
+      },
+      "outputs": [],
+      "source": [
+        "bqclient = bigquery.Client()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "accelerators:prediction,mbsdk"
+      },
+      "source": [
+        "#### Set hardware accelerators\n",
+        "\n",
+        "You can set hardware accelerators for prediction.\n",
+        "\n",
+        "Set the variable `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
+        "\n",
+        "    (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
+        "\n",
+        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
+        "\n",
+        "Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "accelerators:prediction,mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
+        "    DEPLOY_GPU, DEPLOY_NGPU = (\n",
+        "        aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
+        "        int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n",
+        "    )\n",
+        "else:\n",
+        "    DEPLOY_GPU, DEPLOY_NGPU = (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "container:prediction"
+      },
+      "source": [
+        "#### Set pre-built containers\n",
+        "\n",
+        "Set the pre-built Docker container image for prediction.\n",
+        "\n",
+        "- Set the variable `TF` to the TensorFlow version of the container image. For example, `2-1` would be version 2.1, and `1-15` would be version 1.15. The following list shows some of the pre-built images available:\n",
+        "\n",
+        "\n",
+        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "container:prediction"
+      },
+      "outputs": [],
+      "source": [
+        "if os.getenv(\"IS_TESTING_TF\"):\n",
+        "    TF = os.getenv(\"IS_TESTING_TF\")\n",
+        "else:\n",
+        "    TF = \"2.5\".replace(\".\", \"-\")\n",
+        "\n",
+        "if TF[0] == \"2\":\n",
+        "    if DEPLOY_GPU:\n",
+        "        DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
+        "    else:\n",
+        "        DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
+        "else:\n",
+        "    if DEPLOY_GPU:\n",
+        "        DEPLOY_VERSION = \"tf-gpu.{}\".format(TF)\n",
+        "    else:\n",
+        "        DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n",
+        "\n",
+        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
+        "    REGION.split(\"-\")[0], DEPLOY_VERSION\n",
+        ")\n",
+        "\n",
+        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "machine:prediction"
+      },
+      "source": [
+        "#### Set machine type\n",
+        "\n",
+        "Next, set the machine type to use for prediction.\n",
+        "\n",
+        "- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VM you will use for prediction.\n",
+        " - `machine type`\n",
+        "     - `n1-standard`: 3.75GB of memory per vCPU.\n",
+        "     - `n1-highmem`: 6.5GB of memory per vCPU\n",
+        "     - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
+        " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
+        "\n",
+        "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "machine:prediction"
+      },
+      "outputs": [],
+      "source": [
+        "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n",
+        "    MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n",
+        "else:\n",
+        "    MACHINE_TYPE = \"n1-standard\"\n",
+        "\n",
+        "VCPU = \"4\"\n",
+        "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
+        "print(\"Deploy machine type\", DEPLOY_COMPUTE)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_intro"
+      },
+      "source": [
+        "## Bigquery ML introduction\n",
+        "\n",
+        "BigQuery ML (BQML) provides the capability to train ML tabular models, such as classification and regression, in BigQuery using SQL syntax.\n",
+        "\n",
+        "Learn more about [BigQuery ML documentation](https://cloud.google.com/bigquery-ml/docs)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:penguins,bq,lcn"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = \"bq://bigquery-public-data.ml_datasets.penguins\"\n",
+        "BQ_TABLE = \"bigquery-public-data.ml_datasets.penguins\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_create_dataset"
+      },
+      "source": [
+        "### Create BQ dataset/model resource\n",
+        "\n",
+        "First, you create a empty dataset/model resource in your project."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bqml_create_dataset"
+      },
+      "outputs": [],
+      "source": [
+        "BQ_DATASET_NAME = \"penguins\"\n",
+        "DATASET_QUERY = f\"\"\"CREATE SCHEMA {BQ_DATASET_NAME}\n",
+        "\"\"\"\n",
+        "\n",
+        "job = bqclient.query(DATASET_QUERY)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_create_model"
+      },
+      "source": [
+        "### Train BQML model\n",
+        "\n",
+        "Next, you create and train a BQML tabular classification model from the public dataset penguins and store the model in your project.\n",
+        "\n",
+        "Learn more about [The CREATE MODEL statement](https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bqml_create_model"
+      },
+      "outputs": [],
+      "source": [
+        "MODEL_NAME = \"penguins\"\n",
+        "MODEL_QUERY = f\"\"\"\n",
+        "CREATE OR REPLACE MODEL `{BQ_DATASET_NAME}.{MODEL_NAME}`\n",
+        "OPTIONS(\n",
+        "    model_type='DNN_CLASSIFIER',\n",
+        "    labels = ['species']\n",
+        "    )\n",
+        "AS\n",
+        "SELECT *\n",
+        "FROM `{BQ_TABLE}`\n",
+        "\"\"\"\n",
+        "\n",
+        "job = bqclient.query(MODEL_QUERY)\n",
+        "print(job.errors, job.state)\n",
+        "\n",
+        "while job.running():\n",
+        "    from time import sleep\n",
+        "\n",
+        "    sleep(30)\n",
+        "    print(\"Running ...\")\n",
+        "print(job.errors, job.state)\n",
+        "\n",
+        "tblname = job.ddl_target_table\n",
+        "tblname = \"{}.{}\".format(tblname.dataset_id, tblname.table_id)\n",
+        "print(\"{} created in {}\".format(tblname, job.ended - job.started))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_eval_model"
+      },
+      "source": [
+        "### Evaluate the BQML trained model\n",
+        "\n",
+        "Next, retrieve the model evaluation for the trained BQML model.\n",
+        "\n",
+        "Learn more about [The ML.EVALUATE function](https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-evaluate)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bqml_eval_model"
+      },
+      "outputs": [],
+      "source": [
+        "EVAL_QUERY = f\"\"\"\n",
+        "SELECT *\n",
+        "FROM\n",
+        "  ML.EVALUATE(MODEL {BQ_DATASET_NAME}.{MODEL_NAME})\n",
+        "ORDER BY  roc_auc desc\n",
+        "LIMIT 1\"\"\"\n",
+        "\n",
+        "job = bqclient.query(EVAL_QUERY)\n",
+        "results = job.result().to_dataframe()\n",
+        "print(results)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bqml_export_model"
+      },
+      "source": [
+        "### Export the model from BQML\n",
+        "\n",
+        "The model you trained in BQML is a TensorFlow model. Next, you will export the TensorFlow model artifacts in TF.SavedModel format."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bqml_export_model"
+      },
+      "outputs": [],
+      "source": [
+        "param = f\"{PROJECT_ID}:{BQ_DATASET_NAME}.{MODEL_NAME} {BUCKET_NAME}/{MODEL_NAME}\"\n",
+        "! bq extract -m $param\n",
+        "\n",
+        "MODEL_DIR = f\"{BUCKET_NAME}/{BQ_DATASET_NAME}\"\n",
+        "! gsutil ls $MODEL_DIR"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "upload_bqml_model"
+      },
+      "source": [
+        "## Upload the BQML model to a Model resource\n",
+        "\n",
+        "Finally, now that you have the BQML model exported as a TF.SavedModel format, you upload the model artifacts to Vertex AI Model resource, in the same way as if you were uploading a custom trained model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "upload_bqml_model"
+      },
+      "outputs": [],
+      "source": [
+        "bq_model = aip.Model.upload(\n",
+        "    display_name=\"penguins_\" + TIMESTAMP,\n",
+        "    artifact_uri=SERVING_MODEL_DIR,\n",
+        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
+        "    sync=True,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
+        "\n",
+        "- Dataset\n",
+        "- Pipeline\n",
+        "- Model\n",
+        "- Endpoint\n",
+        "- AutoML Training Job\n",
+        "- Batch Job\n",
+        "- Custom Job\n",
+        "- Hyperparameter Tuning Job\n",
+        "- Cloud Storage Bucket"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "delete_all = True\n",
+        "\n",
+        "if delete_all:\n",
+        "    # Delete the dataset using the Vertex dataset object\n",
+        "    try:\n",
+        "        if \"dataset\" in globals():\n",
+        "            dataset.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the model using the Vertex model object\n",
+        "    try:\n",
+        "        if \"model\" in globals():\n",
+        "            model.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the endpoint using the Vertex endpoint object\n",
+        "    try:\n",
+        "        if \"endpoint\" in globals():\n",
+        "            endpoint.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the AutoML or Pipeline training job\n",
+        "    try:\n",
+        "        if \"dag\" in globals():\n",
+        "            dag.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the custom training job\n",
+        "    try:\n",
+        "        if \"job\" in globals():\n",
+        "            job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
+        "    try:\n",
+        "        if \"batch_predict_job\" in globals():\n",
+        "            batch_predict_job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
+        "    try:\n",
+        "        if \"hpt_job\" in globals():\n",
+        "            hpt_job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    if \"BUCKET_NAME\" in globals():\n",
+        "        ! gsutil rm -r $BUCKET_NAME"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "get_started_bqml_training.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/community/ml_ops/stage2/mlops_experimentation.ipynb

@@ -2880,6 +2880,30 @@
         "!gsutil cat $BUCKET_NAME/metadata.jsonl"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
+        "\n",
+        "- Dataset\n",
+        "- Pipeline\n",
+        "- Model\n",
+        "- Endpoint\n",
+        "- AutoML Training Job\n",
+        "- Batch Job\n",
+        "- Custom Job\n",
+        "- Hyperparameter Tuning Job\n",
+        "- Cloud Storage Bucket"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,

notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb

@@ -0,0 +1,873 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2021 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 3 : formalization: get started with AutoML pipeline components\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/ml_ops_stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/ml_ops_stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
+        "      Open in Google Cloud Notebooks\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:flowers,icn"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use prebuilt `Google Cloud Pipeline Components` for `Vertex AI AutoML`.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI Pipelines`\n",
+        "- `Vertex AI AutoML`\n",
+        "- `Google Cloud Pipeline Components`\n",
+        "- `Vertex AI Dataset, Model and Endpoint` resources\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Construct a pipeline for training and deploying a Vertex AI AutoML model.\n",
+        "- Execute a Vertex AI pipeline."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "source": [
+        "## Installations\n",
+        "\n",
+        "Install *one time* the packages for executing the MLOps notebooks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "outputs": [],
+      "source": [
+        "ONCE_ONLY = False\n",
+        "if ONCE_ONLY:\n",
+        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
+        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
+        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
+        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
+        "    ! pip3 install --upgrade kfp $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "project_id"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "set_service_account"
+      },
+      "source": [
+        "#### Service Account\n",
+        "\n",
+        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_service_account"
+      },
+      "outputs": [],
+      "source": [
+        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_service_account"
+      },
+      "outputs": [],
+      "source": [
+        "if (\n",
+        "    SERVICE_ACCOUNT == \"\"\n",
+        "    or SERVICE_ACCOUNT is None\n",
+        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
+        "):\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = !gcloud auth list 2>/dev/null\n",
+        "    SERVICE_ACCOUNT = shell_output[2].strip()\n",
+        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "set_service_account:pipelines"
+      },
+      "source": [
+        "#### Set service account access for Vertex AI Pipelines\n",
+        "\n",
+        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_service_account:pipelines"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
+        "\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aip"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_tf"
+      },
+      "source": [
+        "#### Import TensorFlow\n",
+        "\n",
+        "Import the TensorFlow package into your Python environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_tf"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_kfp"
+      },
+      "outputs": [],
+      "source": [
+        "import json\n",
+        "\n",
+        "from kfp import dsl\n",
+        "from kfp.v2 import compiler\n",
+        "from kfp.v2.dsl import component"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_file:u_dataset,csv"
+      },
+      "source": [
+        "#### Location of Cloud Storage training data.\n",
+        "\n",
+        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:flowers,csv,icn"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = (\n",
+        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\"\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_automl_eval_component"
+      },
+      "source": [
+        "### Create AutoML model evaluation component\n",
+        "\n",
+        "The Vertex AI pre-built pipeline components does not currently have a component for retrieiving the model evaluations for a AutoML model. So, you will first write your own component, as follows:\n",
+        "\n",
+        "- Takes as input the project, region and Model artifacts returned from an AutoML training component.\n",
+        "- Create a client interface to the Vertex AI Model service.\n",
+        "- Construct the resource ID for the model from the model artifact parameter.\n",
+        "- Retrieve the model evaluation\n",
+        "- Return the model evaluation as a string."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_automl_eval_component"
+      },
+      "outputs": [],
+      "source": [
+        "from kfp.v2.dsl import Artifact, Input, Model\n",
+        "\n",
+        "\n",
+        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
+        "def evaluateAutoMLModelOp(model: Input[Artifact], region: str) -> str:\n",
+        "    import logging\n",
+        "\n",
+        "    import google.cloud.aiplatform.gapic as gapic\n",
+        "\n",
+        "    # Get a reference to the Model Service client\n",
+        "    client_options = {\"api_endpoint\": f\"{region}-aiplatform.googleapis.com\"}\n",
+        "    model_service_client = gapic.ModelServiceClient(client_options=client_options)\n",
+        "\n",
+        "    model_id = model.metadata[\"resourceName\"]\n",
+        "\n",
+        "    model_evaluations = model_service_client.list_model_evaluations(parent=model_id)\n",
+        "    model_evaluation = list(model_evaluations)[0]\n",
+        "    logging.info(model_evaluation)\n",
+        "    return str(model_evaluation)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_automl_pipeline:automl,icn"
+      },
+      "source": [
+        "## Construct AutoML Training Pipeline\n",
+        "\n",
+        "In the example below, you construct a pipeline for training an AutoML model using pre-built Google Cloud Pipeline Components for AutoML, as follows:\n",
+        "\n",
+        "1. Use the prebuilt component `ImageDatasetCreateOp` to create a Vertex AI Dataset resource, where:\n",
+        "    - The display name for the dataset is passed into the pipeline.\n",
+        "    - The import file for the dataset is passed into the pipeline.\n",
+        "    - The component returns the dataset resource as `outputs[\"dataset\"]`\n",
+        "2. Use the prebuilt component `AutoMLImageTrainingJobRunOp` to train a Vertex AI AutoML Model resource, where:\n",
+        "    - The display name for the dataset is passed into the pipeline.\n",
+        "    - The dataset is the output from the `ImageDatasetCreateOp`.\n",
+        "3. Use the prebuilt component `EndpointCreateOp` to create a Vertex AI Endpoint to deploy the trained model to, where:\n",
+        "    - Since the component has no dependencies on other components, by default it would be executed in parallel with the model training.\n",
+        "    - The `after(training_op)` is added to serialize its execution, so its only executed if the training operation completes successfully.\n",
+        "4. Use the prebuilt component `ModelDeployOp` to deploy the trained AutoML model to, where:\n",
+        "    - The display name for the dataset is passed into the pipeline.\n",
+        "    - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
+        "    - The endpoint is the output from the `EndpointCreateOp`\n",
+        "\n",
+        "*Note:* Since each component is executed as a graph node in its own execution context, you pass the parameter `project` for each component op, in constrast to doing a `aip.init(project=project)` if this was a Python script calling the SDK methods directly within the same execution context."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_automl_pipeline:automl,icn"
+      },
+      "outputs": [],
+      "source": [
+        "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
+        "\n",
+        "PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_NAME)\n",
+        "\n",
+        "\n",
+        "@dsl.pipeline(\n",
+        "    name=\"automl-icn-training\", description=\"AutoML image classification training\"\n",
+        ")\n",
+        "def pipeline(\n",
+        "    import_file: str, display_name: str, project: str = PROJECT_ID, region: str = REGION\n",
+        "):\n",
+        "\n",
+        "    dataset_op = gcc_aip.ImageDatasetCreateOp(\n",
+        "        project=project,\n",
+        "        display_name=display_name,\n",
+        "        gcs_source=import_file,\n",
+        "        import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
+        "    )\n",
+        "\n",
+        "    training_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
+        "        project=project,\n",
+        "        display_name=display_name,\n",
+        "        prediction_type=\"classification\",\n",
+        "        model_type=\"CLOUD\",\n",
+        "        base_model=None,\n",
+        "        dataset=dataset_op.outputs[\"dataset\"],\n",
+        "        model_display_name=display_name,\n",
+        "        training_fraction_split=0.6,\n",
+        "        validation_fraction_split=0.2,\n",
+        "        test_fraction_split=0.2,\n",
+        "        budget_milli_node_hours=8000,\n",
+        "    )\n",
+        "\n",
+        "    eval_op = evaluateAutoMLModelOp(model=training_op.outputs[\"model\"], region=region)\n",
+        "\n",
+        "    endpoint_op = gcc_aip.EndpointCreateOp(\n",
+        "        project=project,\n",
+        "        location=region,\n",
+        "        display_name=display_name,\n",
+        "    ).after(eval_op)\n",
+        "\n",
+        "    deploy_op = gcc_aip.ModelDeployOp(\n",
+        "        model=training_op.outputs[\"model\"],\n",
+        "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
+        "        automatic_resources_min_replica_count=1,\n",
+        "        automatic_resources_max_replica_count=1,\n",
+        "    )"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "run_automl_pipeline:automl,icn"
+      },
+      "source": [
+        "### Compile and execute the pipeline\n",
+        "\n",
+        "Next, you compile the pipeline and then exeute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:\n",
+        "\n",
+        "- `import_file`: The Cloud Storage path to the dataset index file.\n",
+        "- `display_name`: The display name for the generated Vertex AI resources.\n",
+        "- `project`: The project ID.\n",
+        "- `region`: The region."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "run_automl_pipeline:automl,icn"
+      },
+      "outputs": [],
+      "source": [
+        "compiler.Compiler().compile(\n",
+        "    pipeline_func=pipeline, package_path=\"automl_icn_training.json\"\n",
+        ")\n",
+        "\n",
+        "pipeline = aip.PipelineJob(\n",
+        "    display_name=\"automl_icn_training\",\n",
+        "    template_path=\"automl_icn_training.json\",\n",
+        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    parameter_values={\n",
+        "        \"import_file\": IMPORT_FILE,\n",
+        "        \"display_name\": \"flowers\" + TIMESTAMP,\n",
+        "        \"project\": PROJECT_ID,\n",
+        "        \"region\": REGION,\n",
+        "    },\n",
+        ")\n",
+        "\n",
+        "pipeline.run()\n",
+        "\n",
+        "! rm -f automl_icn_training.json"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view_pipeline_results:automl,icn"
+      },
+      "source": [
+        "### View AutoML training pipeline results\n",
+        "\n",
+        "Finally, you will view the artifact outputs of each task in the pipeline."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "view_pipeline_results:automl,icn"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_NUMBER = pipeline.gca_resource.name.split(\"/\")[1]\n",
+        "print(PROJECT_NUMBER)\n",
+        "\n",
+        "\n",
+        "def print_pipeline_output(job, output_task_name):\n",
+        "    JOB_ID = job.name\n",
+        "    print(JOB_ID)\n",
+        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
+        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
+        "        EXECUTE_OUTPUT = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/executor_output.json\"\n",
+        "        )\n",
+        "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
+        "            ! gsutil cat $EXECUTE_OUTPUT\n",
+        "            break\n",
+        "\n",
+        "    return EXECUTE_OUTPUT\n",
+        "\n",
+        "\n",
+        "print(\"imagedataset-create\")\n",
+        "artifacts = print_pipeline_output(pipeline, \"imagedataset-create\")\n",
+        "print(\"\\n\")\n",
+        "print(\"automlimagetrainingjob-run\")\n",
+        "artifacts = print_pipeline_output(pipeline, \"automlimagetrainingjob-run\")\n",
+        "print(\"\\n\")\n",
+        "print(\"endpoint-create\")\n",
+        "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
+        "print(\"\\n\")\n",
+        "print(\"model-deploy\")\n",
+        "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
+        "print(\"\\n\")\n",
+        "print(\"evaluateautomlmodelop\")\n",
+        "artifacts = print_pipeline_output(pipeline, \"evaluateautomlmodelop\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "source": [
+        "### Delete a pipeline job\n",
+        "\n",
+        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "outputs": [],
+      "source": [
+        "pipeline.delete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
+        "\n",
+        "- Dataset\n",
+        "- Pipeline\n",
+        "- Model\n",
+        "- Endpoint\n",
+        "- AutoML Training Job\n",
+        "- Batch Job\n",
+        "- Custom Job\n",
+        "- Hyperparameter Tuning Job\n",
+        "- Cloud Storage Bucket"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "delete_all = True\n",
+        "\n",
+        "if delete_all:\n",
+        "    # Delete the dataset using the Vertex dataset object\n",
+        "    try:\n",
+        "        if \"dataset\" in globals():\n",
+        "            dataset.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the model using the Vertex model object\n",
+        "    try:\n",
+        "        if \"model\" in globals():\n",
+        "            model.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the endpoint using the Vertex endpoint object\n",
+        "    try:\n",
+        "        if \"endpoint\" in globals():\n",
+        "            endpoint.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the AutoML or Pipeline training job\n",
+        "    try:\n",
+        "        if \"dag\" in globals():\n",
+        "            dag.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the custom training job\n",
+        "    try:\n",
+        "        if \"job\" in globals():\n",
+        "            job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
+        "    try:\n",
+        "        if \"batch_predict_job\" in globals():\n",
+        "            batch_predict_job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
+        "    try:\n",
+        "        if \"hpt_job\" in globals():\n",
+        "            hpt_job.delete()\n",
+        "    except Exception as e:\n",
+        "        print(e)\n",
+        "\n",
+        "    if \"BUCKET_NAME\" in globals():\n",
+        "        ! gsutil rm -r $BUCKET_NAME"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "get_started_with_automl_pipeline_components.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

notebooks/community/ml_ops/stage3/get_started_with_bq_tfdv_pipeline_components.ipynb

@@ -447,50 +447,6 @@
         "import tensorflow as tf"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_bq"
-      },
-      "source": [
-        "#### Import BigQuery\n",
-        "\n",
-        "Import the BigQuery package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_bq"
-      },
-      "outputs": [],
-      "source": [
-        "from google.cloud import bigquery"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_beam"
-      },
-      "source": [
-        "#### Import Apache Beam\n",
-        "\n",
-        "Import the Apache Beam package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_beam"
-      },
-      "outputs": [],
-      "source": [
-        "import apache_beam as beam"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -783,9 +739,6 @@
       },
       "outputs": [],
       "source": [
-        "from tensorflow_metadata.proto.v0 import statistics_pb2\n",
-        "\n",
-        "\n",
         "@component(\n",
         "    packages_to_install=[\n",
         "        \"google-cloud-aiplatform\",\n",
@@ -796,7 +749,7 @@
         ")\n",
         "def statistics(\n",
         "    dataset_id: str, label: str, bucket: str\n",
-        ") -> NamedTuple(\"Outputs\", [(\"stats\", str), (\"schema\", str),]):  # Return parameters\n",
+        ") -> NamedTuple(\"Outputs\", [(\"stats\", str), (\"schema\", str)]):  # Return parameters\n",
         "    import google.cloud.aiplatform as aip\n",
         "    import tensorflow_data_validation as tfdv\n",
         "    from google.cloud import bigquery\n",
@@ -846,9 +799,7 @@
         "\n",
         "\n",
         "@dsl.pipeline(\n",
-        "    name=\"dataset-stats\",\n",
-        "    description=\"Dataset statistics\",\n",
-        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    name=\"dataset-stats\", description=\"Dataset statistics\", pipeline_root=PIPELINE_ROOT\n",
         ")\n",
         "def pipeline(dataset_id: str, label: str, bucket: str):\n",
         "\n",

notebooks/community/ml_ops/stage3/get_started_with_dataflow_pipeline_components.ipynb

@@ -937,17 +937,6 @@
         "! gsutil cp requirements.txt $GCS_REQUIREMENTS_TXT"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "2a0b90e62544"
-      },
-      "outputs": [],
-      "source": [
-        "! gsutil ls $GCS_REQUIREMENTS_TXT"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -963,10 +952,11 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "b191fe000ca8"
+        "id": "import_file:gsod,bq,lrg"
       },
       "outputs": [],
       "source": [
+        "IMPORT_FILE = \"bq://bigquery-public-data.samples.gsod\"\n",
         "BQ_TABLE = \"bigquery-public-data.samples.gsod\""
       ]
     },
@@ -987,7 +977,9 @@
         "- `requirements_file_path`: The required Python modules to install.\n",
         "- `args`: The arguments to pass to the Apache Beam pipeline.\n",
         "\n",
-        "Learn more about [Google Cloud Pipeline Component for Dataflow](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-0.2.0/google_cloud_pipeline_components.experimental.dataflow.html)"
+        "Learn more about [Google Cloud Pipeline Component for Dataflow](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-0.2.0/google_cloud_pipeline_components.experimental.dataflow.html)\n",
+        "\n",
+        "Additional, you add `--requirements_file` to the input args, such that the workers -- i.e., WaitGcpResourcesOp -- share the same pip installation requirements as the Dataflow component."
       ]
     },
     {
@@ -1012,13 +1004,12 @@
         "        BUCKET_NAME,\n",
         "        \"--bq_table\",\n",
         "        BQ_TABLE,\n",
-        "        #'--requirements_file', GCS_REQUIREMENTS_TXT\n",
+        "        \"--requirements_file\",\n",
+        "        GCS_REQUIREMENTS_TXT,\n",
         "    ],\n",
         "    requirements_file_path: str = GCS_REQUIREMENTS_TXT,\n",
         "):\n",
-        "    DataflowPythonJobOp.component_spec.implementation.container.image = (\n",
-        "        \"gcr.io/ml-pipeline/google-cloud-pipeline-components:v0.2.0_dataflow_logs_fix\"\n",
-        "    )\n",
+        "    # DataflowPythonJobOp.component_spec.implementation.container.image = \"gcr.io/ml-pipeline/google-cloud-pipeline-components:v0.2.0_dataflow_logs_fix\"\n",
         "    dataflow_python_op = DataflowPythonJobOp(\n",
         "        project=project_id,\n",
         "        location=location,\n",

notebooks/community/ml_ops/stage3/get_started_with_kubeflow_pipelines.ipynb

@@ -436,28 +436,6 @@
         "import tensorflow as tf"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "import_numpy"
-      },
-      "source": [
-        "#### Import numpy\n",
-        "\n",
-        "Import the numpy package into your Python environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "import_numpy"
-      },
-      "outputs": [],
-      "source": [
-        "import numpy as np"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -839,9 +817,7 @@
         "\n",
         "\n",
         "@dsl.pipeline(\n",
-        "    name=\"numpy\",\n",
-        "    description=\"A simple intro pipeline\",\n",
-        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    name=\"numpy\", description=\"A simple intro pipeline\", pipeline_root=PIPELINE_ROOT\n",
         ")\n",
         "def pipeline(values: list = [2, 3]):\n",
         "    numpy_task = numpy_mean(values)\n",
@@ -919,9 +895,7 @@
         "\n",
         "\n",
         "@dsl.pipeline(\n",
-        "    name=\"add-div2\",\n",
-        "    description=\"A simple intro pipeline\",\n",
-        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    name=\"add-div2\", description=\"A simple intro pipeline\", pipeline_root=PIPELINE_ROOT\n",
         ")\n",
         "def pipeline(v1: int = 4, v2: int = 5):\n",
         "    add_task = add(v1, v2)\n",
@@ -1107,9 +1081,7 @@
         "\n",
         "\n",
         "@dsl.pipeline(\n",
-        "    name=\"parallel\",\n",
-        "    description=\"A simple intro pipeline\",\n",
-        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    name=\"parallel\", description=\"A simple intro pipeline\", pipeline_root=PIPELINE_ROOT\n",
         ")\n",
         "def pipeline(values: list = [1, 2, 3]):\n",
         "    add_list_task = add_list(values)\n",
@@ -1284,9 +1256,7 @@
         "\n",
         "\n",
         "@dsl.pipeline(\n",
-        "    name=\"condition\",\n",
-        "    description=\"A simple intro pipeline\",\n",
-        "    pipeline_root=PIPELINE_ROOT,\n",
+        "    name=\"condition\", description=\"A simple intro pipeline\", pipeline_root=PIPELINE_ROOT\n",
         ")\n",
         "def pipeline():\n",
         "    flip_task = flip()\n",