worked on vertex-pipelines-ml-metadata notebook (#826)

* ran linter test

* Minor changes done

* ran linter test

notebooks/official/automl/sdk_automl_tabular_forecasting_batch.ipynb

@@ -786,7 +786,7 @@
         "    PROJECT_ID, batch_predict_bq_output_dataset_name\n",
         ")\n",
         "# Must be the same region as batch_predict_bq_input_uri\n",
-        "client = bigquery.Client()\n",
+        "client = bigquery.Client(project=PROJECT_ID)\n",
         "bq_dataset = bigquery.Dataset(batch_predict_bq_output_dataset_path)\n",
         "dataset_region = \"US\"  # @param {type : \"string\"}\n",
         "bq_dataset.location = dataset_region\n",

notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb

@@ -273,7 +273,7 @@
         "\n",
         "if not os.getenv(\"IS_TESTING\"):\n",
         "    # Get your Google Cloud project ID from gcloud\n",
-        "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
         "    PROJECT_ID = shell_output[0]\n",
         "    print(\"Project ID: \", PROJECT_ID)"
       ]
@@ -674,7 +674,7 @@
         "NA_VALUES = [\"NA\", \".\"]\n",
         "\n",
         "# Set up BigQuery clients\n",
-        "bqclient = bigquery.Client()\n",
+        "bqclient = bigquery.Client(project=PROJECT_ID)\n",
         "\n",
         "\n",
         "# Download a table\n",

notebooks/official/ml_metadata/vertex-pipelines-ml-metadata.ipynb

@@ -23,6 +23,15 @@
         "# limitations under the License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8a053078cf68"
+      },
+      "source": [
+        "# Vertex AI: Track artifacts and metrics across vertex pipelines runs using vertex ml metadata"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -42,18 +51,16 @@
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/ml_metadata/vertex-pipelines-ml-metadata.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "    \n",
         "</table>"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "WBFL9LagqmwT"
-      },
-      "source": [
-        "#Vertex AI: Track artifacts and metrics across Vertex Pipelines runs using Vertex ML Metadata"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -62,12 +69,15 @@
       "source": [
         "## Overview\n",
         "\n",
-        "This notebook demonstrates how to track metrics and artifacts across Vertex Pipelines runs, and analyze this metadata using the Vertex AI SDK. If you'd prefer to follow a step-by-step tutorial, check out the [codelab version](https://codelabs.developers.google.com/vertex-mlmd-pipelines#0) of this notebook.\n",
-        "\n",
-        "### Dataset\n",
-        "\n",
-        "In this notebook, we will train a model using scikit-learn to classify bean types using the [Dry Beans Dataset](https://archive.ics.uci.edu/ml/datasets/Dry+Bean+Dataset) from UCI Machine Learning. This is a tabular dataset that includes measurements and characteristics of seven different types of beans taken from images.\n",
-        "\n",
+        "This notebook demonstrates how to track metrics and artifacts across Vertex Pipelines runs, and analyze this metadata using the Vertex AI SDK. If you'd prefer to follow a step-by-step tutorial, check out the [codelab version](https://codelabs.developers.google.com/vertex-mlmd-pipelines#0) of this notebook.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4f5bffc6f639"
+      },
+      "source": [
         "### Objective\n",
         "\n",
         "In this notebook, you will learn how to:\n",
@@ -77,17 +87,33 @@
         "* Write custom pipeline components that generate artifacts and metadata\n",
         "* Compare Vertex Pipelines runs, both in the Cloud console and programmatically\n",
         "* Trace the lineage for pipeline-generated artifacts\n",
-        "* Query your pipeline run metadata\n",
+        "* Query your pipeline run metadata"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5c47d74cd689"
+      },
+      "source": [
+        "### Dataset\n",
         "\n",
+        "In this notebook, we will train a model using scikit-learn to classify bean types using the [Dry Beans Dataset](https://archive.ics.uci.edu/ml/datasets/Dry+Bean+Dataset) from UCI Machine Learning. This is a tabular dataset that includes measurements and characteristics of seven different types of beans taken from images.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "de76bb18c85b"
+      },
+      "source": [
         "### Costs \n",
         "\n",
-        "\n",
         "This tutorial uses billable components of Google Cloud:\n",
         "\n",
         "* Vertex AI\n",
         "* Cloud Storage\n",
         "\n",
-        "\n",
         "Learn about [Vertex AI\n",
         "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
         "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
@@ -98,21 +124,14 @@
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "ze4-nDLfK4pw"
+        "id": "gCuSR8GkAgzl"
       },
       "source": [
         "### Set up your local development environment\n",
         "\n",
-        "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
-        "all the requirements to run this notebook. You can skip this step."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "gCuSR8GkAgzl"
-      },
-      "source": [
+        "**If you are using Colab or  Vertex AI Workbench Notebooks**, your environment already meets\n",
+        "all the requirements to run this notebook. You can skip this step.\n",
+        "\n",
         "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
         "You need the following:\n",
         "\n",
@@ -165,34 +184,19 @@
       "source": [
         "import os\n",
         "\n",
-        "# The Google Cloud Notebook product has specific requirements\n",
-        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "# The Vertex AI Workbench Notebook product has specific requirements\n",
+        "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
+        "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
+        "    \"/opt/deeplearning/metadata/env_version\"\n",
+        ")\n",
         "\n",
-        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
+        "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
         "USER_FLAG = \"\"\n",
-        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
-        "    USER_FLAG = \"--user\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "MCQDRsnE3uzz"
-      },
-      "source": [
-        "Install Vertex AI and Kubeflow Pipelines SDKs."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "wyy5Lbnzg5fi"
-      },
-      "outputs": [],
-      "source": [
-        "!pip3 install {USER_FLAG} google-cloud-aiplatform==1.7.0\n",
-        "!pip3 install {USER_FLAG} kfp==1.8.9"
+        "if IS_WORKBENCH_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\"\n",
+        "\n",
+        "! pip3 install {USER_FLAG} google-cloud-aiplatform -q\n",
+        "! pip3 install {USER_FLAG} kfp -q"
       ]
     },
     {
@@ -225,21 +229,13 @@
         "    app.kernel.do_shutdown(True)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "lWEdiXsJg0XY"
-      },
-      "source": [
-        "## Before you begin"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "BF1j6f9HApxa"
       },
       "source": [
+        "## Before you begin\n",
         "### Set up your Google Cloud project\n",
         "\n",
         "**The following steps are required, regardless of your notebook environment.**\n",
@@ -261,21 +257,14 @@
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "e86205a30eb4"
+        "id": "28f75ab2b551"
       },
       "source": [
         "### Authenticate your Google Cloud account\n",
         "\n",
-        "**If you are using AI Platform Notebooks**, your environment is already\n",
-        "authenticated. Skip this step."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "28f75ab2b551"
-      },
-      "source": [
+        "**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
+        "authenticated. Skip this step.\n",
+        "\n",
         "**If you are using Colab**, run the cell below and follow the instructions\n",
         "when prompted to authenticate your account via oAuth.\n",
         "\n",
@@ -300,6 +289,38 @@
         "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "a07454a4fd36"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "# If on Google Cloud Notebooks, then don't execute this code\n",
+        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "\n",
+        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "        !gcloud config set project $PROJECT_ID\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -351,38 +372,6 @@
         "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "fdb343d2ae3c"
-      },
-      "outputs": [],
-      "source": [
-        "import sys\n",
-        "\n",
-        "# If you are running this notebook in Colab, run this cell and follow the\n",
-        "# instructions to authenticate your GCP account. This provides access to your\n",
-        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
-        "# requests.\n",
-        "\n",
-        "# If on Google Cloud Notebooks, then don't execute this code\n",
-        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
-        "\n",
-        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
-        "    if \"google.colab\" in sys.modules:\n",
-        "        from google.colab import auth as google_auth\n",
-        "\n",
-        "        google_auth.authenticate_user()\n",
-        "        !gcloud config set project $PROJECT_ID\n",
-        "\n",
-        "    # If you are running this notebook locally, replace the string below with the\n",
-        "    # path to your service account key and run this cell to authenticate your GCP\n",
-        "    # account.\n",
-        "    elif not os.getenv(\"IS_TESTING\"):\n",
-        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -413,9 +402,9 @@
         "id": "06571eb4063b"
       },
       "source": [
-        "#### Timestamp\n",
+        "#### UUID\n",
         "\n",
-        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
       ]
     },
     {
@@ -426,27 +415,88 @@
       },
       "outputs": [],
       "source": [
-        "from datetime import datetime\n",
+        "import random\n",
+        "import string\n",
         "\n",
-        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+        "\n",
+        "# Generate a uuid of a specifed length(default=8)\n",
+        "def generate_uuid(length: int = 8) -> str:\n",
+        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
+        "\n",
+        "\n",
+        "UUID = generate_uuid()"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "641c8b2873c0"
+        "id": "6f8e3989eeef"
       },
       "source": [
-        "### Create a Cloud Storage Bucket"
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5146e8944cdc"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "68e74d218ea9"
+        "id": "009b87d7e1a6"
       },
       "source": [
-        "To run our Vertex Pipeline, we'll need a storage bucket to store artifacts generated by our pipeline. This bucket needs to be regional. We're using the `us-central1` region here, but you are welcome to use another region (just replace it throughout this lab). If you already have a bucket you can replace the `BUCKET_NAME` variable with the name of your bucket and skip the `gsutil mb` step."
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "df166dd1fa7f"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6de8e05b0a14"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
+        "    BUCKET_URI = \"gs://\" + BUCKET_NAME"
       ]
     },
     {
@@ -457,8 +507,7 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"gs://{}-bucket\".format(PROJECT_ID)\n",
-        "!gsutil mb -l us-central1 $BUCKET_NAME # You only need to run this once"
+        "! gcloud config set project $PROJECT_ID"
       ]
     },
     {
@@ -467,9 +516,87 @@
         "id": "b75e9153c699"
       },
       "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "05a17eb2c793"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5cb19ca24e84"
+      },
+      "source": [
+        "#### Service Account\n",
+        "\n",
+        "If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID\n",
+        "\n",
         "Next, make sure your compute service account has `store.objectAdmin` access to this bucket. Your compute service account will look something like `YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com`."
       ]
     },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "034c005865b1"
+      },
+      "outputs": [],
+      "source": [
+        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5a208288d75d"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "IS_COLAB = \"google.colab\" in sys.modules\n",
+        "\n",
+        "if (\n",
+        "    SERVICE_ACCOUNT == \"\"\n",
+        "    or SERVICE_ACCOUNT is None\n",
+        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
+        "):\n",
+        "    # Get your service account from gcloud\n",
+        "    if not IS_COLAB:\n",
+        "        shell_output = !gcloud auth list 2>/dev/null\n",
+        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
+        "\n",
+        "    else:  # IS_COLAB:\n",
+        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
+        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
+        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
+        "\n",
+        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "749c598c5f5d"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
+        "\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -512,7 +639,7 @@
         "id": "xtXZWmYqJ1bh"
       },
       "source": [
-        "Define some constants"
+        "### Import libraries and define constants"
       ]
     },
     {
@@ -527,7 +654,7 @@
         "%env PATH={PATH}:/home/jupyter/.local/bin\n",
         "REGION = \"us-central1\"\n",
         "\n",
-        "PIPELINE_ROOT = f\"{BUCKET_NAME}/pipeline_root/\"\n",
+        "PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/\"\n",
         "PIPELINE_ROOT"
       ]
     },
@@ -613,28 +740,21 @@
       "source": [
         "## Creating a 3-step pipeline with custom components\n",
         "\n",
-        "The focus of this lab is on understanding *metadata* from pipeline runs. In order to do that, we'll need a pipeline to run on Vertex Pipelines, which is where we'll start. Here we'll define a 3-step pipeline with the following custom components:\n",
+        "The focus of this tutorial is on understanding *metadata* from pipeline runs. In order to do that, we'll need a pipeline to run on  Vertex AI Pipelines, which is where we'll start. Here you define a 3-step pipeline with the following custom components:\n",
         "\n",
         "* `get_dataframe`: Retrieve data from a BigQuery table and convert it into a pandas DataFrame\n",
         "* `train_sklearn_model`: Use the pandas DataFrame to train and export a scikit-learn model, along with some metrics\n",
         "* `deploy_model`: Deploy the exported scikit-learn model to an endpoint in Vertex AI"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "KPY41M9_AhZU"
-      },
-      "source": [
-        "### Create and define Python function based components"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "bfMQSmRuUuX-"
       },
       "source": [
+        "### Create and define Python function based components\n",
+        "\n",
         "First, define the `get_dataframe` component with the code below. This component does the following:\n",
         "* Creates a reference to a BigQuery table using the BigQuery client library\n",
         "* Downloads the BigQuery table and converts it to a shuffled pandas DataFrame\n",
@@ -650,14 +770,14 @@
       "outputs": [],
       "source": [
         "@component(\n",
-        "    packages_to_install=[\"google-cloud-bigquery\", \"pandas\", \"pyarrow\"],\n",
+        "    packages_to_install=[\"google-cloud-bigquery\", \"pandas\", \"pyarrow\", \"db-dtypes\"],\n",
         "    base_image=\"python:3.9\",\n",
         "    output_component_file=\"create_dataset.yaml\",\n",
         ")\n",
-        "def get_dataframe(bq_table: str, output_data_path: OutputPath(\"Dataset\")):\n",
+        "def get_dataframe(bq_table: str, output_data_path: OutputPath(\"Dataset\"), project: str):\n",
         "    from google.cloud import bigquery\n",
         "\n",
-        "    bqclient = bigquery.Client()\n",
+        "    bqclient = bigquery.Client(project=project)\n",
         "    table = bigquery.TableReference.from_string(bq_table)\n",
         "    rows = bqclient.list_rows(table)\n",
         "    dataframe = rows.to_dataframe(\n",
@@ -667,6 +787,15 @@
         "    dataframe.to_csv(output_data_path)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0c1a8a74f881"
+      },
+      "source": [
+        "### Import Data and define model"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -718,6 +847,15 @@
         "    dump(skmodel, model.path + \".joblib\")"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a980c5e4585f"
+      },
+      "source": [
+        "### Train Model and deploy it to an endpoint"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -769,7 +907,9 @@
         "id": "UBXUgxgqA_GB"
       },
       "source": [
-        "### Define and compile the pipeline"
+        "### Define and compile the pipeline\n",
+        "\n",
+        "Now that you have the pipeline, you define its training configuration depending on the defined parameters."
       ]
     },
     {
@@ -792,7 +932,7 @@
         "    project: str = PROJECT_ID,\n",
         "    region: str = REGION,\n",
         "):\n",
-        "    dataset_task = get_dataframe(bq_table)\n",
+        "    dataset_task = get_dataframe(bq_table=bq_table, project=project)\n",
         "\n",
         "    model_task = sklearn_train(dataset_task.output)\n",
         "\n",
@@ -805,7 +945,8 @@
         "id": "910541af051c"
       },
       "source": [
-        "The following will generate a JSON file that you'll use to run the pipeline:"
+        "### Generate a json file for pipeline\n",
+        "The following will generate a JSON file that you use to run the pipeline:"
       ]
     },
     {
@@ -819,37 +960,19 @@
         "compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"mlmd_pipeline.json\")"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "u-iTnzt3B6Z_"
-      },
-      "source": [
-        "### Start two pipeline runs\n",
-        "\n",
-        "Next we'll kick off **two** runs of our pipeline. First let's define a timestamp to use for our pipeline job IDs:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "i2wnpu8_7JfV"
-      },
-      "outputs": [],
-      "source": [
-        "from datetime import datetime\n",
-        "\n",
-        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "3d380ed72490"
       },
       "source": [
-        "Our pipeline takes one parameter when we run it: the `bq_table` we want to use for training data. This pipeline run will use a smaller version of the beans dataset:"
+        "### Start of two pipeline runs\n",
+        "\n",
+        "Next you kick off two runs of our pipeline.\n",
+        "\n",
+        "Our pipeline takes one parameter when we run it: the `bq_table` we want to use for training data. This pipeline run will use a smaller version of the beans dataset.\n",
+        "\n",
+        "It takes json template path, job_id and dataset details as parameters."
       ]
     },
     {
@@ -863,7 +986,7 @@
         "run1 = pipeline_jobs.PipelineJob(\n",
         "    display_name=\"mlmd-pipeline\",\n",
         "    template_path=\"mlmd_pipeline.json\",\n",
-        "    job_id=\"mlmd-pipeline-small-{}\".format(TIMESTAMP),\n",
+        "    job_id=\"mlmd-pipeline-small-{}\".format(UUID),\n",
         "    parameter_values={\"bq_table\": \"sara-vertex-demos.beans_demo.small_dataset\"},\n",
         "    enable_caching=True,\n",
         ")"
@@ -875,6 +998,7 @@
         "id": "555ac88a22cf"
       },
       "source": [
+        "### Define pipeline using a larger version of dataset \n",
         "Next, create another pipeline run using a larger version of the same dataset."
       ]
     },
@@ -889,7 +1013,7 @@
         "run2 = pipeline_jobs.PipelineJob(\n",
         "    display_name=\"mlmd-pipeline\",\n",
         "    template_path=\"mlmd_pipeline.json\",\n",
-        "    job_id=\"mlmd-pipeline-large-{}\".format(TIMESTAMP),\n",
+        "    job_id=\"mlmd-pipeline-large-{}\".format(UUID),\n",
         "    parameter_values={\"bq_table\": \"sara-vertex-demos.beans_demo.large_dataset\"},\n",
         "    enable_caching=True,\n",
         ")"
@@ -901,6 +1025,8 @@
         "id": "5670722f7668"
       },
       "source": [
+        "### Execution of pipeline runs\n",
+        "\n",
         "Finally, kick off pipeline executions for both runs. It's best to do this in two separate notebook cells so you can see the output for each run."
       ]
     },
@@ -946,33 +1072,19 @@
         "**These pipeline runs will take 10-15 minutes to complete.**"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "jZLrJZTfL7tE"
-      },
-      "source": [
-        "## Comparing pipeline runs"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
         "id": "A1PqKxlpOZa2"
       },
       "source": [
-        "Now that you have two pipeline completed pipeline runs, we're ready to take a closer look at pipeline metrics using the Vertex AI SDK.\n",
+        "## Comparing pipeline runs\n",
         "\n",
-        "**For guidance on inspecting pipeline artifacts and metadata in the Vertex AI Console, see [this codelab](https://codelabs.developers.google.com/vertex-mlmd-pipelines#5).**"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "jbRf1WoH_vbY"
-      },
-      "source": [
-        "You can use the `aiplatform.get_pipeline_df()` method to access run metadata. Here, we'll get metadata for the last two runs of the same pipeline and load it into a Pandas DataFrame. The `mlmd-pipeline` parameter here refers to the name we gave our pipeline in our pipeline definition:"
+        "Now that you have two pipeline completed pipeline runs, you're ready to take a closer look at pipeline metrics using the Vertex AI SDK.\n",
+        "\n",
+        "**For guidance on inspecting pipeline artifacts and metadata in the Vertex AI Console, see [this codelab](https://codelabs.developers.google.com/vertex-mlmd-pipelines#5).**\n",
+        "\n",
+        "You can use the `aiplatform.get_pipeline_df()` method to access run metadata. Here, you'll get metadata for the last two runs of the same pipeline and load it into a Pandas DataFrame. The `mlmd-pipeline` parameter here refers to the name you gave our pipeline in our pipeline definition:"
       ]
     },
     {
@@ -993,7 +1105,7 @@
         "id": "d23e2cb66265"
       },
       "source": [
-        "We've only executed our pipeline twice here, but you can imagine how many metrics you'd have with more executions. Next, we'll create a custom visualization with matplotlib to see the relationship between our model's accuracy and the amount of data used for training. Run the following to generate a graph:"
+        "you've only executed our pipeline twice here, but you can imagine how many metrics you'd have with more executions. Next, you'll create a custom visualization with matplotlib to see the relationship between our model's accuracy and the amount of data used for training. Run the following to generate a graph:"
       ]
     },
     {
@@ -1036,7 +1148,7 @@
       "source": [
         "### Getting all Model artifacts\n",
         "\n",
-        "To query artifacts in this way, we'll create a `MetadataServiceClient`:"
+        "To query artifacts in this way, you'll create a `MetadataServiceClient`:"
       ]
     },
     {
@@ -1059,7 +1171,8 @@
         "id": "e5aee9cdc5bd"
       },
       "source": [
-        "Next, we'll make a `list_artifacts` request to that endpoint and pass a filter indicating which artifacts we'd like in our response. First, let's get all the artifacts in our project that are **models**. To do that, run the following in your notebook:"
+        "### List all Model artifacts\n",
+        "Next, you'll make a `list_artifacts` request to that endpoint and pass a filter indicating which artifacts we'd like in our response. First, let's get all the artifacts in our project that are **models**. To do that, run the following in your notebook:"
       ]
     },
     {
@@ -1170,6 +1283,26 @@
         "\n",
         "* Delete the [Cloud Storage bucket](https://console.cloud.google.com/storage/browser/) you created"
       ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "540229ed8d79"
+      },
+      "outputs": [],
+      "source": [
+        "# Delete the pipeline\n",
+        "pipeline_experiments_df = aiplatform.get_pipeline_df(pipeline=\"mlmd-pipeline\")\n",
+        "for i in range(2):\n",
+        "    pipeline_job = aiplatform.PipelineJob.get(pipeline_experiments_df.run_name[i])\n",
+        "    pipeline_job.delete()\n",
+        "\n",
+        "delete_bucket = False\n",
+        "\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -rf {BUCKET_URI}"
+      ]
     }
   ],
   "metadata": {

notebooks/official/pipelines/automl_tabular_classification_beans.ipynb

@@ -309,7 +309,7 @@
       },
       "outputs": [],
       "source": [
-        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+        "PROJECT_ID = \"python-docs-samples-tests\"  # @param {type:\"string\"}"
       ]
     },
     {
@@ -320,7 +320,7 @@
       },
       "outputs": [],
       "source": [
-        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"python-docs-samples-tests\":\n",
         "    # Get your GCP project id from gcloud\n",
         "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
         "    PROJECT_ID = shell_output[0]\n",