Merge branch 'main' into imkc--tabular-regression-batch-bq

notebooks/official/automl/sdk_automl_tabular_regression_batch_bq.ipynb

@@ -0,0 +1,956 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title"
+      },
+      "source": [
+        "# Vertex AI SDK for Python: AutoML training tabular regression model for batch prediction using BigQuery\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_tabular_regression_batch_bq.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_tabular_regression_batch_bq.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_tabular_regression_batch_bq.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>                                                                                               \n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:automl"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use the Vertex AI SDK for Python to create tabular regression models and do batch prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:gsod,lrg"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [GSOD dataset](https://console.cloud.google.com/marketplace/product/noaa-public/gsod) from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). In this version of the dataset, you use the year, month, and day fields to predict the mean daily temperature (mean_temp)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:automl,training,online_prediction"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you create an AutoML tabular regression model and deploy it for batch prediction using the Vertex AI SDK for Python. You can alternatively create and deploy models using the `gcloud` command-line tool or batch using the Cloud Console.\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Create a Vertex AI `Dataset` resource.\n",
+        "- Train the model.\n",
+        "- View the model evaluation.\n",
+        "- Deploy the `Model` resource to a serving `Endpoint` resource.\n",
+        "- Make a prediction.\n",
+        "- Undeploy the `Model`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "costs"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_local"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "If you are using Colab or Vertex AI Workbench, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
+        "\n",
+        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
+        "\n",
+        "- The Cloud Storage SDK\n",
+        "- Git\n",
+        "- Python 3\n",
+        "- virtualenv\n",
+        "- Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
+        "\n",
+        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
+        "\n",
+        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
+        "\n",
+        "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.  Activate the virtual environment.\n",
+        "\n",
+        "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
+        "\n",
+        "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
+        "\n",
+        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the latest version of the Vertex AI SDK for Python."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# Google Cloud Notebook\n",
+        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    USER_FLAG = \"--user\"\n",
+        "else:\n",
+        "    USER_FLAG = \"\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_storage"
+      },
+      "source": [
+        "Install the latest version of *google-cloud-storage*."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_storage"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install -U google-cloud-storage $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b2f8bf1a1c31"
+      },
+      "source": [
+        "Install the latest version of *google-cloud-bigquery*."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fb18bb35a386"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install -U \"google-cloud-bigquery[pandas]\" $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "After installing the packages, restart the notebook kernel."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "before_you_begin:nogpu"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### GPU runtime\n",
+        "\n",
+        "This tutorial does not require a GPU runtime.\n",
+        "\n",
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "\n",
+        "3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
+        "\n",
+        "4. If you are running this notebook locally, install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
+        "\n",
+        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations throughout the rest of this notebook. The following regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
+        "\n",
+        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
+        "\n",
+        "**Otherwise**, follow these steps:\n",
+        "\n",
+        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
+        "\n",
+        "**Click Create service account**.\n",
+        "\n",
+        "In the **Service account name** field, enter a name, and click **Create**.\n",
+        "\n",
+        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
+        "\n",
+        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "outputs": [],
+      "source": [
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If on Google Cloud Notebook, then don't execute this code\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aiplatform"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "source": [
+        "## Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and the corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tutorial_start:automl"
+      },
+      "source": [
+        "# Tutorial\n",
+        "\n",
+        "Now you are ready to start creating your own AutoML tabular regression model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_file:u_dataset,bq"
+      },
+      "source": [
+        "#### Location of BigQuery training data.\n",
+        "\n",
+        "Set the `IMPORT_File` variable to the location of the data table in BigQuery."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:gsod,bq,lrg"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = \"bigquery-public-data.samples.gsod\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "07d8e5973590"
+      },
+      "source": [
+        "#### Prepare the batch prediction data\n",
+        "\n",
+        "Create two datasets from the original data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "63a447565c9c"
+      },
+      "outputs": [],
+      "source": [
+        "from google.cloud import bigquery\n",
+        "\n",
+        "# Create client in default region\n",
+        "bq_client = bigquery.Client(\n",
+        "    project=PROJECT_ID,\n",
+        "    credentials=aiplatform.initializer.global_config.credentials,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "56ade963c098"
+      },
+      "outputs": [],
+      "source": [
+        "# Create training dataset in default region\n",
+        "TRAINING_INPUT_DATASET_ID = f\"gsod_training_{TIMESTAMP}\"\n",
+        "bq_dataset = bigquery.Dataset(f\"{PROJECT_ID}.{TRAINING_INPUT_DATASET_ID}\")\n",
+        "bq_dataset = bq_client.create_dataset(bq_dataset)\n",
+        "print(f\"Created dataset {bq_client.project}.{bq_dataset.dataset_id}\")\n",
+        "\n",
+        "# Create test dataset in default region\n",
+        "PREDICTION_INPUT_DATASET_ID = f\"gsod_prediction_{TIMESTAMP}\"\n",
+        "bq_dataset = bigquery.Dataset(f\"{PROJECT_ID}.{PREDICTION_INPUT_DATASET_ID}\")\n",
+        "bq_dataset = bq_client.create_dataset(bq_dataset)\n",
+        "print(f\"Created dataset {bq_client.project}.{bq_dataset.dataset_id}\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5b790c91182f"
+      },
+      "outputs": [],
+      "source": [
+        "# Select top 3000 rows of dataset\n",
+        "TRAINING_SIZE = 3000\n",
+        "query = f\"\"\"\n",
+        "        SELECT *\n",
+        "        FROM {IMPORT_FILE}\n",
+        "        LIMIT {TRAINING_SIZE}\n",
+        "        \"\"\"\n",
+        "\n",
+        "TRAINING_INPUT_TABLE_ID = f\"{PROJECT_ID}.{TRAINING_INPUT_DATASET_ID}.test\"\n",
+        "job_config = bigquery.QueryJobConfig(destination=TRAINING_INPUT_TABLE_ID)\n",
+        "\n",
+        "query_job = bq_client.query(query, job_config=job_config)  # API request\n",
+        "query_job.result()  # Waits for query to finish"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8f841da4574b"
+      },
+      "outputs": [],
+      "source": [
+        "# Select a subset of the original dataset for testing\n",
+        "PREDICTION_SIZE = 100\n",
+        "query = f\"\"\"\n",
+        "        SELECT *\n",
+        "        FROM {IMPORT_FILE}\n",
+        "        LIMIT {PREDICTION_SIZE}\n",
+        "        OFFSET {TRAINING_SIZE}        \n",
+        "        \"\"\"\n",
+        "\n",
+        "PREDICTION_INPUT_TABLE_ID = f\"{PROJECT_ID}.{PREDICTION_INPUT_DATASET_ID}.prediction\"\n",
+        "job_config = bigquery.QueryJobConfig(destination=PREDICTION_INPUT_TABLE_ID)\n",
+        "\n",
+        "query_job = bq_client.query(query, job_config=job_config)  # API request\n",
+        "query_job.result()  # Waits for query to finish"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_dataset:tabular,bq,lrg"
+      },
+      "source": [
+        "### Create the Dataset\n",
+        "\n",
+        "Use `TabularDataset.create()` to create a `TabularDataset` resource, which takes the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the `Dataset` resource.\n",
+        "- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n",
+        "- `bq_source`: Alternatively, import data items from a BigQuery table into the `Dataset` resource.\n",
+        "\n",
+        "This operation may take several minutes."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_dataset:tabular,bq,lrg"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = aiplatform.TabularDataset.create(\n",
+        "    display_name=\"NOAA historical weather data\" + \"_\" + TIMESTAMP,\n",
+        "    bq_source=[f\"bq://{TRAINING_INPUT_TABLE_ID}\"],\n",
+        ")\n",
+        "\n",
+        "label_column = \"mean_temp\"\n",
+        "\n",
+        "print(dataset.resource_name)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_transformations:gsod"
+      },
+      "outputs": [],
+      "source": [
+        "COLUMN_SPECS = {\n",
+        "    \"year\": \"auto\",\n",
+        "    \"month\": \"auto\",\n",
+        "    \"day\": \"auto\",\n",
+        "}\n",
+        "\n",
+        "label_column = \"mean_temp\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_automl_pipeline:tabular,lrg,transformations"
+      },
+      "source": [
+        "### Create and run training pipeline\n",
+        "\n",
+        "To train an AutoML model, create and run a training pipeline.\n",
+        "\n",
+        "#### Create training pipeline\n",
+        "\n",
+        "Create an AutoML training pipeline using the `AutoMLTabularTrainingJob` class, with the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the `TrainingJob` resource.\n",
+        "- `optimization_prediction_type`: The type task to train the model for.\n",
+        "  - `classification`: A tabular classification model.\n",
+        "  - `regression`: A tabular regression model.\n",
+        "- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
+        "- `optimization_objective`: The optimization objective (minimize or maximize).\n",
+        "  - binary classification:\n",
+        "    - `minimize-log-loss`\n",
+        "    - `maximize-au-roc`\n",
+        "    - `maximize-au-prc`\n",
+        "    - `maximize-precision-at-recall`\n",
+        "    - `maximize-recall-at-precision`\n",
+        "  - multi-class classification:\n",
+        "    - `minimize-log-loss`\n",
+        "  - regression:\n",
+        "    - `minimize-rmse`\n",
+        "    - `minimize-mae`\n",
+        "    - `minimize-rmsle`\n",
+        "\n",
+        "The instantiated object is the DAG (directed acyclic graph) for the training pipeline."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_automl_pipeline:tabular,lrg,transformations"
+      },
+      "outputs": [],
+      "source": [
+        "training_job = aiplatform.AutoMLTabularTrainingJob(\n",
+        "    display_name=\"gsod_\" + TIMESTAMP,\n",
+        "    optimization_prediction_type=\"regression\",\n",
+        "    optimization_objective=\"minimize-rmse\",\n",
+        "    column_specs=COLUMN_SPECS,\n",
+        ")\n",
+        "\n",
+        "print(training_job)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "run_automl_pipeline:tabular"
+      },
+      "source": [
+        "#### Run the training pipeline\n",
+        "\n",
+        "Run the training job by invoking the `run` method with the following parameters:\n",
+        "\n",
+        "- `dataset`: The `Dataset` resource to train the model.\n",
+        "- `model_display_name`: The human readable name for the trained model.\n",
+        "- `training_fraction_split`: The percentage of the dataset to use for training.\n",
+        "- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
+        "- `validation_fraction_split`: The percentage of the dataset to use for validation.\n",
+        "- `target_column`: The name of the column to train as the label.\n",
+        "- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
+        "- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.\n",
+        "\n",
+        "The `run` method when completed returns the `Model` resource.\n",
+        "\n",
+        "The execution of the training pipeline will take upto 20 minutes."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "run_automl_pipeline:tabular"
+      },
+      "outputs": [],
+      "source": [
+        "model = training_job.run(\n",
+        "    dataset=dataset,\n",
+        "    model_display_name=\"gsod_\" + TIMESTAMP,\n",
+        "    training_fraction_split=0.6,\n",
+        "    validation_fraction_split=0.2,\n",
+        "    test_fraction_split=0.2,\n",
+        "    budget_milli_node_hours=1000,\n",
+        "    disable_early_stopping=False,\n",
+        "    target_column=label_column,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "evaluate_the_model:mbsdk"
+      },
+      "source": [
+        "## Review model evaluation scores\n",
+        "After your model has finished training, you can review its evaluation scores."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "evaluate_the_model:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "# Get evaluations\n",
+        "model_evaluations = model.list_model_evaluations()\n",
+        "\n",
+        "model_evaluation = list(model_evaluations)[0]\n",
+        "print(model_evaluation)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "337625a4c33c"
+      },
+      "source": [
+        "## Send a batch prediction request\n",
+        "\n",
+        "Now you can make a batch prediction."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "93bc53cc960d"
+      },
+      "source": [
+        "### Create a results dataset\n",
+        "\n",
+        "Create a dataset to store the prediction results."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f49e90431fcd"
+      },
+      "outputs": [],
+      "source": [
+        "# Create results dataset in default region\n",
+        "RESULTS_DATASET_ID = f\"gsod_results_{TIMESTAMP}\"\n",
+        "bq_dataset = bigquery.Dataset(f\"{PROJECT_ID}.{RESULTS_DATASET_ID}\")\n",
+        "bq_dataset = bq_client.create_dataset(bq_dataset)\n",
+        "print(f\"Created dataset {bq_client.project}.{bq_dataset.dataset_id}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "36c070503d2f"
+      },
+      "source": [
+        "### Make the batch prediction request\n",
+        "\n",
+        "You can make a batch prediction by invoking the batch_predict() method, with the following parameters:\n",
+        "\n",
+        "- `job_display_name`: The human readable name for the batch prediction job.\n",
+        "- `gcs_source`: A list of one or more batch request input files.\n",
+        "- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.\n",
+        "- `instances_format`: The format for the input instances, either 'bigquery', 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
+        "- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
+        "- `machine_type`: The type of machine to use for training.\n",
+        "- `accelerator_type`: The hardware accelerator type.\n",
+        "- `accelerator_count`: The number of accelerators to attach to a worker replica.\n",
+        "- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bbd40d78ea46"
+      },
+      "outputs": [],
+      "source": [
+        "# Note: The bigquery_source and bigquery_destination_prefix must be in the same region\n",
+        "PREDICTION_RESULTS_DATASET_ID = f\"{PROJECT_ID}.{RESULTS_DATASET_ID}\"\n",
+        "\n",
+        "batch_predict_job = model.batch_predict(\n",
+        "    job_display_name=\"tabular_regression_batch_predict_job\",\n",
+        "    bigquery_source=f\"bq://{PREDICTION_INPUT_TABLE_ID}\",\n",
+        "    instances_format=\"bigquery\",\n",
+        "    predictions_format=\"bigquery\",\n",
+        "    bigquery_destination_prefix=f\"bq://{PREDICTION_RESULTS_DATASET_ID}\",\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4fc95aae00b0"
+      },
+      "source": [
+        "###  View the batch prediction results\n",
+        "\n",
+        "Use the BigQuery Python client to query the destination table and return results as a Pandas dataframe."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fafe1b0f654b"
+      },
+      "outputs": [],
+      "source": [
+        "dataframe = (\n",
+        "    bq_client.query(f\"SELECT * FROM `{PREDICTION_RESULTS_DATASET_ID}.*`\")\n",
+        "    .result()\n",
+        "    .to_dataframe()\n",
+        ")\n",
+        "\n",
+        "print(dataframe.head())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
+        "\n",
+        "- Model\n",
+        "- AutoML Training Job\n",
+        "- Batch Job\n",
+        "- Cloud Storage Bucket"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3be2c2bf9146"
+      },
+      "outputs": [],
+      "source": [
+        "# Delete BigQuery datasets\n",
+        "bq_client.delete_dataset(\n",
+        "    f\"{PROJECT_ID}.{TRAINING_INPUT_DATASET_ID}\",\n",
+        "    delete_contents=True,\n",
+        "    not_found_ok=True,\n",
+        ")\n",
+        "\n",
+        "bq_client.delete_dataset(\n",
+        "    f\"{PROJECT_ID}.{PREDICTION_INPUT_DATASET_ID}\",\n",
+        "    delete_contents=True,\n",
+        "    not_found_ok=True,\n",
+        ")\n",
+        "\n",
+        "bq_client.delete_dataset(\n",
+        "    f\"{PROJECT_ID}.{RESULTS_DATASET_ID}\", delete_contents=True, not_found_ok=True\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cleanup:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "# Delete Vertex AI resources\n",
+        "dataset.delete()\n",
+        "model.delete()\n",
+        "training_job.delete()\n",
+        "batch_predict_job.delete()"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "sdk_automl_tabular_regression_batch_bq.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}