From e409cf710b70cff81101b06e894f53b6a9357a72 Mon Sep 17 00:00:00 2001 From: udaypunna Date: Wed, 9 Nov 2022 14:14:06 +0000 Subject: [PATCH] added service account and downgraded gcpc version --- .../automl_tabular_on_vertex_pipelines.ipynb | 2144 ++++++++--------- 1 file changed, 1071 insertions(+), 1073 deletions(-) diff --git a/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb b/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb index 4f807a0d8..adeb5c51a 100644 --- a/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb +++ b/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb @@ -1,1076 +1,1074 @@ { - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "18ebbd838e32" - }, - "outputs": [], - "source": [ - "# Copyright 2022 Google LLC\n", - "#\n", - "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", - "# you may not use this file except in compliance with the License.\n", - "# You may obtain a copy of the License at\n", - "#\n", - "# https://www.apache.org/licenses/LICENSE-2.0\n", - "#\n", - "# Unless required by applicable law or agreed to in writing, software\n", - "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", - "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", - "# See the License for the specific language governing permissions and\n", - "# limitations under the License." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mThXALJl9Yue" - }, - "source": [ - "# Tabular Workflow: AutoML Tabular Pipeline\n", - "\n", - "\n", - " \n", - " \n", - " \n", - "
\n", - " \n", - " \"Colab Run in Colab\n", - " \n", - " \n", - " \n", - " \"GitHub\n", - " View on GitHub\n", - " \n", - " \n", - " \n", - " \"Vertex\n", - " Open in Vertex AI Workbench\n", - " \n", - "
\n", - "


" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fcc745968395" - }, - "source": [ - "## Overview\n", - "\n", - "In this tutorial, you use two Vertex AI Tabular Workflows pipelines to train AutoML models using different configurations. You see how `get_automl_tabular_pipeline_and_parameters` gives you the ability to customize the default AutoML Tabular pipeline, and how `get_skip_architecture_search_pipeline_and_parameters` allows you to reduce the training time and cost for an AutoML model by using the tuning results from a previous pipeline run." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "f887ec5c06c5" - }, - "source": [ - "### Objective\n", - "\n", - "In this tutorial, you learn how to create two regression models using [Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) downloaded from [Google Cloud Pipeline Components](https://cloud.google.com/vertex-ai/docs/pipelines/components-introduction) (GCPC). These pipelines will be Vertex AI Tabular Workflow pipelines which are maintained by Google. These pipelines showcase different ways to customize the Vertex Tabular training process.\n", - "\n", - "The steps performed are:\n", - "\n", - "- Create a training pipeline that reduces the search space from the default to save time.\n", - "- Create a training pipeline that reuses the architecture search results from the previous pipeline to save time." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eac26958afe8" - }, - "source": [ - "### Dataset\n", - "\n", - "The dataset you will be using is [Bank Marketing](https://archive.ics.uci.edu/ml/datasets/bank+marketing).\n", - "The data is for direct marketing campaigns (phone calls) of a Portuguese banking institution. The binary classification goal is to predict if a client subscribe a term deposit. For this notebook, you randomly selected 90% of the rows in the original dataset and saved them in a train.csv file hosted on Cloud Storage. To download the file, click [here](https://storage.googleapis.com/cloud-samples-data/vertex-ai/tabular-workflows/datasets/bank-marketing/train.csv)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "181d4dfbf917" - }, - "source": [ - "### Costs\n", - "\n", - "This tutorial uses billable components of Google Cloud:\n", - "\n", - "* Vertex AI\n", - "* Cloud Storage\n", - "\n", - "Learn about [Vertex AI\n", - "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", - "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", - "Calculator](https://cloud.google.com/products/calculator/)\n", - "to generate a cost estimate based on your projected usage." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tuOonx6suOb7" - }, - "source": [ - "### Set up your local development environment\n", - "\n", - "**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets all the requirements to run this notebook. You can skip this step.\n", - "\n", - "**Otherwise**, make sure your environment meets this notebook's requirements. You need the following:\n", - "\n", - "- The Cloud Storage SDK\n", - "- Python 3\n", - "- virtualenv\n", - "- Jupyter notebook running in a virtual environment with Python 3\n", - "\n", - "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n", - "\n", - "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n", - "\n", - "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n", - "\n", - "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n", - "\n", - "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n", - "\n", - "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", - "\n", - "6. Open this notebook in the Jupyter Notebook Dashboard." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "e85f0288a6df" - }, - "source": [ - "## Install additional packages\n", - "\n", - "Install the latest version of the Google Cloud Pipeline Components (GCPC) SDK." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "E7-SzYTR9bo2" - }, - "outputs": [], - "source": [ - "!pip install -U google-cloud-pipeline-components==1.0.25 -q" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Bj5O0S5RTxzY" - }, - "source": [ - "### Restart the kernel\n", - "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.\n", - "\n", - "\n", - "**Note: Once this cell has finished running, continue on. You do not need to re-run any of the cells above.**\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "023DMKUaTypt" - }, - "outputs": [], - "source": [ - "import os\n", - "\n", - "if not os.getenv(\"IS_TESTING\"):\n", - " # Automatically restart kernel after installs\n", - " import IPython\n", - "\n", - " app = IPython.Application.instance()\n", - " app.kernel.do_shutdown(True)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yfEglUHQk9S3" - }, - "source": [ - "## Before you begin\n", - "\n", - "### GPU runtime\n", - "\n", - "This tutorial does not require a GPU runtime.\n", - "\n", - "### Set up your Google Cloud project\n", - "\n", - "**The following steps are required, regardless of your notebook environment.**\n", - "\n", - "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", - "\n", - "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", - "\n", - "3. [Enable the following APIs: Vertex AI APIs, Dataflow APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,dataflow.googleapis.com,compute_component,storage-component.googleapis.com)\n", - "\n", - "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", - "\n", - "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", - "Cloud SDK uses the right project for all the commands in this notebook.\n", - "\n", - "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zebLBGXOky2A" - }, - "source": [ - "## Notes about service account and permission\n", - "\n", - "**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n", - "\n", - "|Service account email|Description|Roles|\n", - "|---|---|---|\n", - "|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Developer, Dataflow Worker, Storage Admin, BigQuery Data Editor, Vertex AI User, Service Account User|\n", - "|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n", - "\n", - "\n", - "1. Goto https://console.cloud.google.com/iam-admin/iam.\n", - "2. Check the \"Include Google-provided role grants\" checkbox.\n", - "3. Find the above emails.\n", - "4. Grant the corresponding roles.\n", - "\n", - "### Using data source from a different project\n", - "- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n", - "- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "95cb7ffd6895" - }, - "source": [ - "### Set your project ID\n", - "\n", - "Set your project ID below. If you know know your project ID, leave the field blank and the following cells may be able to find it. Optionally, you may also set a service account in the cell below." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "cd85f5c794e5" - }, - "outputs": [], - "source": [ - "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5a604eeffc32" - }, - "outputs": [], - "source": [ - "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", - " # Get your GCP project id from gcloud\n", - " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", - " PROJECT_ID = shell_output[0]\n", - " print(\"Project ID:\", PROJECT_ID)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "30e64c0eda41" - }, - "outputs": [], - "source": [ - "! gcloud config set project $PROJECT_ID" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b12f508d97c6" - }, - "source": [ - "### Region\n", - "You may change the `REGION` variable, which is used for Vertex Forecasting operations\n", - "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", - "\n", - "- Americas: `us-central1`\n", - "- Europe: `europe-west4`\n", - "- Asia Pacific: `asia-east1`\n", - "\n", - "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8e8b7997de7a" - }, - "outputs": [], - "source": [ - "REGION = \"[your-region]\" # @param {type: \"string\"}\n", - "\n", - "if REGION == \"[your-region]\":\n", - " REGION = \"us-central1\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eu0e2TRVxjHb" - }, - "source": [ - "### Authenticate your Google Cloud account\n", - "\n", - "**If you are using Vertex AI Workbench Notebooks**, your environment is already authenticated.\n", - "\n", - "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", - "\n", - "**Otherwise**, follow these steps:\n", - "\n", - "- In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", - "\n", - "- **Click Create service account**.\n", - "\n", - "- In the **Service account name** field, enter a name, and click **Create**.\n", - "\n", - "- In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", - "\n", - "- Click Create. A JSON file that contains your key downloads to your local environment.\n", - "\n", - "- Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_nJ1VqdTxosw" - }, - "outputs": [], - "source": [ - "# If you are running this notebook in Colab, run this cell and follow the\n", - "# instructions to authenticate your GCP account. This provides access to your\n", - "# Cloud Storage bucket and lets you submit training jobs and prediction\n", - "# requests.\n", - "\n", - "import os\n", - "import sys\n", - "\n", - "# If on Vertex AI Workbench, then don't execute this code\n", - "IS_COLAB = \"google.colab\" in sys.modules\n", - "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n", - " \"DL_ANACONDA_HOME\"\n", - "):\n", - " if \"google.colab\" in sys.modules:\n", - " from google.colab import auth as google_auth\n", - "\n", - " google_auth.authenticate_user()\n", - "\n", - " # If you are running this notebook locally, replace the string below with the\n", - " # path to your service account key and run this cell to authenticate your GCP\n", - " # account.\n", - " elif not os.getenv(\"IS_TESTING\"):\n", - " %env GOOGLE_APPLICATION_CREDENTIALS '[your-service-account-key-path]'" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "OUfwWir9yPNV" - }, - "source": [ - "### Create a Cloud Storage bucket\n", - "\n", - "**The following steps are required, regardless of your notebook environment.**\n", - "\n", - "All training related files (TF model checkpoint, TensorBoard file, etc) saved to the GCS bucket. The pipeline not clean up the files since some of them might be useful for you, **please make sure to clean up the files**. For easy cleanup, you can set [GCS bucket level TTL](https://cloud.google.com/storage/docs/lifecycle).\n", - "\n", - "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5325f437b46a" - }, - "outputs": [], - "source": [ - "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n", - "GENERATE_BUCKET_URI = True # @param {type:\"boolean\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "a30ea8551126" - }, - "source": [ - "Create the bucket if it doesn't already exist." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0217c63ed87f" - }, - "outputs": [], - "source": [ - "import uuid\n", - "\n", - "if GENERATE_BUCKET_URI:\n", - " bucket_name = \"gs://test-{}\".format(uuid.uuid4())\n", - " !gsutil mb -p {PROJECT_ID} -l {REGION} {bucket_name}\n", - "\n", - " # set GCS bucket object TTL to 7 days\n", - " !echo '{\"rule\":[{\"action\": {\"type\": \"Delete\"},\"condition\": {\"age\": 7}}]}' > gcs_lifecycle.tmp\n", - " !gsutil lifecycle set gcs_lifecycle.tmp {bucket_name}\n", - " !rm gcs_lifecycle.tmp\n", - "\n", - " BUCKET_URI = bucket_name\n", - " print(f\"changed BUCKET_URI to {BUCKET_URI} due to GENERATE_BUCKET_URI is True\")\n", - "\n", - "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", - " BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + uuid.uuid4()\n", - "\n", - "! gsutil ls -b $BUCKET_URI || gsutil mb -l $DATA_REGION $BUCKET_URI" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c4cf2cdebb50" - }, - "source": [ - "Finally, validate access to your Cloud Storage bucket by examining its contents:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "96ad3d416327" - }, - "outputs": [], - "source": [ - "! gsutil ls -al $BUCKET_URI" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "44accda192d5" - }, - "source": [ - "#### Service Account\n", - "\n", - "You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "c65d12a97f45" - }, - "outputs": [], - "source": [ - "SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "604ae09ab6d3" - }, - "outputs": [], - "source": [ - "if (\n", - " SERVICE_ACCOUNT == \"\"\n", - " or SERVICE_ACCOUNT is None\n", - " or SERVICE_ACCOUNT == \"[your-service-account]\"\n", - "):\n", - " # Get your service account from gcloud\n", - " if not IS_COLAB:\n", - " shell_output = !gcloud auth list 2>/dev/null\n", - " SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n", - "\n", - " else: # IS_COLAB:\n", - " shell_output = ! gcloud projects describe $PROJECT_ID\n", - " project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n", - " SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n", - "\n", - " print(\"Service Account:\", SERVICE_ACCOUNT)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "d1ecb60964d5" - }, - "source": [ - "#### Set service account access for Vertex AI Pipelines\n", - "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "a592f0a380c2" - }, - "outputs": [], - "source": [ - "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n", - "\n", - "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fbbc3479a1da" - }, - "source": [ - "## Import libraries and define constants" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8G6YmJT1yqkV" - }, - "outputs": [], - "source": [ - "# Import required modules\n", - "import json\n", - "from typing import Any, Dict, List\n", - "\n", - "from google.cloud import aiplatform, storage\n", - "from google_cloud_pipeline_components.experimental.automl.tabular import \\\n", - " utils as automl_tabular_utils" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c0423f260423" - }, - "source": [ - "## Initialize Vertex SDK for Python\n", - "\n", - "Initialize the Vertex SDK for Python for your project." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ad69f2590268" - }, - "outputs": [], - "source": [ - "aiplatform.init(project=PROJECT_ID, location=REGION)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3LWH3PRF5o2v" - }, - "source": [ - "### Define helper functions" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "g9FPFT8c5oC0" - }, - "outputs": [], - "source": [ - "# Get the bucket name and path.\n", - "def get_bucket_name_and_path(uri):\n", - " no_prefix_uri = uri[len(\"gs://\") :]\n", - " splits = no_prefix_uri.split(\"/\")\n", - " return splits[0], \"/\".join(splits[1:])\n", - "\n", - "\n", - "# Download from the bucket.\n", - "def download_from_gcs(uri):\n", - " bucket_name, path = get_bucket_name_and_path(uri)\n", - " storage_client = storage.Client(project=PROJECT_ID)\n", - " bucket = storage_client.get_bucket(bucket_name)\n", - " blob = bucket.blob(path)\n", - " return blob.download_as_string()\n", - "\n", - "\n", - "# Write content in to the bucket.\n", - "def write_to_gcs(uri: str, content: str):\n", - " bucket_name, path = get_bucket_name_and_path(uri)\n", - " storage_client = storage.Client()\n", - " bucket = storage_client.get_bucket(bucket_name)\n", - " blob = bucket.blob(path)\n", - " blob.upload_from_string(content)\n", - "\n", - "\n", - "# Generate auto transformations.\n", - "def generate_auto_transformation(column_names: List[str]) -> List[Dict[str, Any]]:\n", - " transformations = []\n", - " for column_name in column_names:\n", - " transformations.append({\"auto\": {\"column_name\": column_name}})\n", - " return transformations\n", - "\n", - "\n", - "# Write auto transformations in to the bucket.\n", - "def write_auto_transformations(uri: str, column_names: List[str]):\n", - " transformations = generate_auto_transformation(column_names)\n", - " write_to_gcs(uri, json.dumps(transformations))\n", - "\n", - "\n", - "# Get the task details filter with task name.\n", - "def get_task_detail(\n", - " task_details: List[Dict[str, Any]], task_name: str\n", - ") -> List[Dict[str, Any]]:\n", - " for task_detail in task_details:\n", - " if task_detail.task_name == task_name:\n", - " return task_detail\n", - "\n", - "\n", - "# Get the deployed model uri from task details.\n", - "def get_deployed_model_uri(\n", - " task_details,\n", - "):\n", - " ensemble_task = get_task_detail(task_details, \"model-upload\")\n", - " return ensemble_task.outputs[\"model\"].artifacts[0].uri\n", - "\n", - "\n", - "# Get the model uri from the task details.\n", - "def get_no_custom_ops_model_uri(task_details):\n", - " ensemble_task = get_task_detail(task_details, \"automl-tabular-ensemble\")\n", - " return download_from_gcs(\n", - " ensemble_task.outputs[\"model_without_custom_ops\"].artifacts[0].uri\n", - " )\n", - "\n", - "\n", - "# Get the feature attributions from task details.\n", - "def get_feature_attributions(\n", - " task_details,\n", - "):\n", - " ensemble_task = get_task_detail(task_details, \"model-evaluation-2\")\n", - " return download_from_gcs(\n", - " ensemble_task.outputs[\"evaluation_metrics\"]\n", - " .artifacts[0]\n", - " .metadata[\"explanation_gcs_path\"]\n", - " )\n", - "\n", - "\n", - "# Get the evaluation metrics from task details.\n", - "def get_evaluation_metrics(\n", - " task_details,\n", - "):\n", - " ensemble_task = get_task_detail(task_details, \"model-evaluation\")\n", - " return download_from_gcs(\n", - " ensemble_task.outputs[\"evaluation_metrics\"].artifacts[0].uri\n", - " )\n", - "\n", - "\n", - "# Print the parsed json.\n", - "def load_and_print_json(s):\n", - " parsed = json.loads(s)\n", - " print(json.dumps(parsed, indent=2, sort_keys=True))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gvNFMRmBegZq" - }, - "source": [ - "### Define training specification" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "eV4JrwB8wAkg" - }, - "outputs": [], - "source": [ - "run_evaluation = True # @param {type:\"boolean\"}\n", - "run_distillation = False # @param {type:\"boolean\"}\n", - "root_dir = os.path.join(BUCKET_URI, \"automl_tabular_pipeline\")\n", - "prediction_type = \"classification\"\n", - "optimization_objective = \"minimize-log-loss\"\n", - "target_column = \"deposit\"\n", - "data_source_csv_filenames = \"gs://cloud-samples-data/vertex-ai/tabular-workflows/datasets/bank-marketing/train.csv\"\n", - "data_source_bigquery_table_path = None # format: bq://bq_project.bq_dataset.bq_table\n", - "\n", - "timestamp_split_key = None # timestamp column name when using timestamp split\n", - "stratified_split_key = None # target column name when using stratified split\n", - "training_fraction = 0.8\n", - "validation_fraction = 0.1\n", - "test_fraction = 0.1\n", - "\n", - "predefined_split_key = None\n", - "if predefined_split_key:\n", - " training_fraction = None\n", - " validation_fraction = None\n", - " test_fraction = None\n", - "\n", - "weight_column = None\n", - "\n", - "features = [\n", - " \"age\",\n", - " \"job\",\n", - " \"marital\",\n", - " \"education\",\n", - " \"default\",\n", - " \"balance\",\n", - " \"housing\",\n", - " \"loan\",\n", - " \"contact\",\n", - " \"day\",\n", - " \"month\",\n", - " \"duration\",\n", - " \"campaign\",\n", - " \"pdays\",\n", - " \"previous\",\n", - " \"poutcome\",\n", - "]\n", - "transformations = generate_auto_transformation(features)\n", - "transform_config_path = os.path.join(root_dir, f\"transform_config_{uuid.uuid4()}.json\")\n", - "write_to_gcs(transform_config_path, json.dumps(transformations))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zyWGg2s09xOk" - }, - "source": [ - "## VPC related config\n", - "\n", - "If you need to use a custom Dataflow subnetwork, you can set it through the `dataflow_subnetwork` parameter. The requirements are:\n", - "1. `dataflow_subnetwork` must be fully qualified subnetwork name.\n", - " [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications)]\n", - "1. The following service accounts must have [Compute Network User role](https://cloud.google.com/compute/docs/access/iam#compute.networkUser) assigned on the specified dataflow subnetwork [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#shared)]:\n", - " 1. Compute Engine default service account: PROJECT_NUMBER-compute@developer.gserviceaccount.com\n", - " 1. Dataflow service account: service-PROJECT_NUMBER@dataflow-service-producer-prod.iam.gserviceaccount.com\n", - "\n", - "If your project has VPC-SC enabled, please make sure:\n", - "\n", - "1. The dataflow subnetwork used in VPC-SC is configured properly for Dataflow.\n", - " [[reference](https://cloud.google.com/dataflow/docs/guides/routes-firewall)]\n", - "1. `dataflow_use_public_ips` is set to False.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_TePNlLl9v1q" - }, - "outputs": [], - "source": [ - "# Dataflow's fully qualified subnetwork name, when empty the default subnetwork will be used.\n", - "# Fully qualified subnetwork name is in the form of\n", - "# https://www.googleapis.com/compute/v1/projects/HOST_PROJECT_ID/regions/REGION_NAME/subnetworks/SUBNETWORK_NAME\n", - "# reference: https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications\n", - "dataflow_subnetwork = None # @param {type:\"string\"}\n", - "# Specifies whether Dataflow workers use public IP addresses.\n", - "dataflow_use_public_ips = True # @param {type:\"boolean\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "N-iXXE14voyR" - }, - "source": [ - "## Customize search space and change training configuration\n", - "\n", - "you create a skip evaluation AutoML Tables pipeline with the following customizations:\n", - "- Limit the hyperparameter search space\n", - "- Change machine type and tuning / training parallelism" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "sG46cXVueb66" - }, - "outputs": [], - "source": [ - "study_spec_parameters_override = [\n", - " {\n", - " \"parameter_id\": \"model_type\",\n", - " \"categorical_value_spec\": {\n", - " \"values\": [\n", - " \"nn\"\n", - " ] # The default value is [\"nn\", \"boosted_trees\"], this reduces the search space\n", - " },\n", - " }\n", - "]\n", - "\n", - "worker_pool_specs_override = [\n", - " {\"machine_spec\": {\"machine_type\": \"n1-standard-8\"}}, # override for TF chief node\n", - " {}, # override for TF worker node, since it's not used, leave it empty\n", - " {}, # override for TF ps node, since it's not used, leave it empty\n", - " {\n", - " \"machine_spec\": {\n", - " \"machine_type\": \"n1-standard-4\" # override for TF evaluator node\n", - " }\n", - " },\n", - "]\n", - "\n", - "# Number of weak models in the final ensemble model is\n", - "# stage_2_num_selected_trials * 5. If unspecified, 5 is the default value for\n", - "# stage_2_num_selected_trials.\n", - "stage_2_num_selected_trials = 5\n", - "\n", - "# The pipeline output a TF saved model contains the following TF custom op:\n", - "# - https://github.com/google/struct2tensor\n", - "#\n", - "# There are a few ways to run the model:\n", - "# - Official prediction server docker image\n", - "# Please follow the \"Run the model server\" section in\n", - "# https://cloud.google.com/vertex-ai/docs/export/export-model-tabular#run-server\n", - "# - Python or cpp runtimes like TF serving\n", - "# Please set export_additional_model_without_custom_ops so the pipeline\n", - "# outputs an additional model does does not depend on struct2tensor.\n", - "# - `get_no_custom_ops_model_uri` shows how to get the model artifact URI.\n", - "# - The input to the model is a dictionary of feature name to tensor. Use\n", - "# `saved_model_cli show --dir {saved_model.pb's path} --signature_def serving_default --tag serve`\n", - "# to find out more details.\n", - "export_additional_model_without_custom_ops = False\n", - "\n", - "train_budget_milli_node_hours = 1000 # 1 hour\n", - "\n", - "(\n", - " template_path,\n", - " parameter_values,\n", - ") = automl_tabular_utils.get_automl_tabular_pipeline_and_parameters(\n", - " PROJECT_ID,\n", - " REGION,\n", - " root_dir,\n", - " target_column,\n", - " prediction_type,\n", - " optimization_objective,\n", - " transform_config_path,\n", - " train_budget_milli_node_hours,\n", - " data_source_csv_filenames=data_source_csv_filenames,\n", - " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", - " weight_column=weight_column,\n", - " predefined_split_key=predefined_split_key,\n", - " timestamp_split_key=timestamp_split_key,\n", - " stratified_split_key=stratified_split_key,\n", - " training_fraction=training_fraction,\n", - " validation_fraction=validation_fraction,\n", - " test_fraction=test_fraction,\n", - " study_spec_parameters_override=study_spec_parameters_override,\n", - " stage_1_tuner_worker_pool_specs_override=worker_pool_specs_override,\n", - " cv_trainer_worker_pool_specs_override=worker_pool_specs_override,\n", - " run_evaluation=run_evaluation,\n", - " run_distillation=run_distillation,\n", - " dataflow_subnetwork=dataflow_subnetwork,\n", - " dataflow_use_public_ips=dataflow_use_public_ips,\n", - " export_additional_model_without_custom_ops=export_additional_model_without_custom_ops,\n", - ")\n", - "\n", - "job_id = \"automl-tabular-{}\".format(uuid.uuid4())\n", - "job = aiplatform.PipelineJob(\n", - " display_name=job_id,\n", - " location=REGION, # launches the pipeline job in the specified region\n", - " template_path=template_path,\n", - " job_id=job_id,\n", - " pipeline_root=root_dir,\n", - " parameter_values=parameter_values,\n", - " enable_caching=False,\n", - ")\n", - "\n", - "job.run(service_account=SERVICE_ACCOUNT)\n", - "\n", - "pipeline_task_details = job.gca_resource.job_detail.task_details\n", - "\n", - "if export_additional_model_without_custom_ops:\n", - " print(\n", - " \"trained model without custom TF ops:\",\n", - " get_no_custom_ops_model_uri(pipeline_task_details),\n", - " )\n", - "\n", - "if run_evaluation:\n", - " print(\"evaluation metrics:\")\n", - " load_and_print_json(get_evaluation_metrics(pipeline_task_details))\n", - "\n", - " print(\"feature attributions:\")\n", - " load_and_print_json(get_feature_attributions(pipeline_task_details))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_sF8a2RKtRhg" - }, - "source": [ - "## Skip architecture search\n", - "Instead of doing architecture search everytime, you can reuse the existing architecture search result. This could help:\n", - "1. reducing the variation of the output model\n", - "2. reducing training cost\n", - "\n", - "The existing architecture search result is stored in the `tuning_result_output` output of the `automl-tabular-stage-1-tuner` component. you can manually input it or get it programmatically." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8hlPps2Rtpq-" - }, - "outputs": [], - "source": [ - "stage_1_tuner_task = get_task_detail(\n", - " pipeline_task_details, \"automl-tabular-stage-1-tuner\"\n", - ")\n", - "stage_1_tuning_result_artifact_uri = (\n", - " stage_1_tuner_task.outputs[\"tuning_result_output\"].artifacts[0].uri\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c5F12ZL_uZZ3" - }, - "source": [ - "### Run the skip architecture search pipeline\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4x4GA5sMuewX" - }, - "outputs": [], - "source": [ - "(\n", - " template_path,\n", - " parameter_values,\n", - ") = automl_tabular_utils.get_skip_architecture_search_pipeline_and_parameters(\n", - " PROJECT_ID,\n", - " REGION,\n", - " root_dir,\n", - " target_column,\n", - " prediction_type,\n", - " optimization_objective,\n", - " transform_config_path,\n", - " train_budget_milli_node_hours,\n", - " data_source_csv_filenames=data_source_csv_filenames,\n", - " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", - " weight_column=weight_column,\n", - " predefined_split_key=predefined_split_key,\n", - " timestamp_split_key=timestamp_split_key,\n", - " stratified_split_key=stratified_split_key,\n", - " training_fraction=training_fraction,\n", - " validation_fraction=validation_fraction,\n", - " test_fraction=test_fraction,\n", - " stage_1_tuning_result_artifact_uri=stage_1_tuning_result_artifact_uri,\n", - " run_evaluation=run_evaluation,\n", - " dataflow_subnetwork=dataflow_subnetwork,\n", - " dataflow_use_public_ips=dataflow_use_public_ips,\n", - ")\n", - "\n", - "job_id = \"automl-tabular-skip-architecture-search-{}\".format(uuid.uuid4())\n", - "job = aiplatform.PipelineJob(\n", - " display_name=job_id,\n", - " location=REGION, # launches the pipeline job in the specified region\n", - " template_path=template_path,\n", - " job_id=job_id,\n", - " pipeline_root=root_dir,\n", - " parameter_values=parameter_values,\n", - " enable_caching=False,\n", - ")\n", - "\n", - "job.run(service_account=SERVICE_ACCOUNT)\n", - "\n", - "# Get model URI\n", - "skip_architecture_search_pipeline_task_details = (\n", - " job.gca_resource.job_detail.task_details\n", - ")\n", - "\n", - "if export_additional_model_without_custom_ops:\n", - " print(\n", - " \"trained model without custom TF ops:\",\n", - " get_no_custom_ops_model_uri(pipeline_task_details),\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "43342a43176e" - }, - "source": [ - "## Clean up Vertex and BigQuery resources\n", - "\n", - "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", - "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", - "\n", - "Otherwise, you can delete the individual resources you created in this tutorial:\n", - "\n", - "- Cloud Storage Bucket" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ad8d12061a65" - }, - "outputs": [], - "source": [ - "if os.getenv(\"IS_TESTING\"):\n", - " ! gsutil rm -r $BUCKET_URI" - ] - } - ], - "metadata": { - "colab": { - "collapsed_sections": [], - "name": "automl_tabular_on_vertex_pipelines.ipynb", - "toc_visible": true - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - } + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "18ebbd838e32" + }, + "outputs": [], + "source": [ + "# Copyright 2022 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] }, - "nbformat": 4, - "nbformat_minor": 0 + { + "cell_type": "markdown", + "metadata": { + "id": "mThXALJl9Yue" + }, + "source": [ + "# Tabular Workflow: AutoML Tabular Pipeline\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + " \n", + " \n", + " \"Vertex\n", + " Open in Vertex AI Workbench\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fcc745968395" + }, + "source": [ + "## Overview\n", + "\n", + "In this tutorial, you will use two Vertex AI Tabular Workflows pipelines to train AutoML models using different configurations. You will see how `get_automl_tabular_pipeline_and_parameters` gives you the ability to customize the default AutoML Tabular pipeline, and how `get_skip_architecture_search_pipeline_and_parameters` allows you to reduce the training time and cost for an AutoML model by using the tuning results from a previous pipeline run." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f887ec5c06c5" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you learn how to create two regression models using [Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) downloaded from [Google Cloud Pipeline Components](https://cloud.google.com/vertex-ai/docs/pipelines/components-introduction) (GCPC). These pipelines will be Vertex AI Tabular Workflow pipelines which are maintained by Google. These pipelines will showcase different ways to customize the Vertex Tabular training process.\n", + "\n", + "The steps performed are:\n", + "\n", + "- Create a training pipeline that reduces the search space from the default to save time.\n", + "- Create a training pipeline that reuses the architecture search results from the previous pipeline to save time." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eac26958afe8" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset you will be using is [Bank Marketing](https://archive.ics.uci.edu/ml/datasets/bank+marketing).\n", + "The data is for direct marketing campaigns (phone calls) of a Portuguese banking institution. The binary classification goal is to predict if a client will subscribe a term deposit. For this notebook, we randomly selected 90% of the rows in the original dataset and saved them in a train.csv file hosted on Cloud Storage. To download the file, click [here](https://storage.googleapis.com/cloud-samples-data/vertex-ai/tabular-workflows/datasets/bank-marketing/train.csv)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "181d4dfbf917" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tuOonx6suOb7" + }, + "source": [ + "### Set up your local development environment\n", + "\n", + "**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets all the requirements to run this notebook. You can skip this step.\n", + "\n", + "**Otherwise**, make sure your environment meets this notebook's requirements. You need the following:\n", + "\n", + "- The Cloud Storage SDK\n", + "- Python 3\n", + "- virtualenv\n", + "- Jupyter notebook running in a virtual environment with Python 3\n", + "\n", + "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n", + "\n", + "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n", + "\n", + "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n", + "\n", + "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n", + "\n", + "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n", + "\n", + "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", + "\n", + "6. Open this notebook in the Jupyter Notebook Dashboard." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e85f0288a6df" + }, + "source": [ + "## Install additional packages\n", + "\n", + "Install the latest version of the Google Cloud Pipeline Components (GCPC) SDK." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "E7-SzYTR9bo2" + }, + "outputs": [], + "source": [ + "!pip install -U google-cloud-pipeline-components==1.0.25 -q" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bj5O0S5RTxzY" + }, + "source": [ + "### Restart the kernel\n", + "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.\n", + "\n", + "\n", + "**Note: Once this cell has finished running, continue on. You do not need to re-run any of the cells above.**\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "023DMKUaTypt" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yfEglUHQk9S3" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "This tutorial does not require a GPU runtime.\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the following APIs: Vertex AI APIs, Dataflow APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,dataflow.googleapis.com,compute_component,storage-component.googleapis.com)\n", + "\n", + "4. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zebLBGXOky2A" + }, + "source": [ + "## Notes about service account and permission\n", + "\n", + "**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n", + "\n", + "|Service account email|Description|Roles|\n", + "|---|---|---|\n", + "|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Developer, Dataflow Worker, Storage Admin, BigQuery Data Editor, Vertex AI User, Service Account User|\n", + "|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n", + "\n", + "\n", + "1. Goto https://console.cloud.google.com/iam-admin/iam.\n", + "2. Check the \"Include Google-provided role grants\" checkbox.\n", + "3. Find the above emails.\n", + "4. Grant the corresponding roles.\n", + "\n", + "### Using data source from a different project\n", + "- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n", + "- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "95cb7ffd6895" + }, + "source": [ + "### Set your project ID\n", + "\n", + "Set your project ID below. If you know know your project ID, leave the field blank and the following cells may be able to find it. Optionally, you may also set a service account in the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cd85f5c794e5" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5a604eeffc32" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "30e64c0eda41" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b12f508d97c6" + }, + "source": [ + "### Region\n", + "You may change the `REGION` variable, which is used for Vertex Forecasting operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8e8b7997de7a" + }, + "outputs": [], + "source": [ + "REGION = \"us-central1\" # @param {type: \"string\"}\n", + "\n", + "if REGION == \"[your-region]\":\n", + " REGION = \"us-central1\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eu0e2TRVxjHb" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "- In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "- **Click Create service account**.\n", + "\n", + "- In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "- In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "- Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "- Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_nJ1VqdTxosw" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your GCP account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "import os\n", + "import sys\n", + "\n", + "# If on Vertex AI Workbench, then don't execute this code\n", + "IS_COLAB = \"google.colab\" in sys.modules\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n", + " \"DL_ANACONDA_HOME\"\n", + "):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS '[your-service-account-key-path]'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OUfwWir9yPNV" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "All training related files (TF model checkpoint, TensorBoard file, etc) will be saved to the GCS bucket. The pipeline will not clean up the files since some of them might be useful for you, **please make sure to clean up the files**. For easy cleanup, you can set [GCS bucket level TTL](https://cloud.google.com/storage/docs/lifecycle).\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5325f437b46a" + }, + "outputs": [], + "source": [ + "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n", + "GENERATE_BUCKET_URI = True # @param {type:\"boolean\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a30ea8551126" + }, + "source": [ + "Create the bucket if it doesn't already exist." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0217c63ed87f" + }, + "outputs": [], + "source": [ + "import uuid\n", + "\n", + "if GENERATE_BUCKET_URI:\n", + " bucket_name = \"gs://test-{}\".format(uuid.uuid4())\n", + " !gsutil mb -p {PROJECT_ID} -l {REGION} {bucket_name}\n", + "\n", + " # set GCS bucket object TTL to 7 days\n", + " !echo '{\"rule\":[{\"action\": {\"type\": \"Delete\"},\"condition\": {\"age\": 7}}]}' > gcs_lifecycle.tmp\n", + " !gsutil lifecycle set gcs_lifecycle.tmp {bucket_name}\n", + " !rm gcs_lifecycle.tmp\n", + "\n", + " BUCKET_URI = bucket_name\n", + " print(f\"changed BUCKET_URI to {BUCKET_URI} due to GENERATE_BUCKET_URI is True\")\n", + "\n", + "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", + " BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + uuid.uuid4()\n", + "\n", + "! gsutil ls -b $BUCKET_URI || gsutil mb -l $DATA_REGION $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c4cf2cdebb50" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "96ad3d416327" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Service Account\n", + "\n", + "You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "SERVICE_ACCOUNT = \"[your-service-account]\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "# If on Vertex AI Workbench, then don't execute this code\n", + "IS_COLAB = \"google.colab\" in sys.modules\n", + "if (\n", + " SERVICE_ACCOUNT == \"\"\n", + " or SERVICE_ACCOUNT is None\n", + " or SERVICE_ACCOUNT == \"[your-service-account]\"\n", + "):\n", + " # Get your service account from gcloud\n", + " if not IS_COLAB:\n", + " shell_output = !gcloud auth list 2>/dev/null\n", + " SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n", + "\n", + " else: # IS_COLAB:\n", + " shell_output = ! gcloud projects describe $PROJECT_ID\n", + " project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n", + " SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n", + "\n", + " print(\"Service Account:\", SERVICE_ACCOUNT)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Set service account access for Vertex AI Pipelines\n", + "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n", + "\n", + "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fbbc3479a1da" + }, + "source": [ + "## Import libraries and define constants" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8G6YmJT1yqkV" + }, + "outputs": [], + "source": [ + "# Import required modules\n", + "import json\n", + "from typing import Any, Dict, List\n", + "from google.cloud import aiplatform, storage\n", + "from google_cloud_pipeline_components.experimental.automl.tabular import \\\n", + " utils as automl_tabular_utils" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c0423f260423" + }, + "source": [ + "## Initialize Vertex SDK for Python\n", + "\n", + "Initialize the Vertex SDK for Python for your project." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ad69f2590268" + }, + "outputs": [], + "source": [ + "aiplatform.init(project=PROJECT_ID, location=REGION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3LWH3PRF5o2v" + }, + "source": [ + "### Define helper functions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "g9FPFT8c5oC0" + }, + "outputs": [], + "source": [ + "def get_bucket_name_and_path(uri):\n", + " no_prefix_uri = uri[len(\"gs://\") :]\n", + " splits = no_prefix_uri.split(\"/\")\n", + " return splits[0], \"/\".join(splits[1:])\n", + "\n", + "\n", + "def download_from_gcs(uri):\n", + " bucket_name, path = get_bucket_name_and_path(uri)\n", + " storage_client = storage.Client(project=PROJECT_ID)\n", + " bucket = storage_client.get_bucket(bucket_name)\n", + " blob = bucket.blob(path)\n", + " return blob.download_as_string()\n", + "\n", + "\n", + "def write_to_gcs(uri: str, content: str):\n", + " bucket_name, path = get_bucket_name_and_path(uri)\n", + " storage_client = storage.Client()\n", + " bucket = storage_client.get_bucket(bucket_name)\n", + " blob = bucket.blob(path)\n", + " blob.upload_from_string(content)\n", + "\n", + "\n", + "def generate_auto_transformation(column_names: List[str]) -> List[Dict[str, Any]]:\n", + " transformations = []\n", + " for column_name in column_names:\n", + " transformations.append({\"auto\": {\"column_name\": column_name}})\n", + " return transformations\n", + "\n", + "\n", + "def write_auto_transformations(uri: str, column_names: List[str]):\n", + " transformations = generate_auto_transformation(column_names)\n", + " write_to_gcs(uri, json.dumps(transformations))\n", + "\n", + "\n", + "def get_task_detail(\n", + " task_details: List[Dict[str, Any]], task_name: str\n", + ") -> List[Dict[str, Any]]:\n", + " for task_detail in task_details:\n", + " if task_detail.task_name == task_name:\n", + " return task_detail\n", + "\n", + "\n", + "def get_deployed_model_uri(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-upload\")\n", + " return ensemble_task.outputs[\"model\"].artifacts[0].uri\n", + "\n", + "\n", + "def get_no_custom_ops_model_uri(task_details):\n", + " ensemble_task = get_task_detail(task_details, \"automl-tabular-ensemble\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"model_without_custom_ops\"].artifacts[0].uri\n", + " )\n", + "\n", + "\n", + "def get_feature_attributions(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-evaluation-2\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"evaluation_metrics\"]\n", + " .artifacts[0]\n", + " .metadata[\"explanation_gcs_path\"]\n", + " )\n", + "\n", + "\n", + "def get_evaluation_metrics(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-evaluation\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"evaluation_metrics\"].artifacts[0].uri\n", + " )\n", + "\n", + "\n", + "def load_and_print_json(s):\n", + " parsed = json.loads(s)\n", + " print(json.dumps(parsed, indent=2, sort_keys=True))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gvNFMRmBegZq" + }, + "source": [ + "### Define training specification" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "eV4JrwB8wAkg" + }, + "outputs": [], + "source": [ + "run_evaluation = True # @param {type:\"boolean\"}\n", + "run_distillation = False # @param {type:\"boolean\"}\n", + "root_dir = os.path.join(BUCKET_URI, \"automl_tabular_pipeline\")\n", + "prediction_type = \"classification\"\n", + "optimization_objective = \"minimize-log-loss\"\n", + "target_column = \"deposit\"\n", + "data_source_csv_filenames = (\n", + " \"gs://cloud-samples-data/vertex-ai/tabular-workflows/datasets/bank-marketing/train.csv\"\n", + ")\n", + "data_source_bigquery_table_path = None # format: bq://bq_project.bq_dataset.bq_table\n", + "\n", + "timestamp_split_key = None # timestamp column name when using timestamp split\n", + "stratified_split_key = None # target column name when using stratified split\n", + "training_fraction = 0.8\n", + "validation_fraction = 0.1\n", + "test_fraction = 0.1\n", + "\n", + "predefined_split_key = None\n", + "if predefined_split_key:\n", + " training_fraction = None\n", + " validation_fraction = None\n", + " test_fraction = None\n", + "\n", + "weight_column = None\n", + "\n", + "features = [\n", + " \"age\",\n", + " \"job\",\n", + " \"marital\",\n", + " \"education\",\n", + " \"default\",\n", + " \"balance\",\n", + " \"housing\",\n", + " \"loan\",\n", + " \"contact\",\n", + " \"day\",\n", + " \"month\",\n", + " \"duration\",\n", + " \"campaign\",\n", + " \"pdays\",\n", + " \"previous\",\n", + " \"poutcome\",\n", + "]\n", + "transformations = generate_auto_transformation(features)\n", + "transform_config_path = os.path.join(root_dir, f\"transform_config_{uuid.uuid4()}.json\")\n", + "write_to_gcs(transform_config_path, json.dumps(transformations))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zyWGg2s09xOk" + }, + "source": [ + "## VPC related config\n", + "\n", + "If you need to use a custom Dataflow subnetwork, you can set it through the `dataflow_subnetwork` parameter. The requirements are:\n", + "1. `dataflow_subnetwork` must be fully qualified subnetwork name.\n", + " [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications)]\n", + "1. The following service accounts must have [Compute Network User role](https://cloud.google.com/compute/docs/access/iam#compute.networkUser) assigned on the specified dataflow subnetwork [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#shared)]:\n", + " 1. Compute Engine default service account: PROJECT_NUMBER-compute@developer.gserviceaccount.com\n", + " 1. Dataflow service account: service-PROJECT_NUMBER@dataflow-service-producer-prod.iam.gserviceaccount.com\n", + "\n", + "If your project has VPC-SC enabled, please make sure:\n", + "\n", + "1. The dataflow subnetwork used in VPC-SC is configured properly for Dataflow.\n", + " [[reference](https://cloud.google.com/dataflow/docs/guides/routes-firewall)]\n", + "1. `dataflow_use_public_ips` is set to False.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_TePNlLl9v1q" + }, + "outputs": [], + "source": [ + "# Dataflow's fully qualified subnetwork name, when empty the default subnetwork will be used.\n", + "# Fully qualified subnetwork name is in the form of\n", + "# https://www.googleapis.com/compute/v1/projects/HOST_PROJECT_ID/regions/REGION_NAME/subnetworks/SUBNETWORK_NAME\n", + "# reference: https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications\n", + "dataflow_subnetwork = None # @param {type:\"string\"}\n", + "# Specifies whether Dataflow workers use public IP addresses.\n", + "dataflow_use_public_ips = True # @param {type:\"boolean\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N-iXXE14voyR" + }, + "source": [ + "## Customize search space and change training configuration\n", + "\n", + "We will create a skip evaluation AutoML Tables pipeline with the following customizations:\n", + "- Limit the hyperparameter search space\n", + "- Change machine type and tuning / training parallelism" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sG46cXVueb66" + }, + "outputs": [], + "source": [ + "study_spec_parameters_override = [\n", + " {\n", + " \"parameter_id\": \"model_type\",\n", + " \"categorical_value_spec\": {\n", + " \"values\": [\n", + " \"nn\"\n", + " ] # The default value is [\"nn\", \"boosted_trees\"], this reduces the search space\n", + " },\n", + " }\n", + "]\n", + "\n", + "worker_pool_specs_override = [\n", + " {\"machine_spec\": {\"machine_type\": \"n1-standard-8\"}}, # override for TF chief node\n", + " {}, # override for TF worker node, since it's not used, leave it empty\n", + " {}, # override for TF ps node, since it's not used, leave it empty\n", + " {\n", + " \"machine_spec\": {\n", + " \"machine_type\": \"n1-standard-4\" # override for TF evaluator node\n", + " }\n", + " },\n", + "]\n", + "\n", + "# Number of weak models in the final ensemble model is\n", + "# stage_2_num_selected_trials * 5. If unspecified, 5 is the default value for\n", + "# stage_2_num_selected_trials.\n", + "stage_2_num_selected_trials = 5\n", + "\n", + "# The pipeline output a TF saved model contains the following TF custom op:\n", + "# - https://github.com/google/struct2tensor\n", + "#\n", + "# There are a few ways to run the model:\n", + "# - Official prediction server docker image\n", + "# Please follow the \"Run the model server\" section in\n", + "# https://cloud.google.com/vertex-ai/docs/export/export-model-tabular#run-server\n", + "# - Python or cpp runtimes like TF serving\n", + "# Please set export_additional_model_without_custom_ops so the pipeline\n", + "# outputs an additional model does does not depend on struct2tensor.\n", + "# - `get_no_custom_ops_model_uri` shows how to get the model artifact URI.\n", + "# - The input to the model is a dictionary of feature name to tensor. Use\n", + "# `saved_model_cli show --dir {saved_model.pb's path} --signature_def serving_default --tag serve`\n", + "# to find out more details.\n", + "export_additional_model_without_custom_ops = False\n", + "\n", + "train_budget_milli_node_hours = 1000 # 1 hour\n", + "\n", + "(\n", + " template_path,\n", + " parameter_values,\n", + ") = automl_tabular_utils.get_automl_tabular_pipeline_and_parameters(\n", + " PROJECT_ID,\n", + " REGION,\n", + " root_dir,\n", + " target_column,\n", + " prediction_type,\n", + " optimization_objective,\n", + " transform_config_path,\n", + " train_budget_milli_node_hours,\n", + " data_source_csv_filenames=data_source_csv_filenames,\n", + " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", + " weight_column=weight_column,\n", + " predefined_split_key=predefined_split_key,\n", + " timestamp_split_key=timestamp_split_key,\n", + " stratified_split_key=stratified_split_key,\n", + " training_fraction=training_fraction,\n", + " validation_fraction=validation_fraction,\n", + " test_fraction=test_fraction,\n", + " study_spec_parameters_override=study_spec_parameters_override,\n", + " stage_1_tuner_worker_pool_specs_override=worker_pool_specs_override,\n", + " cv_trainer_worker_pool_specs_override=worker_pool_specs_override,\n", + " run_evaluation=run_evaluation,\n", + " run_distillation=run_distillation,\n", + " dataflow_subnetwork=dataflow_subnetwork,\n", + " dataflow_use_public_ips=dataflow_use_public_ips,\n", + " export_additional_model_without_custom_ops=export_additional_model_without_custom_ops,\n", + ")\n", + "\n", + "job_id = \"automl-tabular-{}\".format(uuid.uuid4())\n", + "job = aiplatform.PipelineJob(\n", + " display_name=job_id,\n", + " location=REGION, # launches the pipeline job in the specified region\n", + " template_path=template_path,\n", + " job_id=job_id,\n", + " pipeline_root=root_dir,\n", + " parameter_values=parameter_values,\n", + " enable_caching=False,\n", + ")\n", + "\n", + "job.run()\n", + "\n", + "pipeline_task_details = job.gca_resource.job_detail.task_details\n", + "\n", + "if export_additional_model_without_custom_ops:\n", + " print(\n", + " \"trained model without custom TF ops:\",\n", + " get_no_custom_ops_model_uri(pipeline_task_details),\n", + " )\n", + "\n", + "if run_evaluation:\n", + " print(\"evaluation metrics:\")\n", + " load_and_print_json(get_evaluation_metrics(pipeline_task_details))\n", + "\n", + " print(\"feature attributions:\")\n", + " load_and_print_json(get_feature_attributions(pipeline_task_details))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_sF8a2RKtRhg" + }, + "source": [ + "## Skip architecture search\n", + "Instead of doing architecture search everytime, we can reuse the existing architecture search result. This could help:\n", + "1. reducing the variation of the output model\n", + "2. reducing training cost\n", + "\n", + "The existing architecture search result is stored in the `tuning_result_output` output of the `automl-tabular-stage-1-tuner` component. We can manually input it or get it programmatically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8hlPps2Rtpq-" + }, + "outputs": [], + "source": [ + "stage_1_tuner_task = get_task_detail(\n", + " pipeline_task_details, \"automl-tabular-stage-1-tuner\"\n", + ")\n", + "stage_1_tuning_result_artifact_uri = (\n", + " stage_1_tuner_task.outputs[\"tuning_result_output\"].artifacts[0].uri\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c5F12ZL_uZZ3" + }, + "source": [ + "### Run the skip architecture search pipeline\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4x4GA5sMuewX" + }, + "outputs": [], + "source": [ + "(\n", + " template_path,\n", + " parameter_values,\n", + ") = automl_tabular_utils.get_skip_architecture_search_pipeline_and_parameters(\n", + " PROJECT_ID,\n", + " REGION,\n", + " root_dir,\n", + " target_column,\n", + " prediction_type,\n", + " optimization_objective,\n", + " transform_config_path,\n", + " train_budget_milli_node_hours,\n", + " data_source_csv_filenames=data_source_csv_filenames,\n", + " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", + " weight_column=weight_column,\n", + " predefined_split_key=predefined_split_key,\n", + " timestamp_split_key=timestamp_split_key,\n", + " stratified_split_key=stratified_split_key,\n", + " training_fraction=training_fraction,\n", + " validation_fraction=validation_fraction,\n", + " test_fraction=test_fraction,\n", + " stage_1_tuning_result_artifact_uri=stage_1_tuning_result_artifact_uri,\n", + " run_evaluation=run_evaluation,\n", + " dataflow_subnetwork=dataflow_subnetwork,\n", + " dataflow_use_public_ips=dataflow_use_public_ips,\n", + ")\n", + "\n", + "job_id = \"automl-tabular-skip-architecture-search-{}\".format(uuid.uuid4())\n", + "job = aiplatform.PipelineJob(\n", + " display_name=job_id,\n", + " location=REGION, # launches the pipeline job in the specified region\n", + " template_path=template_path,\n", + " job_id=job_id,\n", + " pipeline_root=root_dir,\n", + " parameter_values=parameter_values,\n", + " enable_caching=False,\n", + ")\n", + "\n", + "job.run()\n", + "\n", + "# Get model URI\n", + "skip_architecture_search_pipeline_task_details = (\n", + " job.gca_resource.job_detail.task_details\n", + ")\n", + "\n", + "if export_additional_model_without_custom_ops:\n", + " print(\n", + " \"trained model without custom TF ops:\",\n", + " get_no_custom_ops_model_uri(pipeline_task_details),\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "43342a43176e" + }, + "source": [ + "## Clean up Vertex and BigQuery resources\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ad8d12061a65" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING\"):\n", + " ! gsutil rm -r $BUCKET_URI" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "automl_tabular_on_vertex_pipelines.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.2" + } + }, + "nbformat": 4, + "nbformat_minor": 4 }