diff --git a/notebooks/official/CODEOWNERS b/notebooks/official/CODEOWNERS index 02c5fea99..d56d8cde0 100644 --- a/notebooks/official/CODEOWNERS +++ b/notebooks/official/CODEOWNERS @@ -24,3 +24,4 @@ /pipelines/google_cloud_pipeline_components_bqml_text.ipynb @inardini /pipelines/google_cloud_pipelines_dataproc_tabular @inardini /automl/automl_forecasting_bqml_arima_plus_comparison.ipynb @TheMichaelHu +/automl/automl_tabular_on_vertex_pipelines.ipynb @helinwang diff --git a/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb b/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb new file mode 100644 index 000000000..e3ca7d1fa --- /dev/null +++ b/notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb @@ -0,0 +1,1026 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "18ebbd838e32" + }, + "outputs": [], + "source": [ + "# Copyright 2022 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mThXALJl9Yue" + }, + "source": [ + "# Tabular Workflow: AutoML Tabular Pipeline\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + " \n", + " \n", + " \"Vertex\n", + " Open in Vertex AI Workbench\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fcc745968395" + }, + "source": [ + "## Overview\n", + "\n", + "In this tutorial, you will use two Vertex AI Tabular Workflows pipelines to train AutoML models using different configurations. You will see how `get_automl_tabular_pipeline_and_parameters` gives you the ability to customize the default AutoML Tabular pipeline, and how `get_skip_architecture_search_pipeline_and_parameters` allows you to reduce the training time and cost for an AutoML model by using the tuning results from a previous pipeline run." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eac26958afe8" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset you will be using is the [Safe Driver Prediction](https://www.kaggle.com/competitions/porto-seguro-safe-driver-prediction/data?select=train.csv) dataset for predicting the probability of an auto insurance policy holder filing a claim for a given incident." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f887ec5c06c5" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you create two regression models using [Vertex Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) downloaded from [Google Cloud Pipeline Components](https://cloud.google.com/vertex-ai/docs/pipelines/components-introduction) (GCPC). These pipelines will be Vertex AI Tabular Workflow pipelines which are maintained by Google. These pipelines will showcase different ways to customize the Vertex Tabular training process.\n", + "\n", + "The steps performed are:\n", + "\n", + "- Create a training pipeline that reduces the search space from the default to save time.\n", + "- Create a training pipeline that reuses the architecture search results from the previous pipeline to save time." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "181d4dfbf917" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tuOonx6suOb7" + }, + "source": [ + "### Set up your local development environment\n", + "\n", + "**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets all the requirements to run this notebook. You can skip this step.\n", + "\n", + "**Otherwise**, make sure your environment meets this notebook's requirements. You need the following:\n", + "\n", + "- The Cloud Storage SDK\n", + "- Python 3\n", + "- virtualenv\n", + "- Jupyter notebook running in a virtual environment with Python 3\n", + "\n", + "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n", + "\n", + "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n", + "\n", + "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n", + "\n", + "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n", + "\n", + "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n", + "\n", + "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", + "\n", + "6. Open this notebook in the Jupyter Notebook Dashboard." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e85f0288a6df" + }, + "source": [ + "## Install additional packages\n", + "\n", + "Install the latest version of the Google Cloud Pipeline Components (GCPC) SDK." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "E7-SzYTR9bo2" + }, + "outputs": [], + "source": [ + "!pip install -U google-cloud-pipeline-components -q" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bj5O0S5RTxzY" + }, + "source": [ + "### Restart the kernel\n", + "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.\n", + "\n", + "\n", + "**Note: Once this cell has finished running, continue on. You do not need to re-run any of the cells above.**\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "023DMKUaTypt" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yfEglUHQk9S3" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "This tutorial does not require a GPU runtime.\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the following APIs: Vertex AI APIs, Dataflow APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,dataflow.googleapis.com,compute_component,storage-component.googleapis.com)\n", + "\n", + "4. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zebLBGXOky2A" + }, + "source": [ + "## Notes about service account and permission\n", + "\n", + "**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n", + "\n", + "|Service account email|Description|Roles|\n", + "|---|---|---|\n", + "|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Admin, Dataflow Worker, Storage Admin, BigQuery Admin, Vertex AI User|\n", + "|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n", + "\n", + "\n", + "1. Goto https://console.cloud.google.com/iam-admin/iam.\n", + "2. Check the \"Include Google-provided role grants\" checkbox.\n", + "3. Find the above emails.\n", + "4. Grant the corresponding roles.\n", + "\n", + "### Using data source from a different project\n", + "- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n", + "- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "95cb7ffd6895" + }, + "source": [ + "### Set your project ID\n", + "\n", + "Set your project ID below. If you know know your project ID, leave the field blank and the following cells may be able to find it. Optionally, you may also set a service account in the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cd85f5c794e5" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5a604eeffc32" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "30e64c0eda41" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b12f508d97c6" + }, + "source": [ + "### Region\n", + "You may change the `REGION` variable, which is used for Vertex Forecasting operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8e8b7997de7a" + }, + "outputs": [], + "source": [ + "REGION = \"[your-region]\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eu0e2TRVxjHb" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "- In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "- **Click Create service account**.\n", + "\n", + "- In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "- In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "- Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "- Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_nJ1VqdTxosw" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your GCP account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "import os\n", + "import sys\n", + "\n", + "# If on Google Cloud Notebook, then don't execute this code\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account. Alternatively, you may edit this notebook to authenticate using\n", + " # gcloud.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OUfwWir9yPNV" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "All training related files (TF model checkpoint, TensorBoard file, etc) will be saved to the GCS bucket. The pipeline will not clean up the files since some of them might be useful for you, **please make sure to clean up the files**. For easy cleanup, you can set [GCS bucket level TTL](https://cloud.google.com/storage/docs/lifecycle).\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5325f437b46a" + }, + "outputs": [], + "source": [ + "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n", + "GENERATE_BUCKET_URI = True # @param {type:\"boolean\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a30ea8551126" + }, + "source": [ + "Create the bucket if it doesn't already exist." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0217c63ed87f" + }, + "outputs": [], + "source": [ + "import uuid\n", + "\n", + "if GENERATE_BUCKET_URI:\n", + " bucket_name = \"gs://test-{}\".format(uuid.uuid4())\n", + " !gsutil mb -p {PROJECT_ID} -l {REGION} {bucket_name}\n", + "\n", + " # set GCS bucket object TTL to 7 days\n", + " !echo '{\"rule\":[{\"action\": {\"type\": \"Delete\"},\"condition\": {\"age\": 7}}]}' > gcs_lifecycle.tmp\n", + " !gsutil lifecycle set gcs_lifecycle.tmp {bucket_name}\n", + " !rm gcs_lifecycle.tmp\n", + "\n", + " BUCKET_URI = bucket_name\n", + " print(f\"changed BUCKET_URI to {BUCKET_URI} due to GENERATE_BUCKET_URI is True\")\n", + "\n", + "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", + " BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + uuid.uuid4()\n", + "\n", + "! gsutil ls -b $BUCKET_URI || gsutil mb -l $DATA_REGION $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c4cf2cdebb50" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "96ad3d416327" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fbbc3479a1da" + }, + "source": [ + "## Import libraries and define constants" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8G6YmJT1yqkV" + }, + "outputs": [], + "source": [ + "# Import required modules\n", + "import json\n", + "from typing import Any, Dict, List\n", + "\n", + "from google.cloud import aiplatform, storage\n", + "from google_cloud_pipeline_components.experimental.automl.tabular import \\\n", + " utils as automl_tabular_utils" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c0423f260423" + }, + "source": [ + "## Initialize Vertex SDK for Python\n", + "\n", + "Initialize the Vertex SDK for Python for your project." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ad69f2590268" + }, + "outputs": [], + "source": [ + "aiplatform.init(project=PROJECT_ID, location=REGION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3LWH3PRF5o2v" + }, + "source": [ + "### Define helper functions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "g9FPFT8c5oC0" + }, + "outputs": [], + "source": [ + "def get_bucket_name_and_path(uri):\n", + " no_prefix_uri = uri[len(\"gs://\") :]\n", + " splits = no_prefix_uri.split(\"/\")\n", + " return splits[0], \"/\".join(splits[1:])\n", + "\n", + "\n", + "def download_from_gcs(uri):\n", + " bucket_name, path = get_bucket_name_and_path(uri)\n", + " storage_client = storage.Client(project=PROJECT_ID)\n", + " bucket = storage_client.get_bucket(bucket_name)\n", + " blob = bucket.blob(path)\n", + " return blob.download_as_string()\n", + "\n", + "\n", + "def write_to_gcs(uri: str, content: str):\n", + " bucket_name, path = get_bucket_name_and_path(uri)\n", + " storage_client = storage.Client()\n", + " bucket = storage_client.get_bucket(bucket_name)\n", + " blob = bucket.blob(path)\n", + " blob.upload_from_string(content)\n", + "\n", + "\n", + "def generate_auto_transformation(column_names: List[str]) -> List[Dict[str, Any]]:\n", + " transformations = []\n", + " for column_name in column_names:\n", + " transformations.append({\"auto\": {\"column_name\": column_name}})\n", + " return transformations\n", + "\n", + "\n", + "def write_auto_transformations(uri: str, column_names: List[str]):\n", + " transformations = generate_auto_transformation(column_names)\n", + " write_to_gcs(uri, json.dumps(transformations))\n", + "\n", + "\n", + "def get_task_detail(\n", + " task_details: List[Dict[str, Any]], task_name: str\n", + ") -> List[Dict[str, Any]]:\n", + " for task_detail in task_details:\n", + " if task_detail.task_name == task_name:\n", + " return task_detail\n", + "\n", + "\n", + "def get_deployed_model_uri(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-upload\")\n", + " return ensemble_task.outputs[\"model\"].artifacts[0].uri\n", + "\n", + "\n", + "def get_no_custom_ops_model_uri(task_details):\n", + " ensemble_task = get_task_detail(task_details, \"automl-tabular-ensemble\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"model_without_custom_ops\"].artifacts[0].uri\n", + " )\n", + "\n", + "\n", + "def get_feature_attributions(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-evaluation-2\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"evaluation_metrics\"]\n", + " .artifacts[0]\n", + " .metadata[\"explanation_gcs_path\"]\n", + " )\n", + "\n", + "\n", + "def get_evaluation_metrics(\n", + " task_details,\n", + "):\n", + " ensemble_task = get_task_detail(task_details, \"model-evaluation\")\n", + " return download_from_gcs(\n", + " ensemble_task.outputs[\"evaluation_metrics\"].artifacts[0].uri\n", + " )\n", + "\n", + "\n", + "def load_and_print_json(s):\n", + " parsed = json.loads(s)\n", + " print(json.dumps(parsed, indent=2, sort_keys=True))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gvNFMRmBegZq" + }, + "source": [ + "### Define training specification" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "eV4JrwB8wAkg" + }, + "outputs": [], + "source": [ + "run_evaluation = True # @param {type:\"boolean\"}\n", + "run_distillation = False # @param {type:\"boolean\"}\n", + "root_dir = os.path.join(BUCKET_URI, \"automl_tabular_pipeline\")\n", + "prediction_type = \"classification\"\n", + "optimization_objective = \"minimize-log-loss\"\n", + "target_column = \"target\"\n", + "data_source_csv_filenames = (\n", + " \"gs://cloud-samples-data/vertex-ai/tabular-workflows/datasets/safe-driver/train.csv\"\n", + ")\n", + "data_source_bigquery_table_path = None # format: bq://bq_project.bq_dataset.bq_table\n", + "\n", + "timestamp_split_key = None # timestamp column name when using timestamp split\n", + "stratified_split_key = None # target column name when using stratified split\n", + "training_fraction = 0.8\n", + "validation_fraction = 0.1\n", + "test_fraction = 0.1\n", + "\n", + "predefined_split_key = None\n", + "if predefined_split_key:\n", + " training_fraction = None\n", + " validation_fraction = None\n", + " test_fraction = None\n", + "\n", + "weight_column = None\n", + "\n", + "features = [\n", + " \"ps_ind_01\",\n", + " \"ps_ind_02_cat\",\n", + " \"ps_ind_03\",\n", + " \"ps_ind_04_cat\",\n", + " \"ps_ind_05_cat\",\n", + " \"ps_ind_06_bin\",\n", + " \"ps_ind_07_bin\",\n", + " \"ps_ind_08_bin\",\n", + " \"ps_ind_09_bin\",\n", + " \"ps_ind_10_bin\",\n", + " \"ps_ind_11_bin\",\n", + " \"ps_ind_12_bin\",\n", + " \"ps_ind_13_bin\",\n", + " \"ps_ind_14\",\n", + " \"ps_ind_15\",\n", + " \"ps_ind_16_bin\",\n", + " \"ps_ind_17_bin\",\n", + " \"ps_ind_18_bin\",\n", + " \"ps_reg_01\",\n", + " \"ps_reg_02\",\n", + " \"ps_reg_03\",\n", + " \"ps_car_01_cat\",\n", + " \"ps_car_02_cat\",\n", + " \"ps_car_03_cat\",\n", + " \"ps_car_04_cat\",\n", + " \"ps_car_05_cat\",\n", + " \"ps_car_06_cat\",\n", + " \"ps_car_07_cat\",\n", + " \"ps_car_08_cat\",\n", + " \"ps_car_09_cat\",\n", + " \"ps_car_10_cat\",\n", + " \"ps_car_11_cat\",\n", + " \"ps_car_11\",\n", + " \"ps_car_12\",\n", + " \"ps_car_13\",\n", + " \"ps_car_14\",\n", + " \"ps_car_15\",\n", + " \"ps_calc_01\",\n", + " \"ps_calc_02\",\n", + " \"ps_calc_03\",\n", + " \"ps_calc_04\",\n", + " \"ps_calc_05\",\n", + " \"ps_calc_06\",\n", + " \"ps_calc_07\",\n", + " \"ps_calc_08\",\n", + " \"ps_calc_09\",\n", + " \"ps_calc_10\",\n", + " \"ps_calc_11\",\n", + " \"ps_calc_12\",\n", + " \"ps_calc_13\",\n", + " \"ps_calc_14\",\n", + " \"ps_calc_15_bin\",\n", + " \"ps_calc_16_bin\",\n", + " \"ps_calc_17_bin\",\n", + " \"ps_calc_18_bin\",\n", + " \"ps_calc_19_bin\",\n", + " \"ps_calc_20_bin\",\n", + "]\n", + "transformations = generate_auto_transformation(features)\n", + "transform_config_path = os.path.join(root_dir, f\"transform_config_{uuid.uuid4()}.json\")\n", + "write_to_gcs(transform_config_path, json.dumps(transformations))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zyWGg2s09xOk" + }, + "source": [ + "## VPC related config\n", + "\n", + "If you need to use a custom Dataflow subnetwork, you can set it through the `dataflow_subnetwork` parameter. The requirements are:\n", + "1. `dataflow_subnetwork` must be fully qualified subnetwork name.\n", + " [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications)]\n", + "1. The following service accounts must have [Compute Network User role](https://cloud.google.com/compute/docs/access/iam#compute.networkUser) assigned on the specified dataflow subnetwork [[reference](https://cloud.google.com/dataflow/docs/guides/specifying-networks#shared)]:\n", + " 1. Compute Engine default service account: PROJECT_NUMBER-compute@developer.gserviceaccount.com\n", + " 1. Dataflow service account: service-PROJECT_NUMBER@dataflow-service-producer-prod.iam.gserviceaccount.com\n", + "\n", + "If your project has VPC-SC enabled, please make sure:\n", + "\n", + "1. The dataflow subnetwork used in VPC-SC is configured properly for Dataflow.\n", + " [[reference](https://cloud.google.com/dataflow/docs/guides/routes-firewall)]\n", + "1. `dataflow_use_public_ips` is set to False.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_TePNlLl9v1q" + }, + "outputs": [], + "source": [ + "# Dataflow's fully qualified subnetwork name, when empty the default subnetwork will be used.\n", + "# Fully qualified subnetwork name is in the form of\n", + "# https://www.googleapis.com/compute/v1/projects/HOST_PROJECT_ID/regions/REGION_NAME/subnetworks/SUBNETWORK_NAME\n", + "# reference: https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications\n", + "dataflow_subnetwork = \"\" # @param {type:\"string\"}\n", + "# Specifies whether Dataflow workers use public IP addresses.\n", + "dataflow_use_public_ips = True # @param {type:\"boolean\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N-iXXE14voyR" + }, + "source": [ + "## Customize search space and change training configuration\n", + "\n", + "We will create a skip evaluation AutoML Tables pipeline with the following customizations:\n", + "- Limit the hyperparameter search space\n", + "- Change machine type and tuning / training parallelism" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sG46cXVueb66" + }, + "outputs": [], + "source": [ + "study_spec_parameters_override = [\n", + " {\n", + " \"parameter_id\": \"model_type\",\n", + " \"categorical_value_spec\": {\n", + " \"values\": [\n", + " \"nn\"\n", + " ] # The default value is [\"nn\", \"boosted_trees\"], this reduces the search space\n", + " },\n", + " }\n", + "]\n", + "\n", + "worker_pool_specs_override = [\n", + " {\"machine_spec\": {\"machine_type\": \"n1-standard-8\"}}, # override for TF chief node\n", + " {}, # override for TF worker node, since it's not used, leave it empty\n", + " {}, # override for TF ps node, since it's not used, leave it empty\n", + " {\n", + " \"machine_spec\": {\n", + " \"machine_type\": \"n1-standard-4\" # override for TF evaluator node\n", + " }\n", + " },\n", + "]\n", + "\n", + "# Number of weak models in the final ensemble model is\n", + "# stage_2_num_selected_trials * 5. If unspecified, 5 is the default value for\n", + "# stage_2_num_selected_trials.\n", + "stage_2_num_selected_trials = 5\n", + "\n", + "# The pipeline output a TF saved model contains the following TF custom op:\n", + "# - https://github.com/google/struct2tensor\n", + "#\n", + "# There are a few ways to run the model:\n", + "# - Official prediction server docker image\n", + "# Please follow the \"Run the model server\" section in\n", + "# https://cloud.google.com/vertex-ai/docs/export/export-model-tabular#run-server\n", + "# - Python or cpp runtimes like TF serving\n", + "# Please set export_additional_model_without_custom_ops so the pipeline\n", + "# outputs an additional model does does not depend on struct2tensor.\n", + "# - `get_no_custom_ops_model_uri` shows how to get the model artifact URI.\n", + "# - The input to the model is a dictionary of feature name to tensor. Use\n", + "# `saved_model_cli show --dir {saved_model.pb's path} --signature_def serving_default --tag serve`\n", + "# to find out more details.\n", + "export_additional_model_without_custom_ops = False\n", + "\n", + "train_budget_milli_node_hours = 1000 # 1 hour\n", + "\n", + "(\n", + " template_path,\n", + " parameter_values,\n", + ") = automl_tabular_utils.get_automl_tabular_pipeline_and_parameters(\n", + " PROJECT_ID,\n", + " REGION,\n", + " root_dir,\n", + " target_column,\n", + " prediction_type,\n", + " optimization_objective,\n", + " transform_config_path,\n", + " train_budget_milli_node_hours,\n", + " data_source_csv_filenames=data_source_csv_filenames,\n", + " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", + " weight_column=weight_column,\n", + " predefined_split_key=predefined_split_key,\n", + " timestamp_split_key=timestamp_split_key,\n", + " stratified_split_key=stratified_split_key,\n", + " training_fraction=training_fraction,\n", + " validation_fraction=validation_fraction,\n", + " test_fraction=test_fraction,\n", + " study_spec_parameters_override=study_spec_parameters_override,\n", + " stage_1_tuner_worker_pool_specs_override=worker_pool_specs_override,\n", + " cv_trainer_worker_pool_specs_override=worker_pool_specs_override,\n", + " run_evaluation=run_evaluation,\n", + " run_distillation=run_distillation,\n", + ")\n", + "\n", + "job_id = \"automl-tabular-{}\".format(uuid.uuid4())\n", + "job = aiplatform.PipelineJob(\n", + " display_name=job_id,\n", + " location=REGION, # launches the pipeline job in the specified region\n", + " template_path=template_path,\n", + " job_id=job_id,\n", + " pipeline_root=root_dir,\n", + " parameter_values=parameter_values,\n", + " enable_caching=False,\n", + ")\n", + "\n", + "job.run()\n", + "\n", + "pipeline_task_details = job.gca_resource.job_detail.task_details\n", + "\n", + "if export_additional_model_without_custom_ops:\n", + " print(\n", + " \"trained model without custom TF ops:\",\n", + " get_no_custom_ops_model_uri(pipeline_task_details),\n", + " )\n", + "\n", + "if run_evaluation:\n", + " print(\"evaluation metrics:\")\n", + " load_and_print_json(get_evaluation_metrics(pipeline_task_details))\n", + "\n", + " print(\"feature attributions:\")\n", + " load_and_print_json(get_feature_attributions(pipeline_task_details))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_sF8a2RKtRhg" + }, + "source": [ + "## Skip architecture search\n", + "Instead of doing architecture search everytime, we can reuse the existing architecture search result. This could help:\n", + "1. reducing the variation of the output model\n", + "2. reducing training cost\n", + "\n", + "The existing architecture search result is stored in the `tuning_result_output` output of the `automl-tabular-stage-1-tuner` component. We can manually input it or get it programmatically." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8hlPps2Rtpq-" + }, + "outputs": [], + "source": [ + "stage_1_tuner_task = get_task_detail(\n", + " pipeline_task_details, \"automl-tabular-stage-1-tuner\"\n", + ")\n", + "stage_1_tuning_result_artifact_uri = (\n", + " stage_1_tuner_task.outputs[\"tuning_result_output\"].artifacts[0].uri\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c5F12ZL_uZZ3" + }, + "source": [ + "### Run the skip architecture search pipeline\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4x4GA5sMuewX" + }, + "outputs": [], + "source": [ + "(\n", + " template_path,\n", + " parameter_values,\n", + ") = automl_tabular_utils.get_skip_architecture_search_pipeline_and_parameters(\n", + " PROJECT_ID,\n", + " REGION,\n", + " root_dir,\n", + " target_column,\n", + " prediction_type,\n", + " optimization_objective,\n", + " transform_config_path,\n", + " train_budget_milli_node_hours,\n", + " data_source_csv_filenames=data_source_csv_filenames,\n", + " data_source_bigquery_table_path=data_source_bigquery_table_path,\n", + " weight_column=weight_column,\n", + " predefined_split_key=predefined_split_key,\n", + " timestamp_split_key=timestamp_split_key,\n", + " stratified_split_key=stratified_split_key,\n", + " training_fraction=training_fraction,\n", + " validation_fraction=validation_fraction,\n", + " test_fraction=test_fraction,\n", + " stage_1_tuning_result_artifact_uri=stage_1_tuning_result_artifact_uri,\n", + " run_evaluation=run_evaluation,\n", + ")\n", + "\n", + "job_id = \"automl-tabular-skip-architecture-search-{}\".format(uuid.uuid4())\n", + "job = aiplatform.PipelineJob(\n", + " display_name=job_id,\n", + " location=REGION, # launches the pipeline job in the specified region\n", + " template_path=template_path,\n", + " job_id=job_id,\n", + " pipeline_root=root_dir,\n", + " parameter_values=parameter_values,\n", + " enable_caching=False,\n", + ")\n", + "\n", + "job.run()\n", + "\n", + "# Get model URI\n", + "skip_architecture_search_pipeline_task_details = (\n", + " job.gca_resource.job_detail.task_details\n", + ")\n", + "\n", + "if export_additional_model_without_custom_ops:\n", + " print(\n", + " \"trained model without custom TF ops:\",\n", + " get_no_custom_ops_model_uri(pipeline_task_details),\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "43342a43176e" + }, + "source": [ + "## Clean up Vertex and BigQuery resources\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ad8d12061a65" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING\"):\n", + " ! gsutil rm -r $BUCKET_URI" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "automl_tabular_on_vertex_pipelines.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}