fix: todo

notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb

@@ -0,0 +1,689 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "ur8xi4C7S06n"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2023 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fsv4jGuU89rX"
+      },
+      "source": [
+        "# Train a TabNet model using Vertex AI Remote Training with Vertex AI SDK 2.0\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
+        "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "    <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb\">\n",
+        "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "</table>\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:automl"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI SDK 2.0 for remote TabNet model training of a local TabNet model training job."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cTelQGJjifdu"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn to use `Vertex AI SDK 2.0` to remotely train a TabNet model as a local (on-prem) training job.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI Training`\n",
+        "- `Vertex AI Remote Training`\n",
+        "\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Download and split the dataset\n",
+        "- Ingest the data in a Dataframe and perform transformations.\n",
+        "- Train a tabular classification model.\n",
+        "- Train a tabular regression model.\n",
+        "\n",
+        "\n",
+        "**TabNet remote training**\n",
+        "```\n",
+        "from google.cloud.aiplatform.private_preview.vertex_ai.tabular_models import TabNetTrainer\n",
+        "\n",
+        "vertex_ai.init(remote=True, project=\"my-project\", location=\"my-location\", staging_bucket=\"gs://my-bucket\")\n",
+        "\n",
+        "# Instantiate TabNetTrainer\n",
+        "tabnet_trainer = TabNetTrainer(...)\n",
+        "\n",
+        "# Optional set training config\n",
+        "tabnet_trainer.fit.vertex.training_config.staging_bucket = \"gs://my-bucket\"\n",
+        "\n",
+        "# This `fit` call will be executed remotely\n",
+        "tabnet_trainer.fit(...)\n",
+        "\n",
+        "# This `predict` call will be executed locally\n",
+        "tabnet_trainer.predict(...)\n",
+        "```\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aef4f59195ad"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "This tutorial uses the <a href=\"https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\">IRIS dataset</a>, which predicts the iris species."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "costs"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_aip:mbsdk"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the following packages required to execute this notebook."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "F9dJ5Of-dORl"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install --upgrade --quiet google-cloud-aiplatform[preview] \\\n",
+        "                                 scikit-learn \\\n",
+        "                                 pyarrow \\\n",
+        "                                 fastparquet \\\n",
+        "                                 tensorflow"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Colab only: Uncomment the following cell to restart the kernel"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "D-ZBOjErv5mM"
+      },
+      "outputs": [],
+      "source": [
+        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
+        "# import IPython\n",
+        "\n",
+        "# app = IPython.Application.instance()\n",
+        "# app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "before_you_begin:nogpu"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "\n",
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2dw8q9fdQEH5"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gcp_authenticate"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
+        "\n",
+        "**1. Vertex AI Workbench**\n",
+        "* Do nothing as you are already authenticated.\n",
+        "\n",
+        "**2. Local JupyterLab instance, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ce6043da7b33"
+      },
+      "outputs": [],
+      "source": [
+        "# ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0367eac06a10"
+      },
+      "source": [
+        "**3. Colab, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "21ad4dbb4a61"
+      },
+      "outputs": [],
+      "source": [
+        "# from google.colab import auth\n",
+        "# auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "c13224697bfb"
+      },
+      "source": [
+        "**4. Service account or other**\n",
+        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "Create a storage bucket to store intermediate artifacts such as datasets."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "91c46850b49b"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "LF2bwT6q-of1"
+      },
+      "outputs": [],
+      "source": [
+        "import uuid\n",
+        "\n",
+        "import pandas as pd\n",
+        "import vertexai.preview\n",
+        "from sklearn.datasets import load_iris\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "from vertexai.preview.tabular_models import TabNetTrainer"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk"
+      },
+      "source": [
+        "## Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "-p0YAOY64cA5"
+      },
+      "outputs": [],
+      "source": [
+        "vertexai.init(\n",
+        "    project=PROJECT_ID,\n",
+        "    location=REGION,\n",
+        "    staging_bucket=BUCKET_URI,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "innAVhRdxe8c"
+      },
+      "source": [
+        "### Prepare the dataset\n",
+        "\n",
+        "Now load the Iris dataset and split the data into train, evaluation and test sets."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "HPT2EX7ii9jO"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = load_iris()\n",
+        "\n",
+        "X, X_validation, y, y_validation = train_test_split(\n",
+        "    dataset.data, dataset.target, test_size=0.60, random_state=42\n",
+        ")\n",
+        "X_train, X_test, y_train, y_test = train_test_split(\n",
+        "    X, y, test_size=0.20, random_state=42\n",
+        ")\n",
+        "\n",
+        "print(\"Data size: \", len(dataset.target))\n",
+        "print(\"X_train size: \", len(X_train))\n",
+        "print(\"X_validation size: \", len(X_validation))\n",
+        "print(\"X_test size: \", len(X_test))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XzQfxEwuY22S"
+      },
+      "source": [
+        "## Feature transformation\n",
+        "\n",
+        "Next, you do ingest the training and evaluation data in pandas Dataframe and perform feature transformations."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sAHGB2nYjcDg"
+      },
+      "outputs": [],
+      "source": [
+        "columns = [f\"arg_{i}\" for i in range(X_train.shape[1])]\n",
+        "\n",
+        "training_data = pd.DataFrame(X_train, columns=columns)\n",
+        "training_data[\"target\"] = y_train\n",
+        "training_data[\"target\"] = training_data[\"target\"].astype(\"category\")\n",
+        "\n",
+        "validation_data = pd.DataFrame(X_validation, columns=columns)\n",
+        "validation_data[\"target\"] = y_validation\n",
+        "validation_data[\"target\"] = validation_data[\"target\"].astype(\"category\")\n",
+        "\n",
+        "eval_data = pd.DataFrame(X_test, columns=columns)\n",
+        "\n",
+        "training_data.head()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0kyuTJLli5eB"
+      },
+      "source": [
+        "## Tabular classification model"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "LE4qd1T3kfpR"
+      },
+      "source": [
+        "### Initialize TabNetTrainer\n",
+        "\n",
+        "Next, initialize the TabNetTrainer for training a tabular classification model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "eGXmfieOklSs"
+      },
+      "outputs": [],
+      "source": [
+        "tabnet_trainer = TabNetTrainer(\n",
+        "    model_type=\"classification\",\n",
+        "    target_column=\"target\",\n",
+        "    learning_rate=0.01,\n",
+        "    max_train_secs=1800,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "HU1a1bf9k5sY"
+      },
+      "source": [
+        "### Remote training\n",
+        "\n",
+        "Now, train a TabNet tabular classification model as a remote training job."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Hk3Gp95blJ4K"
+      },
+      "outputs": [],
+      "source": [
+        "REMOTE_JOB_NAME = \"test-sdk2-remote-tabnet-training\"\n",
+        "REMOTE_JOB_BUCKET = \"/\".join([BUCKET_URI, REMOTE_JOB_NAME])\n",
+        "\n",
+        "# Sets the staging bucket for remote training\n",
+        "tabnet_trainer.fit.vertex.remote_config.staging_bucket = (\n",
+        "    REMOTE_JOB_BUCKET + f\"_{uuid.uuid4()}\"\n",
+        ")\n",
+        "\n",
+        "# Executes remote training\n",
+        "tabnet_trainer.fit(training_data=training_data, validation_data=validation_data)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vyoVwMoPlYUi"
+      },
+      "source": [
+        "### Local prediction\n",
+        "\n",
+        "Finally, make a local prediction with the trained tabular classification model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c_3rQyLeleig"
+      },
+      "outputs": [],
+      "source": [
+        "from sklearn.metrics import accuracy_score\n",
+        "\n",
+        "predictions = tabnet_trainer.predict(eval_data)\n",
+        "accuracy_score(y_test, predictions)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tXCf6wCHwvAt"
+      },
+      "source": [
+        "## Tabular regression model"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "HtTBSRnu0RGX"
+      },
+      "source": [
+        "### Initialize TabNetTrainer\n",
+        "\n",
+        "Next, initialize the TabNetTrainer for training a tabular regression model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "-8hp__390V__"
+      },
+      "outputs": [],
+      "source": [
+        "tabnet_trainer = TabNetTrainer(\n",
+        "    model_type=\"regression\",\n",
+        "    target_column=\"target\",\n",
+        "    learning_rate=0.01,\n",
+        "    max_train_secs=1800,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "k3CDcyfw0g9K"
+      },
+      "source": [
+        "### Remote training\n",
+        "\n",
+        "Now, train a TabNet tabular regression model as a remote training job."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "I3OgDbb405N1"
+      },
+      "outputs": [],
+      "source": [
+        "# Sets the staging bucket for remote training\n",
+        "tabnet_trainer.fit.vertex.remote_config.staging_bucket = (\n",
+        "    REMOTE_JOB_BUCKET + f\"_{uuid.uuid4()}\"\n",
+        ")\n",
+        "\n",
+        "# Executes remote training\n",
+        "tabnet_trainer.fit(training_data=training_data, validation_data=validation_data)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vyoVwMoPlYUi"
+      },
+      "source": [
+        "### Local prediction\n",
+        "\n",
+        "Finally, make a local prediction with the trained tabular regression model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "x_y_cwbx0-0O"
+      },
+      "outputs": [],
+      "source": [
+        "from sklearn.metrics import mean_squared_error\n",
+        "\n",
+        "predictions = tabnet_trainer.predict(eval_data)\n",
+        "mean_squared_error(y_test, predictions, squared=False)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6944380b1e25"
+      },
+      "source": [
+        "## Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "655e39c67585"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "delete_bucket = False\n",
+        "\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -rf {BUCKET_URI}"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "sdk2_remote_tabnet_training.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}