xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore, refactor(egen): edits sdk_automl_forecasting_hierarchical_batch notebook (#3327)

Browse Source 
* chore, refactor:  refactors and edits according to the template.

* chore: lint

* chore: update objective and other verbiage

* chore: edits title of the notebook
This commit is contained in:
Kaushik Koilada
2024-08-14 23:59:28 +00:00
committed by GitHub
parent 34ffeaab3a
commit 04cc933895
1 changed files with 93 additions and 195 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/automl/sdk_automl_forecasting_hierarchical_batch.ipynb
+93 -195
View File
@@ -29,23 +29,28 @@
"id": "title"
},
"source": [
"# Vertex AI SDK for Python: AutoML training hierarchical forecasting for batch prediction\n",
"# Vertex AI SDK for Python: Vertex AI AutoML training hierarchical forecasting for batch prediction\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_forecasting_hierarchical_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fautoml%2Fsdk_automl_forecasting_hierarchical_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_forecasting_hierarchical_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br>\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/automl/sdk_automl_forecasting_hierarchical_batch.ipynb\" target='_blank'>\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br>\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -61,7 +66,7 @@
"## Overview\n",
"\n",
"\n",
"This tutorial demonstrates how to use the Vertex AI SDK for Python to create hierarchical forecasting models using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users)and do batch prediction. Specifically, you predict a fictional store's sales based on historical sales data.\n",
"This tutorial demonstrates how to use the Vertex AI SDK for Python to create hierarchical forecasting models using Google Cloud Vertex AI and do batch prediction. Specifically, you predict a fictional store's sales based on historical sales data.\n",
"\n",
"Learn more about [Hierarchical forecasting for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/forecasting/hierarchical)."
]
@@ -75,16 +80,16 @@
"### Objective\n",
"\n",
"In this tutorial, you create an AutoML hierarchical forecasting model and deploy it for batch prediction using the Vertex AI SDK for Python. You can alternatively create and deploy models using the `gcloud` command-line tool or batch using the Cloud Console.\n",
"The rationale for a hierarchical forecasting model is to minimize the error for a given group of sales data. In this tutorial, you will be minimizing the error for sale predictions at the \"product\" level.\n",
"The rationale for a hierarchical forecasting model is to minimize the error for a given group of sales data. The objective of this tutorial is to minimize the error for sale predictions at the \"product\" level.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"This tutorial uses the following Google Cloud Vertex AI services:\n",
"\n",
"- `AutoML Training`\n",
"- `Vertex AI Datasets`\n",
"- AutoML training\n",
"- Vertex AI datasets\n",
"\n",
"The steps performed include:\n",
"\n",
"- Create a Vertex AI `TimeSeriesDataset` resource.\n",
"- Create a Vertex AI TimeSeriesDataset resource.\n",
"- Train the model.\n",
"- View the model evaluation.\n",
"- Make a batch prediction."
@@ -139,183 +144,121 @@
{
"cell_type": "markdown",
"metadata": {
"id": "install_aip"
"id": "f0316df526f8"
},
"source": [
"## Installation\n",
"\n",
"Install the latest version of Cloud Storage, Bigquery and Vertex AI SDKs for Python."
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a2c2cb2109a0"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1fd00fa70a2a"
"id": "6319584dc783"
},
"outputs": [],
"source": [
"# Install the packages\n",
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
" google-cloud-storage \\\n",
" google-cloud-bigquery[pandas] \\\n",
" seaborn \\\n",
" scikit-learn"
"! pip3 install --upgrade google-cloud-aiplatform google-cloud-storage google-cloud-bigquery[pandas] seaborn scikit-learn"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "blGlVGFYW9Pt"
"id": "ff555b32bab8"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel."
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0JrvuK6LUYnQ"
"id": "f09b4dff629a"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a47846030fef"
"id": "ee775571c2b5"
},
"source": [
"## Before you begin"
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "project_id"
"id": "92e68cfc3a90"
},
"source": [
"#### Set your project ID\n",
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3c8049930470"
"id": "46604f70e831"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4f872cd812d0"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "294fe4e5a671"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a54f9d7c1876"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3aaadaaf9b30"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5c0404984792"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BaFKzJ_xXpvm"
},
"source": [
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_fV-KyGAX4Xl"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7uXB1HAPX6L_"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ab_TRMQIYCCX"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vx25htmYYExI"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uZdA0-jBYGqt"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
]
},
{
@@ -337,7 +280,7 @@
},
"outputs": [],
"source": [
"BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}"
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -346,7 +289,7 @@
"id": "create_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
@@ -357,7 +300,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
},
{
@@ -382,52 +325,7 @@
"from google.cloud import aiplatform\n",
"\n",
"# Initialize the Vertex AI SDK\n",
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "setup_vars"
},
"source": [
"### Set up variables\n",
"\n",
"Next, set up some variables used throughout the tutorial.\n",
"### Import libraries and define constants"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "import_aip:mbsdk"
},
"outputs": [],
"source": [
"import google.cloud.aiplatform as aiplatform"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk"
},
"source": [
"## Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project and the corresponding bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "init_aip:mbsdk"
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
"aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
@@ -449,11 +347,11 @@
"source": [
"### Create the Dataset\n",
"\n",
"Use `TimeSeriesDataset.create()` to create a `TimeSeriesDataset` resource, which takes the following parameters:\n",
"Use `TimeSeriesDataset.create()` to create a TimeSeriesDataset resource, which takes the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `Dataset` resource.\n",
"- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n",
"- `bq_source`: Alternatively, import data items from a BigQuery table into the `Dataset` resource.\n",
"- `display_name`: The human readable name for the dataset resource.\n",
"- `gcs_source`: A list of one or more dataset index files to import the data items into the dataset resource.\n",
"- `bq_source`: Alternatively, import data items from a BigQuery table into the dataset resource.\n",
"\n",
"This operation may take several minutes."
]
@@ -594,7 +492,7 @@
"\n",
"Create an AutoML training pipeline using the `AutoMLForecastingTrainingJob` class, with the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `TrainingJob` resource.\n",
"- `display_name`: The human readable name for the TrainingJob resource.\n",
"- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
"- `optimization_objective`: The optimization objective (minimize or maximize).\n",
" - regression:\n",
@@ -664,7 +562,7 @@
"\n",
"Run the training job by invoking the `run` method with the following parameters:\n",
"\n",
"- `dataset`: The `Dataset` resource to train the model.\n",
"- `dataset`: The dataset resource to train the model.\n",
"- `model_display_name`: The human readable name for the trained model.\n",
"- `training_fraction_split`: The percentage of the dataset to use for training.\n",
"- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).\n",
@@ -672,7 +570,7 @@
"- `target_column`: The name of the column to train as the label.\n",
"- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).\n",
"\n",
"The `run` method when completed returns the `Model` resource.\n",
"The `run` method when completed returns the model resource.\n",
"\n",
"#### Setting the hierarchical parameters\n",
"We want to group by 'product' to minimize the error at this level.\n",
@@ -912,7 +810,7 @@
"outputs": [],
"source": [
"batch_predict_bq_output_uri_prefix = create_bigquery_dataset(\n",
" name=\"hierarchical_forecasting_unique\", region=REGION\n",
" name=\"hierarchical_forecasting_unique\", region=LOCATION\n",
")"
]
},
@@ -924,7 +822,7 @@
"source": [
"### Make the batch prediction request\n",
"\n",
"You can make a batch prediction by invoking the batch_predict() method, with the following parameters:\n",
"You can make a batch prediction by invoking the `batch_predict()` method, with the following parameters:\n",
"\n",
"- `job_display_name`: The human readable name for the batch prediction job.\n",
"- `gcs_source`: A list of one or more batch request input files.\n",