xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore,refactor(egen): Changed REGION variable name to LOCATION, removed IS_TESTING from the cleaning up section, refactored code according to template guidelines and performed linter test. (#3280)

Browse Source 
* chore,refactor(egen): Changed REGION variable name to LOCATION, removed IS_TESTING from the cleaning up section, refactored code according to template guidelines and performed linter test.

* chore(egen): Done changes according to @kittyabs review and performed linter test.

---------

Co-authored-by: sriramya2610 <sriramya.peddapally@egen.ai>
This commit is contained in:
Sri Ramya - SpringML
2024-07-21 17:46:16 +00:00
committed by GitHub
co-authored by sriramya2610
parent 3238d99f08
commit 67abf6ad8b
1 changed files with 84 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb
+84 -136
View File
@@ -32,25 +32,27 @@
"# Tabular Workflows: TabNet Pipeline\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Ftabular_workflows%2Ftabnet_on_vertex_pipelines.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
"</table>\n",
"<br/><br/><br/>\n"
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
{
@@ -138,12 +140,19 @@
{
"cell_type": "markdown",
"metadata": {
"id": "i7EUnXsZhAGF"
"id": "2b9e4bcab250"
},
"source": [
"## Installation\n",
"\n",
"Install the following packages required to execute this notebook."
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "be898f74332d"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages"
]
},
{
@@ -161,60 +170,80 @@
{
"cell_type": "markdown",
"metadata": {
"id": "restart"
"id": "16220914acc5"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel.\n"
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bzPxhxS5lugp"
"id": "157953ab28f0"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d2qpIurSjmpT"
"id": "b96b39fd4d7b"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"3. [Enable the following APIs: Vertex AI API, Cloud Resource Manager API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,cloudresourcemanager.googleapis.com).\n",
"\n",
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "project_id"
"id": "ff666ce4051c"
},
"source": [
"#### Set your project ID\n",
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cc7251520a07"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b02382a1fea6"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -226,89 +255,7 @@
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a54f9d7c1876"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "region"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5c0404984792"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nt8cEM2GjmpU"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XUSL_JcpjmpU"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_2zemfGvjmpU"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TCPJ38n7jmpU"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"LOCATION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
@@ -356,7 +303,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -378,7 +325,7 @@
"source": [
"#### Service Account\n",
"\n",
"You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
"You use a service account to create Vertex AI Pipeline jobs. If you don't want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
]
},
{
@@ -489,7 +436,7 @@
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
@@ -532,7 +479,7 @@
" task = get_task_detail(task_details, \"model-upload\")\n",
" # in format https://<location>-aiplatform.googleapis.com/v1/projects/<project_number>/locations/<location>/models/<model_id>\n",
" model_id = task.outputs[\"model\"].artifacts[0].uri.split(\"/\")[-1]\n",
" return f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/models/{model_id}?project={PROJECT_ID}\"\n",
" return f\"https://console.cloud.google.com/vertex-ai/locations/{LOCATION}/models/{model_id}?project={PROJECT_ID}\"\n",
"\n",
"\n",
"# Get the bucket name and path.\n",
@@ -641,6 +588,7 @@
" * Full automatic transformations: FTE automatically configures a set of built-in transformations for each input column based on its data statistics. This can be set via `tf_auto_transform_features` in the training pipeline.\n",
" * Fully specified transformations: All transformations on input columns are explicitly specified with FTE's built-in transformations. Chaining of multiple transformations on a single column is also supported. These transformations can be saved to JSON configuration file and specified via `tf_transformations_path` argument of the training pipeline.\n",
" * Custom transformations: Custom, bring-your-own transform function, where you can define and import your own transform function and use it with other FTE's built-in transformations. You can specify custom transformations as an array of JSON object and pass through the `tf_custom_transformation_definitions` argument of the training pipeline.\n",
" \n",
"\n",
"* **BigQuery-based dataset-level transformations**:\n",
" * Fully specified transformations: All transformations on input columns are explicitly specified with FTE's built-in transformations. These transformations can be specified as an array of JSON objects via `dataset_level_transformations` argument of the training pipeline.\n",
@@ -870,7 +818,7 @@
" parameter_values,\n",
") = automl_tabular_utils.get_tabnet_trainer_pipeline_and_parameters(\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
" location=LOCATION,\n",
" root_dir=pipeline_job_root_dir,\n",
" max_steps=max_steps,\n",
" max_train_secs=max_train_secs,\n",
@@ -944,7 +892,7 @@
"source": [
"## Customize TabNet HyperparameterTuningJob configuration and create pipeline\n",
"\n",
"To get the best set of hyperparameters on your dataset, it is recommended to run a HyperparameterTuningJob.\n",
"To get the best set of hyperparameters on your dataset, it's recommended that you run a HyperparameterTuningJob.\n",
"\n",
"Hyperparameters that can be tuned are set with the optional `study_spec_parameters_override` parameter. You provide a helper function named `get_tabnet_study_spec_parameters_override` to get these hyperparameters. To this helper function, you provide:\n",
"\n",
@@ -1025,7 +973,7 @@
" parameter_values,\n",
") = automl_tabular_utils.get_tabnet_hyperparameter_tuning_job_pipeline_and_parameters(\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
" location=LOCATION,\n",
" root_dir=pipeline_job_root_dir,\n",
" target_column=target_column,\n",
" prediction_type=prediction_type,\n",
@@ -1137,9 +1085,9 @@
"custom_job_model.delete()\n",
"hpt_job_model.delete()\n",
"\n",
"# Delete bucket\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
"# Delete Cloud Storage objects that were created\n",
"delete_bucket = False # Set True for deletion\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_URI"
]
}