xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(egen): Refactored the Custom tabular bq managed dataset code as per template guidelines (#3053)

Browse Source 
* refactor: Refactored custom-tabular-bq-managed-dataset code according to the template notebook

* refactor: Included markdown code in the beginning of the license cell to make it collapsible

* chore: linter test

* refactor, chore: Updated markdown according to the updated notebook template and performed linter test

* chore: Grouped all the imports present in the notebook

* chore: linter test

* refactor, chore(egen): REGION is replaced with LOCATION, linter test
This commit is contained in:
Rohith Alla
2024-06-11 17:53:12 +00:00
committed by GitHub
parent acb4fbb684
commit 04059ebadb
1 changed files with 118 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb
+118 -148
View File
@@ -4,6 +4,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "copyright"
},
"outputs": [],
@@ -32,23 +33,26 @@
"# Training a TensorFlow model on BigQuery data\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fcustom%2Fcustom-tabular-bq-managed-dataset.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td> \n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
@@ -128,12 +132,19 @@
{
"cell_type": "markdown",
"metadata": {
"id": "install_aip"
"id": "3b1ffd5ab768"
},
"source": [
"## Installation\n",
"\n",
"Install the latest version of Cloud Storage, Bigquery and Vertex AI SDKs for Python."
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dc848186ab0e"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages"
]
},
{
@@ -145,7 +156,7 @@
"outputs": [],
"source": [
"# Install the packages\n",
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
"! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
" google-cloud-storage \\\n",
" 'google-cloud-bigquery[pandas]'"
]
@@ -153,48 +164,80 @@
{
"cell_type": "markdown",
"metadata": {
"id": "blGlVGFYW9Pt"
"id": "ff555b32bab8"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel."
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0JrvuK6LUYnQ"
"id": "f09b4dff629a"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a47846030fef"
"id": "54c5ef8a8f43"
},
"source": [
"## Before you begin"
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "project_id"
"id": "f82e28c631cc"
},
"source": [
"#### Set your project ID\n",
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "46604f70e831"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "107c51893a64"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -206,103 +249,7 @@
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a54f9d7c1876"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3aaadaaf9b30"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5c0404984792"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BaFKzJ_xXpvm"
},
"source": [
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_fV-KyGAX4Xl"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7uXB1HAPX6L_"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ab_TRMQIYCCX"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vx25htmYYExI"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uZdA0-jBYGqt"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
]
},
{
@@ -344,7 +291,38 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4f1319830b6e"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "poijnGfZCFYi"
},
"source": [
"Import the Vertex AI Python SDK and other required Python libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b9a0a5a74fa6"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from google.cloud import aiplatform, bigquery"
]
},
{
@@ -366,10 +344,8 @@
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"# Initialize the Vertex AI SDK\n",
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
"# Initialize the Vertex AI SDK for Python\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
@@ -393,8 +369,6 @@
},
"outputs": [],
"source": [
"from google.cloud import bigquery\n",
"\n",
"# Set up BigQuery client\n",
"bq_client = bigquery.Client(project=PROJECT_ID)"
]
@@ -405,7 +379,7 @@
"id": "0a2c41bc91a6"
},
"source": [
"## Preprocess data and split data\n",
"### Preprocess data and split data\n",
"First you should download and preprocess your data for training and testing.\n",
"\n",
"- Convert categorical features to numeric\n",
@@ -422,9 +396,6 @@
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"LABEL_COLUMN = \"species\"\n",
"\n",
"# Define the BigQuery source dataset\n",
@@ -456,7 +427,7 @@
"id": "a39df4692a70"
},
"source": [
"## Create a Vertex AI Tabular Dataset from BigQuery dataset\n",
"### Create a Vertex AI Tabular dataset from BigQuery dataset\n",
"\n",
"Create a Vertex AI tabular dataset resource from your BigQuery training data.\n",
"\n",
@@ -498,7 +469,7 @@
"id": "train_custom_model"
},
"source": [
"## Train a model\n",
"### Train a model\n",
"\n",
"There are two ways you can train a model using a container image:\n",
"\n",
@@ -811,7 +782,7 @@
"id": "make_prediction"
},
"source": [
"## Make an online prediction request\n",
"### Make an online prediction request\n",
"\n",
"Send an online prediction request to your deployed model."
]
@@ -884,7 +855,7 @@
"id": "undeploy_model"
},
"source": [
"## Undeploy models\n",
"### Undeploy models\n",
"\n",
"To undeploy all `Model` resources from the serving `Endpoint` resource, use the endpoint's `undeploy_all` method."
]
@@ -908,7 +879,8 @@
"source": [
"## Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
@@ -926,8 +898,6 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# Delete the training job\n",
"job.delete()\n",
"\n",
@@ -938,9 +908,9 @@
"endpoint.delete()\n",
"\n",
"# Warning: Setting this to true deletes everything in your bucket\n",
"delete_bucket = False\n",
"delete_bucket = True\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI"
]
}