xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor, fix, chore(egen): edits get_started_bq_datasets (#3248)

Browse Source 
* chore: refactor according to template, removes boilerplate, adds colab enterprise

* refactore: adds testing variables

* fix, chore: end to end testing with version change as fix

* chore: lint

* chore: addresses review comments and runs lint
This commit is contained in:
Kaushik Koilada
2024-07-17 00:26:55 +00:00
committed by GitHub
parent 575d2f9479
commit 0791c52923
1 changed files with 112 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/datasets/get_started_bq_datasets.ipynb
+112 -159
View File
@@ -32,22 +32,28 @@
"# Get started with BigQuery datasets\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/datasets/get_started_bq_datasets.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/datasets/get_started_bq_datasets.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fdatasets%2Fget_started_bq_datasets.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/datasets/get_started_bq_datasets.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br>\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/datasets/get_started_bq_datasets.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br>\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"<br/><br/><br/>"
]
@@ -63,7 +69,7 @@
"\n",
"This tutorial demonstrates how to use Vertex AI in production. This tutorial covers data management: get started with BigQuery datasets.\n",
"\n",
"Learn more about [BigQuery Datasets](https://cloud.google.com/bigquery/docs/datasets-intro) and [Vertex AI for BigQuery users](https://cloud.google.com/vertex-ai/docs/beginner/bqml)."
"Learn more about [BigQuery datasets](https://cloud.google.com/bigquery/docs/datasets-intro) and [Vertex AI for BigQuery users](https://cloud.google.com/vertex-ai/docs/beginner/bqml)."
]
},
{
@@ -74,22 +80,22 @@
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to use `BigQuery` as a dataset for training with `Vertex AI`.\n",
"In this tutorial, you learn how to use BigQuery as a dataset for training with Vertex AI.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Datasets`\n",
"- `BigQuery Datasets`\n",
"- Vertex AI datasets\n",
"- BigQuery datasets\n",
"\n",
"The steps performed include:\n",
"\n",
"- Create a Vertex AI `Dataset` resource from `BigQuery` table -- compatible for `AutoML` training.\n",
"- Extract a copy of the dataset from `BigQuery` to a CSV file in Cloud Storage -- compatible for `AutoML` or custom training.\n",
"- Select rows from a `BigQuery` dataset into a `pandas` dataframe -- compatible for custom training.\n",
"- Select rows from a `BigQuery` dataset into a `tf.data.Dataset` -- compatible for custom training `TensorFlow` models.\n",
"- Select rows from extracted CSV files into a `tf.data.Dataset` -- compatible for custom training `TensorFlow` models.\n",
"- Create a `BigQuery` dataset from CSV files.\n",
"- Extract data from `BigQuery` table into a `DMatrix` -- compatible for custom training `XGBoost` models."
"- Create a Vertex AI dataset resource from BigQuery table -- compatible for AutoML training.\n",
"- Extract a copy of the dataset from BigQuery to a CSV file in Cloud Storage -- compatible for AutoML or custom training.\n",
"- Select rows from a BigQuery dataset into a pandas dataframe -- compatible for custom training.\n",
"- Select rows from a BigQuery dataset into a `tf.data.Dataset` -- compatible for custom training TensorFlow models.\n",
"- Select rows from extracted CSV files into a `tf.data.Dataset` -- compatible for custom training TensorFlow models.\n",
"- Create a BigQuery dataset from CSV files.\n",
"- Extract data from BigQuery table into a DMatrix -- compatible for custom training XGBoost models."
]
},
{
@@ -103,7 +109,7 @@
"When doing E2E MLOps on Google Cloud, following are the best practices when dealing with structured (tabular) data in BigQuery:\n",
"\n",
"- For AutoML training:\n",
" - Create a managed dataset with Vertex AI `TabularDataset`.\n",
" - Create a managed dataset with Vertex AI TabularDataset.\n",
" - Use the BigQuery table as the input to the dataset.\n",
" - Specify columns and columns transformations when running the AutoML training pipeline job.\n",
"\n",
@@ -164,164 +170,134 @@
{
"cell_type": "markdown",
"metadata": {
"id": "install_mlops"
"id": "d1ea81ac77f0"
},
"source": [
"## Installations\n",
"\n",
"Install the following packages to execute this notebook."
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e5d353aa47ac"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_mlops"
"id": "9bbcbe73e685"
},
"outputs": [],
"source": [
"! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
" google-cloud-bigquery \\\n",
" tensorflow \\\n",
" tensorflow-io==0.18 \\\n",
" tensorflow-io \\\n",
" xgboost \\\n",
" numpy \\\n",
" pandas \\\n",
" pyarrow"
" pyarrow \\\n",
" db-dtypes"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "restart"
"id": "16220914acc5"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel"
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "D-ZBOjErv5mM"
"id": "157953ab28f0"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yfEglUHQk9S3"
"id": "c87a2a5d7e35"
},
"source": [
"## Before you begin\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5dccb1c8feb6"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"### Set your project ID\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "set_project_id"
"id": "cc7251520a07"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c2fc3d7b6bfa"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "45769dc0c97d"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "region"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gcp_authenticate"
},
"source": [
"### Authenticate your Google Cloud account\n",
"from google.cloud import aiplatform\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
@@ -352,7 +328,7 @@
"id": "create_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
@@ -363,7 +339,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
"! gsutil mb -l $LOCATION $BUCKET_URI"
]
},
{
@@ -383,34 +359,11 @@
},
"outputs": [],
"source": [
"import google.cloud.aiplatform as aiplatform\n",
"import pandas as pd\n",
"import xgboost as xgb\n",
"from google.cloud import bigquery"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk,region"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "init_aip:mbsdk,region"
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -462,14 +415,14 @@
"id": "create_dataset:tabular,bq,lrg"
},
"source": [
"### Create the Dataset\n",
"### Create the dataset\n",
"\n",
"#### BigQuery input data\n",
"\n",
"Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
"Next, create the dataset resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `Dataset` resource.\n",
"- `bq_source`: Import data items from a BigQuery table into the `Dataset` resource.\n",
"- `display_name`: The human readable name for the dataset resource.\n",
"- `bq_source`: Import data items from a BigQuery table into the dataset resource.\n",
"- `labels`: User defined metadata. In this example, you store the location of the Cloud Storage bucket containing the user defined data.\n",
"\n",
"Learn more about [TabularDataset from BigQuery table](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-api#aiplatform_create_dataset_tabular_bigquery_sample-python)."
@@ -535,14 +488,14 @@
"id": "create_dataset:tabular,lrg"
},
"source": [
"### Create the Dataset\n",
"### Create the dataset\n",
"\n",
"#### CSV input data\n",
"\n",
"Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
"Next, create the dataset resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `Dataset` resource.\n",
"- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n",
"- `display_name`: The human readable name for the dataset resource.\n",
"- `gcs_source`: A list of one or more dataset index files to import the data items into the dataset resource.\n",
"- `labels`: User defined metadata. In this example, you store the location of the Cloud Storage bucket containing the user defined data.\n",
"\n",
"Learn more about [TabularDataset from CSV files](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-api#aiplatform_create_dataset_tabular_gcs_sample-python)"
@@ -1045,9 +998,9 @@
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
"- Vertex AI Dataset resource\n",
"- Vertex AI dataset resource\n",
"- Cloud Storage Bucket\n",
"- BigQuery Dataset\n",
"- BigQuery dataset\n",
"\n",
"Set `delete_storage` to _True_ to delete the storage resources used in this notebook."
]