diff --git a/notebooks/official/datasets/get_started_bq_datasets.ipynb b/notebooks/official/datasets/get_started_bq_datasets.ipynb index e940cdd8e..8f63b1623 100644 --- a/notebooks/official/datasets/get_started_bq_datasets.ipynb +++ b/notebooks/official/datasets/get_started_bq_datasets.ipynb @@ -32,22 +32,28 @@ "# Get started with BigQuery datasets\n", "\n", "\n", - " \n", - " \n", + " \n", + " \n", "
\n", - " \n", - " \"GitHub\n", - " View on GitHub\n", - " \n", - " \n", + " \n", " \n", - " \"Colab Run in Colab\n", + " \"Colab
Open in Colab\n", " \n", " 		\n", + " \n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", " \n", - " \"Vertex\n", + " \"Vertex
\n", " Open in Vertex AI Workbench\n", " \n", " 		\n", + " \n", + " \"GitHub
\n", + " View on GitHub\n", + " \n", + "
\n", "


" ] @@ -63,7 +69,7 @@ "\n", "This tutorial demonstrates how to use Vertex AI in production. This tutorial covers data management: get started with BigQuery datasets.\n", "\n", - "Learn more about [BigQuery Datasets](https://cloud.google.com/bigquery/docs/datasets-intro) and [Vertex AI for BigQuery users](https://cloud.google.com/vertex-ai/docs/beginner/bqml)." + "Learn more about [BigQuery datasets](https://cloud.google.com/bigquery/docs/datasets-intro) and [Vertex AI for BigQuery users](https://cloud.google.com/vertex-ai/docs/beginner/bqml)." ] }, { @@ -74,22 +80,22 @@ "source": [ "### Objective\n", "\n", - "In this tutorial, you learn how to use `BigQuery` as a dataset for training with `Vertex AI`.\n", + "In this tutorial, you learn how to use BigQuery as a dataset for training with Vertex AI.\n", "\n", "This tutorial uses the following Google Cloud ML services:\n", "\n", - "- `Vertex AI Datasets`\n", - "- `BigQuery Datasets`\n", + "- Vertex AI datasets\n", + "- BigQuery datasets\n", "\n", "The steps performed include:\n", "\n", - "- Create a Vertex AI `Dataset` resource from `BigQuery` table -- compatible for `AutoML` training.\n", - "- Extract a copy of the dataset from `BigQuery` to a CSV file in Cloud Storage -- compatible for `AutoML` or custom training.\n", - "- Select rows from a `BigQuery` dataset into a `pandas` dataframe -- compatible for custom training.\n", - "- Select rows from a `BigQuery` dataset into a `tf.data.Dataset` -- compatible for custom training `TensorFlow` models.\n", - "- Select rows from extracted CSV files into a `tf.data.Dataset` -- compatible for custom training `TensorFlow` models.\n", - "- Create a `BigQuery` dataset from CSV files.\n", - "- Extract data from `BigQuery` table into a `DMatrix` -- compatible for custom training `XGBoost` models." + "- Create a Vertex AI dataset resource from BigQuery table -- compatible for AutoML training.\n", + "- Extract a copy of the dataset from BigQuery to a CSV file in Cloud Storage -- compatible for AutoML or custom training.\n", + "- Select rows from a BigQuery dataset into a pandas dataframe -- compatible for custom training.\n", + "- Select rows from a BigQuery dataset into a `tf.data.Dataset` -- compatible for custom training TensorFlow models.\n", + "- Select rows from extracted CSV files into a `tf.data.Dataset` -- compatible for custom training TensorFlow models.\n", + "- Create a BigQuery dataset from CSV files.\n", + "- Extract data from BigQuery table into a DMatrix -- compatible for custom training XGBoost models." ] }, { @@ -103,7 +109,7 @@ "When doing E2E MLOps on Google Cloud, following are the best practices when dealing with structured (tabular) data in BigQuery:\n", "\n", "- For AutoML training:\n", - " - Create a managed dataset with Vertex AI `TabularDataset`.\n", + " - Create a managed dataset with Vertex AI TabularDataset.\n", " - Use the BigQuery table as the input to the dataset.\n", " - Specify columns and columns transformations when running the AutoML training pipeline job.\n", "\n", @@ -164,164 +170,134 @@ { "cell_type": "markdown", "metadata": { - "id": "install_mlops" + "id": "d1ea81ac77f0" }, "source": [ - "## Installations\n", - "\n", - "Install the following packages to execute this notebook." + "## Get started" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e5d353aa47ac" + }, + "source": [ + "### Install Vertex AI SDK for Python and other required packages\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "install_mlops" + "id": "9bbcbe73e685" }, "outputs": [], "source": [ "! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n", " google-cloud-bigquery \\\n", " tensorflow \\\n", - " tensorflow-io==0.18 \\\n", + " tensorflow-io \\\n", " xgboost \\\n", " numpy \\\n", " pandas \\\n", - " pyarrow" + " pyarrow \\\n", + " db-dtypes" ] }, { "cell_type": "markdown", "metadata": { - "id": "restart" + "id": "16220914acc5" }, "source": [ - "### Colab only: Uncomment the following cell to restart the kernel" + "### Restart runtime (Colab only)\n", + "\n", + "To use the newly installed packages, you must restart the runtime on Google Colab." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "D-ZBOjErv5mM" + "id": "157953ab28f0" }, "outputs": [], "source": [ - "# Automatically restart kernel after installs so that your environment can access the new packages\n", - "# import IPython\n", + "import sys\n", "\n", - "# app = IPython.Application.instance()\n", - "# app.kernel.do_shutdown(True)" + "if \"google.colab\" in sys.modules:\n", + "\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "metadata": { - "id": "yfEglUHQk9S3" + "id": "c87a2a5d7e35" }, "source": [ - "## Before you begin\n", + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5dccb1c8feb6" + }, + "source": [ + "### Authenticate your notebook environment (Colab only)\n", "\n", - "### Set your project ID\n", - "\n", - "**If you don't know your project ID**, try the following:\n", - "* Run `gcloud config list`.\n", - "* Run `gcloud projects list`.\n", - "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" + "Authenticate your environment on Google Colab.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "set_project_id" + "id": "cc7251520a07" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c2fc3d7b6bfa" + }, + "source": [ + "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n", + "\n", + "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "45769dc0c97d" }, "outputs": [], "source": [ "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}\n", "\n", - "# Set the project id\n", - "! gcloud config set project {PROJECT_ID}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "region" - }, - "source": [ - "#### Region\n", "\n", - "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "region" - }, - "outputs": [], - "source": [ - "REGION = \"us-central1\" # @param {type: \"string\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gcp_authenticate" - }, - "source": [ - "### Authenticate your Google Cloud account\n", + "from google.cloud import aiplatform\n", "\n", - "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n", - "\n", - "**1. Vertex AI Workbench**\n", - "* Do nothing as you are already authenticated.\n", - "\n", - "**2. Local JupyterLab instance, uncomment and run:**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ce6043da7b33" - }, - "outputs": [], - "source": [ - "# ! gcloud auth login" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0367eac06a10" - }, - "source": [ - "**3. Colab, uncomment and run:**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "21ad4dbb4a61" - }, - "outputs": [], - "source": [ - "# from google.colab import auth\n", - "# auth.authenticate_user()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c13224697bfb" - }, - "source": [ - "**4. Service account or other**\n", - "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples." + "aiplatform.init(project=PROJECT_ID, location=LOCATION)" ] }, { @@ -352,7 +328,7 @@ "id": "create_bucket" }, "source": [ - "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." ] }, { @@ -363,7 +339,7 @@ }, "outputs": [], "source": [ - "! gsutil mb -l $REGION $BUCKET_URI" + "! gsutil mb -l $LOCATION $BUCKET_URI" ] }, { @@ -383,34 +359,11 @@ }, "outputs": [], "source": [ - "import google.cloud.aiplatform as aiplatform\n", "import pandas as pd\n", "import xgboost as xgb\n", "from google.cloud import bigquery" ] }, - { - "cell_type": "markdown", - "metadata": { - "id": "init_aip:mbsdk,region" - }, - "source": [ - "### Initialize Vertex AI SDK for Python\n", - "\n", - "Initialize the Vertex AI SDK for Python for your project and corresponding bucket." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "init_aip:mbsdk,region" - }, - "outputs": [], - "source": [ - "aiplatform.init(project=PROJECT_ID, location=REGION)" - ] - }, { "cell_type": "markdown", "metadata": { @@ -462,14 +415,14 @@ "id": "create_dataset:tabular,bq,lrg" }, "source": [ - "### Create the Dataset\n", + "### Create the dataset\n", "\n", "#### BigQuery input data\n", "\n", - "Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n", + "Next, create the dataset resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n", "\n", - "- `display_name`: The human readable name for the `Dataset` resource.\n", - "- `bq_source`: Import data items from a BigQuery table into the `Dataset` resource.\n", + "- `display_name`: The human readable name for the dataset resource.\n", + "- `bq_source`: Import data items from a BigQuery table into the dataset resource.\n", "- `labels`: User defined metadata. In this example, you store the location of the Cloud Storage bucket containing the user defined data.\n", "\n", "Learn more about [TabularDataset from BigQuery table](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-api#aiplatform_create_dataset_tabular_bigquery_sample-python)." @@ -535,14 +488,14 @@ "id": "create_dataset:tabular,lrg" }, "source": [ - "### Create the Dataset\n", + "### Create the dataset\n", "\n", "#### CSV input data\n", "\n", - "Next, create the `Dataset` resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n", + "Next, create the dataset resource using the `create` method for the `TabularDataset` class, which takes the following parameters:\n", "\n", - "- `display_name`: The human readable name for the `Dataset` resource.\n", - "- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n", + "- `display_name`: The human readable name for the dataset resource.\n", + "- `gcs_source`: A list of one or more dataset index files to import the data items into the dataset resource.\n", "- `labels`: User defined metadata. In this example, you store the location of the Cloud Storage bucket containing the user defined data.\n", "\n", "Learn more about [TabularDataset from CSV files](https://cloud.google.com/vertex-ai/docs/datasets/create-dataset-api#aiplatform_create_dataset_tabular_gcs_sample-python)" @@ -1045,9 +998,9 @@ "\n", "Otherwise, you can delete the individual resources you created in this tutorial:\n", "\n", - "- Vertex AI Dataset resource\n", + "- Vertex AI dataset resource\n", "- Cloud Storage Bucket\n", - "- BigQuery Dataset\n", + "- BigQuery dataset\n", "\n", "Set `delete_storage` to _True_ to delete the storage resources used in this notebook." ]