xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix,chore,refactor(egen): Changed CLUSTER_REGION variable name to CLUSTER_LOCATION, added gcloud command to enable dataproc api, modified bigquery dataset name by replacing hyphens to underscores, removed uuid code generation and replaced uuid with unique, refactored code according to template guidelines and perfomed linter test. (#3286)

Browse Source 
Co-authored-by: sriramya2610 <sriramya.peddapally@egen.ai>
This commit is contained in:
Sri Ramya - SpringML
2024-07-21 17:40:26 +00:00
committed by GitHub
co-authored by sriramya2610
parent e5e68af5fa
commit 5388cd53f7
1 changed files with 116 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/workbench/spark/spark_bigquery.ipynb
+116 -72
View File
@@ -32,24 +32,26 @@
"# Digest and analyze data from BigQuery with Dataproc\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
"<a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\" target='_blank'>\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\" target='_blank'>\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fworkbench%2Fspark%2Fspark_bigquery.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_bigquery.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
@@ -80,8 +82,8 @@
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Dataproc`\n",
"- `BigQuery`\n",
"- Dataproc\n",
"- BigQuery\n",
"\n",
"The steps performed are:\n",
"\n",
@@ -124,15 +126,22 @@
"You can use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4b331e2fd155"
},
"source": [
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WReHDGG5g0XY"
},
"source": [
"### Installation\n",
"\n",
"Install the following packages to run this notebook."
"### Install Vertex AI SDK for Python and other required packages"
]
},
{
@@ -178,6 +187,74 @@
" ! pip install pyspark==$PYSPARK_VER -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "16220914acc5"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "157953ab28f0"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b96b39fd4d7b"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ff666ce4051c"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cc7251520a07"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -199,15 +276,17 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" CLUSTER_NAME = \"[your-cluster]\" # @param {type: \"string\"}\n",
" CLUSTER_REGION = \"[your-region]\" # @param {type: \"string\"}\n",
" CLUSTER_LOCATION = \"[your-location]\" # @param {type: \"string\"}\n",
"\n",
" if CLUSTER_REGION == \"[your-region]\":\n",
" CLUSTER_REGION = \"us-central1\"\n",
" if CLUSTER_LOCATION == \"[your-location]\":\n",
" CLUSTER_LOCATION = \"us-central1\"\n",
"\n",
" print(f\"CLUSTER_NAME: {CLUSTER_NAME}\")\n",
" print(f\"CLUSTER_REGION: {CLUSTER_REGION}\")"
" print(f\"CLUSTER_LOCATION: {CLUSTER_LOCATION}\")"
]
},
{
@@ -218,9 +297,12 @@
},
"outputs": [],
"source": [
"# Enable the Dataproc api\n",
"! gcloud services enable dataproc.googleapis.com\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" !gcloud dataproc clusters create $CLUSTER_NAME \\\n",
" --region=$CLUSTER_REGION \\\n",
" --region=$CLUSTER_LOCATION \\\n",
" --enable-component-gateway \\\n",
" --image-version=2.0 \\\n",
" --optional-components=JUPYTER"
@@ -254,12 +336,9 @@
"id": "WReHDGG5g0XY"
},
"source": [
"### Set your project ID\n",
"### Set Google Cloud project information\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -270,10 +349,7 @@
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
@@ -287,37 +363,6 @@
"The Spark DataFrame created in this tutorial is stored in BigQuery."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "06571eb4063b"
},
"source": [
"#### UUID\n",
"\n",
"To avoid name collisions, you can create a UUID for the current notebook session, then append the UUID to the BigQuery dataset that you create in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "697568e92bd6"
},
"outputs": [],
"source": [
"import random\n",
"import string\n",
"\n",
"\n",
"# Generate a uuid of a specifed length(default=8)\n",
"def generate_uuid(length: int = 8) -> str:\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -343,9 +388,10 @@
" or DATASET_NAME is None\n",
" or DATASET_NAME == \"[your-dataset-name]\"\n",
" ):\n",
" DATASET_NAME = f\"{PROJECT_ID}{UUID}\"\n",
" DATASET_NAME = PROJECT_ID.replace(\"-\", \"_\")\n",
" DATASET_NAME = f\"{DATASET_NAME}_unique\"\n",
"else:\n",
" DATASET_NAME = f\"python_docs_samples_tests_spark_{UUID}\""
" DATASET_NAME = \"python_docs_samples_tests_spark_unique\""
]
},
{
@@ -365,8 +411,6 @@
"id": "XoEqT2Y4DJmf"
},
"source": [
"## Tutorial\n",
"\n",
"### Import required libraries"
]
},
@@ -1130,7 +1174,7 @@
"source": [
"### Delete Dataproc Cluster\n",
"\n",
"It is not possible to delete the cluster you are currently using unless you switch the kernel to local. To delete it, you need to switch the kernel to local `Python 3` or `PySpark`, set your `CLUSTER_NAME` and `CLUSTER_REGION` manually in the following cell, and execute the `gcloud` command.\n",
"It's not possible to delete the cluster you're currently using unless you switch the kernel to local. To delete it, you need to switch the kernel to local `Python 3` or `PySpark`, set your `CLUSTER_NAME` and `CLUSTER_LOCATION` manually in the following cell, and execute the `gcloud` command.\n",
"\n",
"See [Deleting a cluster](https://cloud.google.com/dataproc/docs/guides/manage-cluster#console) to delete the Dataproc cluster created in this tutorial."
]
@@ -1144,7 +1188,7 @@
"outputs": [],
"source": [
"CLUSTER_NAME = \"[your-cluster-name]\"\n",
"CLUSTER_REGION = \"[your-cluster-region]\""
"CLUSTER_LOCATION = \"[your-cluster-location]\""
]
},
{
@@ -1156,7 +1200,7 @@
"outputs": [],
"source": [
"if not os.getenv(\"IS_TESTING\"):\n",
" ! gcloud dataproc clusters delete $CLUSTER_NAME --region=$CLUSTER_REGION -q"
" ! gcloud dataproc clusters delete $CLUSTER_NAME --region=$CLUSTER_LOCATION -q"
]
}
],