xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix,chore,refactor(egen): Changed REGION variable name to LOCATION, changed CLUSTER_REGION variable name to CLUSTER_LOCATION, added gcloud command to enable dataproc cluster, refactored code according to the template guidelines and performed linter test. (#3284)

Browse Source 
* fix,chore,refactor(egen): Changed REGION variable name to LOCATION, changed CLUSTER_REGION variable name to CLUSTER_LOCATION, added gcloud command to enable dataproc cluster, refactored code according to the template guidelines and performed linter test.

* chore(egen): Done changes according to @kittyabs review and performed linter test.

---------

Co-authored-by: sriramya2610 <sriramya.peddapally@egen.ai>
This commit is contained in:
Sri Ramya - SpringML
2024-07-21 17:44:03 +00:00
committed by GitHub
co-authored by sriramya2610
parent 5388cd53f7
commit 3238d99f08
1 changed files with 125 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/workbench/spark/spark_ml.ipynb
+125 -178
View File
@@ -32,24 +32,26 @@
"# SparkML with Dataproc and BigQuery\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
"<a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_ml.ipynb\" target='_blank'>\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_ml.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_ml.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/workbench/spark/spark_ml.ipynb\" target='_blank'>\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fworkbench%2Fspark%2Fspark_ml.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/workbench/spark/spark_ml.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_ml.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
@@ -78,9 +80,9 @@
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Dataproc`\n",
"- `BigQuery`\n",
"- `Vertex AI Training`\n",
"- Dataproc\n",
"- BigQuery\n",
"- Vertex AI Training\n",
"\n",
"The steps performed are:\n",
"\n",
@@ -127,15 +129,22 @@
"You can use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4b331e2fd155"
},
"source": [
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XoEqT2Y4DJmf"
},
"source": [
"### Installation\n",
"\n",
"Install the following packages:"
"### Install Vertex AI SDK for Python and other required packages"
]
},
{
@@ -172,6 +181,74 @@
" ! pip install pyspark==$PYSPARK_VER geopandas pyarrow rtree seaborn numpy==1.19.5 -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ff555b32bab8"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f09b4dff629a"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4a2b7b59bbf7"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f82e28c631cc"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "46604f70e831"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -195,16 +272,16 @@
},
"outputs": [],
"source": [
"# The testing environment does not use a Dataproc cluster so cluster creation is skipped during testing.\n",
"# The testing environment doesn't use a Dataproc cluster so cluster creation is skipped during testing.\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" CLUSTER_NAME = \"[your-cluster]\" # @param {type: \"string\"}\n",
" CLUSTER_REGION = \"[your-region]\" # @param {type: \"string\"}\n",
" CLUSTER_LOCATION = \"[your-location]\" # @param {type: \"string\"}\n",
"\n",
" if CLUSTER_REGION == \"[your-region]\":\n",
" CLUSTER_REGION = \"us-central1\"\n",
" if CLUSTER_LOCATION == \"[your-location]\":\n",
" CLUSTER_LOCATION = \"us-central1\"\n",
"\n",
" print(f\"CLUSTER_NAME: {CLUSTER_NAME}\")\n",
" print(f\"CLUSTER_REGION: {CLUSTER_REGION}\")"
" print(f\"CLUSTER_LOCATION: {CLUSTER_LOCATION}\")"
]
},
{
@@ -215,9 +292,12 @@
},
"outputs": [],
"source": [
"# Enable the Dataproc api\n",
"! gcloud services enable dataproc.googleapis.com\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" !gcloud dataproc clusters create $CLUSTER_NAME \\\n",
" --region=$CLUSTER_REGION \\\n",
" --region=$CLUSTER_LOCATION \\\n",
" --enable-component-gateway \\\n",
" --image-version=2.0 \\\n",
" --optional-components=JUPYTER"
@@ -242,12 +322,9 @@
"id": "XoEqT2Y4DJmf"
},
"source": [
"### Set your project ID\n",
"### Set Google Cloud project information\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"To get started using Vertex AI, you must have an existing Google Cloud project. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -260,134 +337,11 @@
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"LOCATION = \"[your-location]\" # @param {type: \"string\"}\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2dw8q9fdQEH5"
},
"outputs": [],
"source": [
"REGION = \"us-central1\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7f1992d08421"
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
"\n",
"# If you do not specify a region, it is set to \"us-central1\".\n",
"if not REGION or REGION == \"\" or REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XoEqT2Y4DJmf"
},
"source": [
"#### UUID\n",
"\n",
"To avoid name collisions you can create a UUID for the current notebook session, then append the UUID to the name of resources such as the Cloud Storage bucket and BigQuery dataset created in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6988e7272299"
},
"outputs": [],
"source": [
"import random\n",
"import string\n",
"\n",
"\n",
"# Generate a uuid of a specifed length(default=8)\n",
"def generate_uuid(length: int = 8) -> str:\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gcp_authenticate"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"# If you don't specify a location, it is set to \"us-central1\".\n",
"if not LOCATION or LOCATION == \"\" or LOCATION == \"[your-location]\":\n",
" LOCATION = \"us-central1\""
]
},
{
@@ -429,7 +383,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -438,8 +392,6 @@
"id": "XoEqT2Y4DJmf"
},
"source": [
"## Tutorial\n",
"\n",
"### Import required libraries"
]
},
@@ -543,7 +495,6 @@
" .load()\n",
")\n",
"\n",
"\n",
"# Load NYC_Citibike in Github Activity Public dataset from BQ.\n",
"bike_df = (\n",
" spark.read.format(\"bigquery\")\n",
@@ -663,7 +614,7 @@
"id": "XoEqT2Y4DJmf"
},
"source": [
"From this summary, you are able to know a lot of information.\n",
"From this summary, you're able to know a lot of information.\n",
"\n",
" * There are over 11 million trip histories for Green Taxi in 2017, which include some abnormal values, such as negative values.\n",
" * Until 2016, exact latitude and longitude were used for pickup and the dropoff locations. This data raised [privacy concerns](https://agkn.wordpress.com/2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/). `pickup_location_id` and `dropoff_location_id` in the dataset corresponded to the [NYC Taxi Zones](https://data.cityofnewyork.us/Transportation/NYC-Taxi-Zones/d3c5-ddgc), roughly based on NYC Department of City Planning’s Neighborhood Tabulation Areas (NTAs), and are meant to indicate approximate neighborhood locations.\n",
@@ -948,7 +899,7 @@
"source": [
"Assuming that Citi Bike users move through Manhattan's streets, which are the perpendicular, the trip distance can be calculated by applying [Manhattan distance](https://en.wikipedia.org/wiki/Taxicab_geometry).\n",
"\n",
"Manhattan distance calculation example: Assume you are at **W 15th St and 9th Ave, where Google NYC is located,** and you are going to **W 33rd St and 5th Ave, where the Empire State building is located**. You move east from 15th Street until you reach 5th Avenue, and then you travel north until you reach W 33rd St. In this case, **W 16th St and 5th Ave** or **W 33rd St and 9th Ave** can be a hinge point. If we set the starting point as **S**, the ending point as **E**, and the hinge point as **H**, the formula of Manhattan distance is `distance(S, H) + distance(H, E)`.\n",
"Manhattan distance calculation example: Assume you're at **W 15th St and 9th Ave, where Google NYC is located,** and you're going to **W 33rd St and 5th Ave, where the Empire State building is located**. You move east from 15th Street until you reach 5th Avenue, and then you travel north until you reach W 33rd St. In this case, **W 16th St and 5th Ave** or **W 33rd St and 9th Ave** can be a hinge point. If you set the starting point as **S**, the ending point as **E**, and the hinge point as **H**, the formula of Manhattan distance is `distance(S, H) + distance(H, E)`.\n",
"\n",
"However, additional calculation is needed to obtain the \"real\" Manhattan distance. First, unlike the Cartesian coordinate system, actual distance in longitude varies with latitude. For example, 1 degree of longitude represents roughly 69 miles at the equator, while it represents approximately 49 miles at 45 degrees north or south latitude. To calculate the exact distance in the real world using latitudes and longitudes, use [Haversine distance](https://en.wikipedia.org/wiki/Haversine_formula).\n",
"\n",
@@ -1293,7 +1244,7 @@
"### Feature Selection\n",
"\n",
"Not all features in our dataset will be useful. \n",
"Since the purpose of this tutorial is to compare two datasets, we need to use the same features for training. There are some useful features for training the Taxi dataset, such as `fare_amount`, but since the Citi Bike dataset does not have it, it will not be used in this tutorial.\n",
"Since the purpose of this tutorial is to compare two datasets, you need to use the same features for training. There are some useful features for training the Taxi dataset, such as `fare_amount`, but since the Citi Bike dataset does not have it, it won't be used in this tutorial.\n",
"\n",
"After choose the following columns as features, they must be assembled with `VectorAssembler()`, which is feature transformer that merges multiple columns into a vector column."
]
@@ -1506,11 +1457,9 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"DELETE_BUCKET = True\n",
"\n",
"if DELETE_BUCKET or os.getenv(\"IS_TESTING\"):\n",
"# Delete the Cloud Storage bucket\n",
"delete_bucket = False # Set True for deletion\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI"
]
},
@@ -1522,7 +1471,7 @@
"source": [
"### Delete Dataproc Cluster\n",
"\n",
"It is not possible to delete the cluster you are currently using unless you switch the kernel to local. To delete it, you need to switch the kernel to local `Python 3` or `PySpark`, set your `CLUSTER_NAME` and `CLUSTER_REGION` manually in the following cell, and execute the `gcloud` command.\n",
"It's not possible to delete the cluster you're currently using unless you switch the kernel to local. To delete it, you need to switch the kernel to local `Python 3` or `PySpark`, set your `CLUSTER_NAME` and `CLUSTER_LOCATION` manually in the following cell, and execute the `gcloud` command.\n",
"\n",
"See [Deleting a cluster](https://cloud.google.com/dataproc/docs/guides/manage-cluster#console) to delete the Dataproc cluster created in this tutorial."
]
@@ -1531,26 +1480,24 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5f6cae539ccc"
"id": "4bb6fda3082f"
},
"outputs": [],
"source": [
"CLUSTER_NAME = \"[your-cluster-name]\"\n",
"CLUSTER_REGION = \"[your-cluster-region]\""
"CLUSTER_LOCATION = \"[your-cluster-location]\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "61244eb74c17"
"id": "858cf6de26e4"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" ! gcloud dataproc clusters delete $CLUSTER_NAME --region=$CLUSTER_REGION -q"
" ! gcloud dataproc clusters delete $CLUSTER_NAME --region=$CLUSTER_LOCATION -q"
]
}
],