xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixes the build failure (#2195)

Browse Source 
* feat: boilerplate reduction 59

* fix: bucket

* spark runtime

* dataproc runtine -> 1.1.20

* updates the build steps + clean up steps

* ran linter test

* adds gcloud components update step for test env + adds needed services in before you begin section

* ran linter test

* moves the gcloud components update to the start + updates experiment fetching

* ran linter test

* updates the aiplatform sdk to the latest

* ran linter test

* removes gcloud components update and adds try except at get_dataframe() method

* ran linter test

* initializes experiment + adds gcloud update + removes try catch

* ran linter test

---------

Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Krishna Chaithanya Movva
2023-08-29 19:33:36 +00:00
committed by GitHub
co-authored by Andrew Ferlitsch
parent 66472e2642
commit bda6fb06d8
1 changed files with 128 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb
+128 -270
View File
@@ -53,6 +53,17 @@
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "962e636b5cee"
},
"source": [
"**_NOTE_**: This notebook has been tested in the following environment:\n",
"\n",
"* Python version = 3.9"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -128,61 +139,12 @@
{
"cell_type": "markdown",
"metadata": {
"id": "ze4-nDLfK4pw"
"id": "install_aip:mbsdk"
},
"source": [
"### Set up your local development environment\n",
"## Installation\n",
"\n",
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
"all the requirements to run this notebook. You can skip this step."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gCuSR8GkAgzl"
},
"source": [
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
"You need the following:\n",
"\n",
"* The Google Cloud SDK\n",
"* Git\n",
"* Python 3\n",
"* virtualenv\n",
"* Jupyter notebook running in a virtual environment with Python 3\n",
"\n",
"The Google Cloud guide to [Setting up a Python development\n",
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
"for meeting these requirements. The following steps provide a condensed set of\n",
"instructions:\n",
"\n",
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
"\n",
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
"\n",
"1. [Install\n",
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"\n",
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
"command-line in a terminal shell.\n",
"\n",
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"\n",
"1. Open this notebook in the Jupyter Notebook Dashboard."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7EUnXsZhAGF"
},
"source": [
"## Install additional packages\n",
"\n",
"Install additional package dependencies not installed in your notebook environment, such as Pyspark, MLeap and others. Use the latest major GA version of each package."
"Install the packages required for executing this notebook."
]
},
{
@@ -195,59 +157,38 @@
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"# (optional) update gcloud if needed\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! gcloud components update --quiet\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
" \n",
"! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform==1.18.1 \\\n",
" kfp==1.8.14 \\\n",
" google-cloud-pipeline-components==1.0.24 --quiet --no-warn-conflicts"
"\n",
"! pip3 install --upgrade --quiet google-cloud-aiplatform==1.30.1 \\\n",
" kfp==1.8.14 \\\n",
" google-cloud-pipeline-components==1.0.33 --no-warn-conflicts"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hhq5zEbGg0XX"
"id": "58707a750154"
},
"source": [
"### Restart the kernel\n",
"\n",
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
"### Colab only: Uncomment the following cell to restart the kernel."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EzrelQZ22IZj"
"id": "f200f10a1da3"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lWEdiXsJg0XY"
},
"source": [
"## Before you begin"
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
@@ -256,22 +197,19 @@
"id": "BF1j6f9HApxa"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI, Cloud Build, Cloud Storage, and Artifact Registry APIs.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleleapis.com,%20cloudbuild.googleapis.com,%20storage-api.googleapis.com,%20artifactregistry.googleapis.com&_ga=2.11105652.1331351212.1648374605-1663626427.1647020959). \n",
"3. [Enable the Artifact Registry, Cloud Build, Container Registry, Dataproc and Vertex AI APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,artifactregistry.googleapis.com,cloudbuild.googleapis.com,containerregistry.googleapis.com,dataproc.googleapis.com,aiplatform.googleapis.com).\n",
"\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
"\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
]
},
{
@@ -282,7 +220,10 @@
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
@@ -293,76 +234,32 @@
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "riG_qUokg0XZ"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_B-bdB17cQ_3"
},
"outputs": [],
"source": [
"! gcloud config set project $PROJECT_ID --quiet"
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6a5a0be04af6"
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
"- Asia Pacific: `asia-east1`\n",
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8e2a112253e5"
"id": "region"
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "66cc6bbc7e31"
},
"outputs": [],
"source": [
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
@@ -399,86 +296,59 @@
{
"cell_type": "markdown",
"metadata": {
"id": "dr--iN2kAylZ"
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
"authenticated. Skip this step."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sBCra4QMA2wR"
},
"source": [
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"2. Click **Create service account**.\n",
"\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type the following role and select them\n",
"\n",
" - Artifact Registry Administrator\n",
" - Artifact Registry Repository Administrator\n",
" - Cloud Build Editor\n",
" - Compute Network Admin\n",
" - Dataproc Administrator\n",
" - Dataproc Worker\n",
" - Service Account User\n",
" - Storage Admin\n",
" - Storage Object Admin\n",
" - Vertex AI Administrator\n",
"\n",
"\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-"
"id": "254614fa0c46"
},
"outputs": [],
"source": [
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"import os\n",
"import sys\n",
"\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ef21552ccea8"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "603adbbf0532"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6b2ccc891ed"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
@@ -489,7 +359,7 @@
"source": [
"### Enable Google Cloud services\n",
"\n",
"Enable the following services in your project:\n",
"Enable the following services in your project if not already done:\n",
"\n",
"* Artifact Registry\n",
"* Cloud Build\n",
@@ -522,12 +392,7 @@
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you submit a Spark job using the Cloud SDK, you need a bucket to read the python module associated and the data required to it. \n",
"\n",
"Set the name of your Cloud Storage bucket below. It must be unique across all\n",
"Cloud Storage buckets."
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
@@ -538,19 +403,7 @@
},
"outputs": [],
"source": [
"BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cf221059d072"
},
"outputs": [],
"source": [
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
" BUCKET_URI = \"gs://\" + PROJECT_ID + \"-aip-\" + UUID"
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -570,27 +423,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ucvCsknMCims"
},
"source": [
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vhOb7YnwClBb"
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -623,6 +456,11 @@
},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"\n",
"if (\n",
" SERVICE_ACCOUNT == \"\"\n",
" or SERVICE_ACCOUNT is None\n",
@@ -739,8 +577,10 @@
},
"outputs": [],
"source": [
"# set repo name\n",
"REPO_NAME = \"loan-eligibility-spark-demo\"\n",
"\n",
"# create the repository\n",
"!gcloud artifacts repositories create $REPO_NAME \\\n",
" --repository-format=docker \\\n",
" --location=$REGION \\\n",
@@ -759,7 +599,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"id": "pRUOFELefqf1"
},
@@ -784,7 +624,7 @@
"outputs": [],
"source": [
"# Setup\n",
"DATAPROC_RUNTIME_VERSION = \"1.1.3\"\n",
"DATAPROC_RUNTIME_VERSION = \"1.1.20\"\n",
"SRC = path(\"src\")\n",
"BUILD_PATH = path(\"build\")\n",
"DELIVERABLES = path(\"deliverables\")\n",
@@ -883,7 +723,12 @@
},
"outputs": [],
"source": [
"vertex_ai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
"vertex_ai.init(\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
" staging_bucket=BUCKET_URI,\n",
" experiment=EXPERIMENT_NAME,\n",
")"
]
},
{
@@ -1956,10 +1801,9 @@
"# The following packages are installed in the default image, it is strongly\n",
"# recommended to include all of them.\n",
"#\n",
"# Use mamba to install packages quickly.\n",
"RUN ${CONDA_HOME}/bin/conda install mamba -n base -c conda-forge \\\n",
" && ${CONDA_HOME}/bin/mamba install \\\n",
" conda \\\n",
"# Use mamba solver to install packages quickly.\n",
"RUN ${CONDA_HOME}/bin/conda install -n base conda-libmamba-solver\n",
"RUN ${CONDA_HOME}/bin/conda install \\\n",
" cython \\\n",
" fastavro \\\n",
" fastparquet \\\n",
@@ -1973,7 +1817,7 @@
" scikit-image \\\n",
" scikit-learn \\\n",
" scipy \\\n",
" mleap\n",
" mleap --solver=libmamba\n",
"\n",
"# (Required) Create the 'spark' group/user.\n",
"# The GID and UID must be 1099. Home directory is required.\n",
@@ -2011,7 +1855,7 @@
"source": [
"#### Build the Dataproc Serverless custom runtime using Cloud Build\n",
"\n",
"**Note:** this step may take approximately 5 to 10 minutes to complete."
"**Note:** this step may take approximately upto 20 minutes to complete."
]
},
{
@@ -2022,7 +1866,7 @@
},
"outputs": [],
"source": [
"!gcloud builds submit --tag $RUNTIME_CONTAINER_IMAGE $BUILD_PATH --machine-type=N1_HIGHCPU_32 --timeout=900s --verbosity=info"
"!gcloud builds submit --tag $RUNTIME_CONTAINER_IMAGE $BUILD_PATH --machine-type=N1_HIGHCPU_32 --timeout=3600s --verbosity=info"
]
},
{
@@ -2117,7 +1961,6 @@
" metrics: Output[Metrics],\n",
" plots: Output[ClassificationMetrics],\n",
") -> NamedTuple(\"Outputs\", [(\"threshold_metric\", float)]):\n",
"\n",
" # Libraries --------------------------------------------------------------------------------------------------------------------------\n",
" import json\n",
"\n",
@@ -2516,7 +2359,6 @@
" evaluate_model_op.outputs[\"threshold_metric\"] >= threshold,\n",
" name=AUPR_HYPERTUNE_CONDITION,\n",
" ):\n",
"\n",
" build_hpt_args_op = build_hpt_args(\n",
" train_path=train_path,\n",
" model_path=hpt_model_path,\n",
@@ -2668,7 +2510,11 @@
},
"outputs": [],
"source": [
"experiment_df = vertex_ai.get_experiment_df(experiment=EXPERIMENT_NAME)\n",
"# get the experiment by name\n",
"experiment = vertex_ai.Experiment(experiment_name=EXPERIMENT_NAME)\n",
"\n",
"# export the data as a dataframe\n",
"experiment_df = experiment.get_data_frame()\n",
"\n",
"# Show successfully completed experiment runs, sorted by F1 score\n",
"experiment_df.query('state == \"COMPLETE\"').sort_values(\n",
@@ -2773,7 +2619,15 @@
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial with the following code."
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
"- Vertex AI Pipeline\n",
"- Vertex AI Endpoint\n",
"- Vertex AI Model\n",
"- Vertex AI Experiment\n",
"- Artifact Repository\n",
"- Cloud Storage bucket\n",
"- Local src, build and cloned repo folders"
]
},
{
@@ -2796,7 +2650,10 @@
"# Delete model\n",
"model_list = vertex_ai.Model.list(filter=f'display_name=\"{MODEL_NAME}\"')\n",
"for model in model_list:\n",
" model.delete()"
" model.delete()\n",
"\n",
"# Delete experiment\n",
"experiment.delete()"
]
},
{
@@ -2842,7 +2699,8 @@
},
"outputs": [],
"source": [
"!rm -rf $SRC $BUILD_PATH"
"# remove the local src, build and repo folders\n",
"!rm -rf $SRC $BUILD_PATH cloud-builders-community vertex-ai-spark-ml-serving"
]
}
],