diff --git a/notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb b/notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb index 0fa812806..c79a08c91 100644 --- a/notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb +++ b/notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb @@ -53,6 +53,17 @@ "" ] }, + { + "cell_type": "markdown", + "metadata": { + "id": "962e636b5cee" + }, + "source": [ + "**_NOTE_**: This notebook has been tested in the following environment:\n", + "\n", + "* Python version = 3.9" + ] + }, { "cell_type": "markdown", "metadata": { @@ -128,61 +139,12 @@ { "cell_type": "markdown", "metadata": { - "id": "ze4-nDLfK4pw" + "id": "install_aip:mbsdk" }, "source": [ - "### Set up your local development environment\n", + "## Installation\n", "\n", - "**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n", - "all the requirements to run this notebook. You can skip this step." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gCuSR8GkAgzl" - }, - "source": [ - "**Otherwise**, make sure your environment meets this notebook's requirements.\n", - "You need the following:\n", - "\n", - "* The Google Cloud SDK\n", - "* Git\n", - "* Python 3\n", - "* virtualenv\n", - "* Jupyter notebook running in a virtual environment with Python 3\n", - "\n", - "The Google Cloud guide to [Setting up a Python development\n", - "environment](https://cloud.google.com/python/setup) and the [Jupyter\n", - "installation guide](https://jupyter.org/install) provide detailed instructions\n", - "for meeting these requirements. The following steps provide a condensed set of\n", - "instructions:\n", - "\n", - "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n", - "\n", - "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n", - "\n", - "1. [Install\n", - " virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n", - " and create a virtual environment that uses Python 3. Activate the virtual environment.\n", - "\n", - "1. To install Jupyter, run `pip3 install jupyter` on the\n", - "command-line in a terminal shell.\n", - "\n", - "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", - "\n", - "1. Open this notebook in the Jupyter Notebook Dashboard." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "i7EUnXsZhAGF" - }, - "source": [ - "## Install additional packages\n", - "\n", - "Install additional package dependencies not installed in your notebook environment, such as Pyspark, MLeap and others. Use the latest major GA version of each package." + "Install the packages required for executing this notebook." ] }, { @@ -195,59 +157,38 @@ "source": [ "import os\n", "\n", - "# The Vertex AI Workbench Notebook product has specific requirements\n", - "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n", - "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n", - " \"/opt/deeplearning/metadata/env_version\"\n", - ")\n", + "# (optional) update gcloud if needed\n", + "if os.getenv(\"IS_TESTING\"):\n", + " ! gcloud components update --quiet\n", "\n", - "# Vertex AI Notebook requires dependencies to be installed with '--user'\n", - "USER_FLAG = \"\"\n", - "if IS_WORKBENCH_NOTEBOOK:\n", - " USER_FLAG = \"--user\"\n", - " \n", - "! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform==1.18.1 \\\n", - " kfp==1.8.14 \\\n", - " google-cloud-pipeline-components==1.0.24 --quiet --no-warn-conflicts" + "\n", + "! pip3 install --upgrade --quiet google-cloud-aiplatform==1.30.1 \\\n", + " kfp==1.8.14 \\\n", + " google-cloud-pipeline-components==1.0.33 --no-warn-conflicts" ] }, { "cell_type": "markdown", "metadata": { - "id": "hhq5zEbGg0XX" + "id": "58707a750154" }, "source": [ - "### Restart the kernel\n", - "\n", - "After you install the additional packages, you need to restart the notebook kernel so it can find the packages." + "### Colab only: Uncomment the following cell to restart the kernel." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "EzrelQZ22IZj" + "id": "f200f10a1da3" }, "outputs": [], "source": [ - "# Automatically restart kernel after installs\n", - "import os\n", + "# Automatically restart kernel after installs so that your environment can access the new packages\n", + "# import IPython\n", "\n", - "if not os.getenv(\"IS_TESTING\"):\n", - " # Automatically restart kernel after installs\n", - " import IPython\n", - "\n", - " app = IPython.Application.instance()\n", - " app.kernel.do_shutdown(True)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "lWEdiXsJg0XY" - }, - "source": [ - "## Before you begin" + "# app = IPython.Application.instance()\n", + "# app.kernel.do_shutdown(True)" ] }, { @@ -256,22 +197,19 @@ "id": "BF1j6f9HApxa" }, "source": [ + "## Before you begin\n", + "\n", "### Set up your Google Cloud project\n", "\n", "**The following steps are required, regardless of your notebook environment.**\n", "\n", "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", "\n", - "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", "\n", - "1. [Enable the Vertex AI, Cloud Build, Cloud Storage, and Artifact Registry APIs.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleleapis.com,%20cloudbuild.googleapis.com,%20storage-api.googleapis.com,%20artifactregistry.googleapis.com&_ga=2.11105652.1331351212.1648374605-1663626427.1647020959). \n", + "3. [Enable the Artifact Registry, Cloud Build, Container Registry, Dataproc and Vertex AI APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,artifactregistry.googleapis.com,cloudbuild.googleapis.com,containerregistry.googleapis.com,dataproc.googleapis.com,aiplatform.googleapis.com).\n", "\n", - "1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", - "\n", - "1. Enter your project ID in the cell below. Then run the cell to make sure the\n", - "Cloud SDK uses the right project for all the commands in this notebook.\n", - "\n", - "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands." + "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)." ] }, { @@ -282,7 +220,10 @@ "source": [ "#### Set your project ID\n", "\n", - "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`." + "**If you don't know your project ID**, try the following:\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" ] }, { @@ -293,76 +234,32 @@ }, "outputs": [], "source": [ - "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "riG_qUokg0XZ" - }, - "outputs": [], - "source": [ - "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", - " # Get your GCP project id from gcloud\n", - " shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n", - " PROJECT_ID = shell_output[0]\n", - " print(\"Project ID:\", PROJECT_ID)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_B-bdB17cQ_3" - }, - "outputs": [], - "source": [ - "! gcloud config set project $PROJECT_ID --quiet" + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "\n", + "# Set the project id\n", + "! gcloud config set project {PROJECT_ID}" ] }, { "cell_type": "markdown", "metadata": { - "id": "6a5a0be04af6" + "id": "region" }, "source": [ "#### Region\n", "\n", - "You can also change the `REGION` variable, which is used for operations\n", - "throughout the rest of this notebook. Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.\n", - "\n", - "- Americas: `us-central1`\n", - "- Europe: `europe-west4`\n", - "- Asia Pacific: `asia-east1`\n", - "\n", - "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n", - "\n", - "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." + "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "8e2a112253e5" + "id": "region" }, "outputs": [], "source": [ - "REGION = \"[your-region]\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "66cc6bbc7e31" - }, - "outputs": [], - "source": [ - "if REGION == \"[your-region]\":\n", - " REGION = \"us-central1\"" + "REGION = \"us-central1\" # @param {type: \"string\"}" ] }, { @@ -399,86 +296,59 @@ { "cell_type": "markdown", "metadata": { - "id": "dr--iN2kAylZ" + "id": "sBCra4QMA2wR" }, "source": [ "### Authenticate your Google Cloud account\n", "\n", - "**If you are using Vertex AI Workbench Notebooks**, your environment is already\n", - "authenticated. Skip this step." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sBCra4QMA2wR" - }, - "source": [ - "**If you are using Colab**, run the cell below and follow the instructions\n", - "when prompted to authenticate your account via oAuth.\n", + "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n", "\n", - "**Otherwise**, follow these steps:\n", + "**1. Vertex AI Workbench**\n", + "* Do nothing as you are already authenticated.\n", "\n", - "1. In the Cloud Console, go to the [**Create service account key**\n", - " page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n", - "\n", - "2. Click **Create service account**.\n", - "\n", - "3. In the **Service account name** field, enter a name, and\n", - " click **Create**.\n", - "\n", - "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type the following role and select them\n", - "\n", - " - Artifact Registry Administrator\n", - " - Artifact Registry Repository Administrator\n", - " - Cloud Build Editor\n", - " - Compute Network Admin\n", - " - Dataproc Administrator\n", - " - Dataproc Worker\n", - " - Service Account User\n", - " - Storage Admin\n", - " - Storage Object Admin\n", - " - Vertex AI Administrator\n", - "\n", - "\n", - "5. Click *Create*. A JSON file that contains your key downloads to your\n", - "local environment.\n", - "\n", - "6. Enter the path to your service account key as the\n", - "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + "**2. Local JupyterLab instance, uncomment and run:**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "PyQmSRbKA8r-" + "id": "254614fa0c46" }, "outputs": [], "source": [ - "# If you are running this notebook in Colab, run this cell and follow the\n", - "# instructions to authenticate your GCP account. This provides access to your\n", - "# Cloud Storage bucket and lets you submit training jobs and prediction\n", - "# requests.\n", - "\n", - "import os\n", - "import sys\n", - "\n", - "# If on Vertex AI Workbench, then don't execute this code\n", - "IS_COLAB = \"google.colab\" in sys.modules\n", - "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n", - " \"DL_ANACONDA_HOME\"\n", - "):\n", - " if \"google.colab\" in sys.modules:\n", - " from google.colab import auth as google_auth\n", - "\n", - " google_auth.authenticate_user()\n", - "\n", - " # If you are running this notebook locally, replace the string below with the\n", - " # path to your service account key and run this cell to authenticate your GCP\n", - " # account.\n", - " elif not os.getenv(\"IS_TESTING\"):\n", - " %env GOOGLE_APPLICATION_CREDENTIALS ''" + "# ! gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ef21552ccea8" + }, + "source": [ + "**3. Colab, uncomment and run:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "603adbbf0532" + }, + "outputs": [], + "source": [ + "# from google.colab import auth\n", + "# auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f6b2ccc891ed" + }, + "source": [ + "**4. Service account or other**\n", + "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples." ] }, { @@ -489,7 +359,7 @@ "source": [ "### Enable Google Cloud services\n", "\n", - "Enable the following services in your project:\n", + "Enable the following services in your project if not already done:\n", "\n", "* Artifact Registry\n", "* Cloud Build\n", @@ -522,12 +392,7 @@ "source": [ "### Create a Cloud Storage bucket\n", "\n", - "**The following steps are required, regardless of your notebook environment.**\n", - "\n", - "When you submit a Spark job using the Cloud SDK, you need a bucket to read the python module associated and the data required to it. \n", - "\n", - "Set the name of your Cloud Storage bucket below. It must be unique across all\n", - "Cloud Storage buckets." + "Create a storage bucket to store intermediate artifacts such as datasets." ] }, { @@ -538,19 +403,7 @@ }, "outputs": [], "source": [ - "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "cf221059d072" - }, - "outputs": [], - "source": [ - "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", - " BUCKET_URI = \"gs://\" + PROJECT_ID + \"-aip-\" + UUID" + "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}" ] }, { @@ -570,27 +423,7 @@ }, "outputs": [], "source": [ - "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ucvCsknMCims" - }, - "source": [ - "Finally, validate access to your Cloud Storage bucket by examining its contents:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "vhOb7YnwClBb" - }, - "outputs": [], - "source": [ - "! gsutil ls -al $BUCKET_URI" + "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}" ] }, { @@ -623,6 +456,11 @@ }, "outputs": [], "source": [ + "import os\n", + "import sys\n", + "\n", + "IS_COLAB = \"google.colab\" in sys.modules\n", + "\n", "if (\n", " SERVICE_ACCOUNT == \"\"\n", " or SERVICE_ACCOUNT is None\n", @@ -739,8 +577,10 @@ }, "outputs": [], "source": [ + "# set repo name\n", "REPO_NAME = \"loan-eligibility-spark-demo\"\n", "\n", + "# create the repository\n", "!gcloud artifacts repositories create $REPO_NAME \\\n", " --repository-format=docker \\\n", " --location=$REGION \\\n", @@ -759,7 +599,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "id": "pRUOFELefqf1" }, @@ -784,7 +624,7 @@ "outputs": [], "source": [ "# Setup\n", - "DATAPROC_RUNTIME_VERSION = \"1.1.3\"\n", + "DATAPROC_RUNTIME_VERSION = \"1.1.20\"\n", "SRC = path(\"src\")\n", "BUILD_PATH = path(\"build\")\n", "DELIVERABLES = path(\"deliverables\")\n", @@ -883,7 +723,12 @@ }, "outputs": [], "source": [ - "vertex_ai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)" + "vertex_ai.init(\n", + " project=PROJECT_ID,\n", + " location=REGION,\n", + " staging_bucket=BUCKET_URI,\n", + " experiment=EXPERIMENT_NAME,\n", + ")" ] }, { @@ -1956,10 +1801,9 @@ "# The following packages are installed in the default image, it is strongly\n", "# recommended to include all of them.\n", "#\n", - "# Use mamba to install packages quickly.\n", - "RUN ${CONDA_HOME}/bin/conda install mamba -n base -c conda-forge \\\n", - " && ${CONDA_HOME}/bin/mamba install \\\n", - " conda \\\n", + "# Use mamba solver to install packages quickly.\n", + "RUN ${CONDA_HOME}/bin/conda install -n base conda-libmamba-solver\n", + "RUN ${CONDA_HOME}/bin/conda install \\\n", " cython \\\n", " fastavro \\\n", " fastparquet \\\n", @@ -1973,7 +1817,7 @@ " scikit-image \\\n", " scikit-learn \\\n", " scipy \\\n", - " mleap\n", + " mleap --solver=libmamba\n", "\n", "# (Required) Create the 'spark' group/user.\n", "# The GID and UID must be 1099. Home directory is required.\n", @@ -2011,7 +1855,7 @@ "source": [ "#### Build the Dataproc Serverless custom runtime using Cloud Build\n", "\n", - "**Note:** this step may take approximately 5 to 10 minutes to complete." + "**Note:** this step may take approximately upto 20 minutes to complete." ] }, { @@ -2022,7 +1866,7 @@ }, "outputs": [], "source": [ - "!gcloud builds submit --tag $RUNTIME_CONTAINER_IMAGE $BUILD_PATH --machine-type=N1_HIGHCPU_32 --timeout=900s --verbosity=info" + "!gcloud builds submit --tag $RUNTIME_CONTAINER_IMAGE $BUILD_PATH --machine-type=N1_HIGHCPU_32 --timeout=3600s --verbosity=info" ] }, { @@ -2117,7 +1961,6 @@ " metrics: Output[Metrics],\n", " plots: Output[ClassificationMetrics],\n", ") -> NamedTuple(\"Outputs\", [(\"threshold_metric\", float)]):\n", - "\n", " # Libraries --------------------------------------------------------------------------------------------------------------------------\n", " import json\n", "\n", @@ -2516,7 +2359,6 @@ " evaluate_model_op.outputs[\"threshold_metric\"] >= threshold,\n", " name=AUPR_HYPERTUNE_CONDITION,\n", " ):\n", - "\n", " build_hpt_args_op = build_hpt_args(\n", " train_path=train_path,\n", " model_path=hpt_model_path,\n", @@ -2668,7 +2510,11 @@ }, "outputs": [], "source": [ - "experiment_df = vertex_ai.get_experiment_df(experiment=EXPERIMENT_NAME)\n", + "# get the experiment by name\n", + "experiment = vertex_ai.Experiment(experiment_name=EXPERIMENT_NAME)\n", + "\n", + "# export the data as a dataframe\n", + "experiment_df = experiment.get_data_frame()\n", "\n", "# Show successfully completed experiment runs, sorted by F1 score\n", "experiment_df.query('state == \"COMPLETE\"').sort_values(\n", @@ -2773,7 +2619,15 @@ "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", "\n", - "Otherwise, you can delete the individual resources you created in this tutorial with the following code." + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Vertex AI Pipeline\n", + "- Vertex AI Endpoint\n", + "- Vertex AI Model\n", + "- Vertex AI Experiment\n", + "- Artifact Repository\n", + "- Cloud Storage bucket\n", + "- Local src, build and cloned repo folders" ] }, { @@ -2796,7 +2650,10 @@ "# Delete model\n", "model_list = vertex_ai.Model.list(filter=f'display_name=\"{MODEL_NAME}\"')\n", "for model in model_list:\n", - " model.delete()" + " model.delete()\n", + "\n", + "# Delete experiment\n", + "experiment.delete()" ] }, { @@ -2842,7 +2699,8 @@ }, "outputs": [], "source": [ - "!rm -rf $SRC $BUILD_PATH" + "# remove the local src, build and repo folders\n", + "!rm -rf $SRC $BUILD_PATH cloud-builders-community vertex-ai-spark-ml-serving" ] } ],