Vertex AI Experiments - Compare pipeline runs - Notebook review (#854)

* experiments cuj1 review

* linter test passed

* typo

* linter test passed

* add correct library. add IAM roles

* linter test passed

* minor fix

* linter test passed

* add api

* linter test passed

* remove library

* linter test passed

* fix

* linter passed

* check

* linter passed

* add text

* linter passed

* minor changes

* andy reviews

* linter passed

* fix link. fix warnings

* linter passed

Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>

notebooks/official/experiments/comparing_pipeline_runs.ipynb

@@ -55,7 +55,7 @@
         "    </a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/comparing_pipeline_runs.ipynb\">\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/experiments/comparing_pipeline_runs.ipynb\">\n",
         "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
         "      Open in Vertex AI Workbench\n",
         "    </a>\n",
@@ -70,6 +70,8 @@
         "id": "tvgnzT1CKxrO"
       },
       "source": [
+        "# Compare pipeline runs with Vertex AI Experiments\n",
+        "\n",
         "## Overview\n",
         "\n",
         "Depending on the model life cycle of your data science team, you would like to experiment and track training Pipeline runs and its associated parameters. Then, you would to compare runs of these Pipelines to each others in order to figure out which is the best configuration generates the model you will register in the Vertex AI Model Registry."
@@ -207,10 +209,8 @@
         "if IS_WORKBENCH_NOTEBOOK:\n",
         "    USER_FLAG = \"--user\"\n",
         "\n",
-        "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform \\\n",
+        "!pip3 install {USER_FLAG} --force-reinstall 'google-cloud-aiplatform>=1.15' -q --no-warn-conflicts\n",
-        "                                    google-cloud-storage \\\n",
+        "!pip3 install {USER_FLAG} kfp -q --no-warn-conflicts"
-        "                                    google-auth \\\n",
-        "                                    kfp -q"
       ]
     },
     {
@@ -270,7 +270,7 @@
         "\n",
         "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
         "\n",
-        "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
+        "1. [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=cloudresourcemanager.googleapis.com,aiplatform.googleapis.com).\n",
         "\n",
         "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
         "\n",
@@ -451,9 +451,14 @@
         "3. In the **Service account name** field, enter a name, and\n",
         "   click **Create**.\n",
         "\n",
-        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
+        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type and select\n",
-        "into the filter box, and select\n",
+        "the following role into the filter box:\n",
-        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "    *   Storage Admin\n",
+        "    *   Storage Object Admin\n",
+        "    *   Service Account User\n",
+        "    *   Vertex AI Administrator\n",
+        "\n",
         "\n",
         "5. Click *Create*. A JSON file that contains your key downloads to your\n",
         "local environment.\n",
@@ -720,6 +725,7 @@
         "import kfp.v2.dsl as dsl\n",
         "# Vertex AI\n",
         "from google.cloud import aiplatform as vertex_ai\n",
+        "from google.cloud.aiplatform_v1.types.pipeline_state import PipelineState\n",
         "from kfp.v2.dsl import Metrics, Model, Output, component"
       ]
     },
@@ -738,6 +744,7 @@
         "EXPERIMENT_NAME = f\"{PROJECT_ID}-{TASK}-{MODEL_TYPE}-{UUID}\"\n",
         "\n",
         "# Pipeline\n",
+        "PIPELINE_TEMPLATE_FILE = \"pipeline.json\"\n",
         "PIPELINE_URI = f\"{BUCKET_URI}/pipelines\"\n",
         "TRAIN_URI = f\"{BUCKET_URI}/iris/iris_data.csv\"\n",
         "LABEL_URI = f\"{BUCKET_URI}/iris/iris_target.csv\"\n",
@@ -819,9 +826,7 @@
       "source": [
         "Before you start running your pipeline experiments, you have to formalize your training as pipeline component.\n",
         "\n",
-        "To do that, you will use the `kfp.v2.dsl.component` decorator to convert your training task into a pipeline component.\n",
+        "To do that, you build the pipeline by using the `kfp.v2.dsl.component` decorator to convert your training task into a pipeline component. "
-        "\n",
-        "Training code will import required libraries to train,evaluate and save a model with mentioned features. "
       ]
     },
     {
@@ -1075,7 +1080,7 @@
         "\n",
         "    job = vertex_ai.PipelineJob(\n",
         "        display_name=f\"{EXPERIMENT_NAME}-pipeline-run-{i}\",\n",
-        "        template_path=\"pipeline.json\",\n",
+        "        template_path=PIPELINE_TEMPLATE_FILE,\n",
         "        pipeline_root=PIPELINE_URI,\n",
         "        parameter_values={\n",
         "            \"train_uri\": TRAIN_URI,\n",
@@ -1160,10 +1165,9 @@
       "source": [
         "# Get the PipelineJob resource using the experiment run name\n",
         "pipeline_experiments_df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)\n",
-        "for i in range(5):\n",
+        "job = vertex_ai.PipelineJob.get(pipeline_experiments_df.run_name[0])\n",
-        "    job = vertex_ai.PipelineJob.get(pipeline_experiments_df.run_name[i])\n",
+        "print(\"Pipeline job name: \", job.resource_name)\n",
-        "    print(job.resource_name)\n",
+        "print(\"Pipeline Run UI link: \", job._dashboard_uri())"
-        "    print(job._dashboard_uri())"
       ]
     },
     {
@@ -1191,13 +1195,16 @@
       "outputs": [],
       "source": [
         "# Delete the pipeline\n",
-        "# Get the PipelineJob resource using the experiment run name\n",
+        "while True:\n",
-        "pipeline_experiments_df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)\n",
+        "    for i in range(0, len(runs)):\n",
-        "for i in range(5):\n",
+        "        pipeline_job = vertex_ai.PipelineJob.get(pipeline_experiments_df.run_name[i])\n",
-        "    job = vertex_ai.PipelineJob.get(pipeline_experiments_df.run_name[i])\n",
+        "        if pipeline_job.state != PipelineState.PIPELINE_STATE_SUCCEEDED:\n",
-        "    print(job.resource_name)\n",
+        "            print(\"Pipeline job is still running...\")\n",
-        "    print(job._dashboard_uri())\n",
+        "            time.sleep(60)\n",
-        "    job.delete()\n",
+        "        else:\n",
+        "            print(\"Pipeline job is complete.\")\n",
+        "            pipeline_job.delete()\n",
+        "    break\n",
         "\n",
         "# Delete experiment\n",
         "exp = vertex_ai.Experiment(EXPERIMENT_NAME)\n",
@@ -1206,7 +1213,11 @@
         "# Delete bucket\n",
         "delete_bucket = False\n",
         "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
-        "    ! gsutil rm -rf {BUCKET_URI}"
+        "    ! gsutil rm -rf {BUCKET_URI}\n",
+        "\n",
+        "# Remove local files\n",
+        "\n",
+        "!rm {PIPELINE_TEMPLATE_FILE}"
       ]
     }
   ],