Merge branch 'ml_ops_6v3' of https://github.com/GoogleCloudPlatform/vertex-ai-samples into ml_ops_6v3

fix: v1 upgrades
2026-09-26 14:42:04 +00:00 · 2022-03-03 23:56:46 +00:00 · 2022-03-03 23:56:27 +00:00 · 2022-03-03 23:54:22 +00:00 · 2022-03-03 10:20:29 -06:00 · 2022-03-02 12:25:03 -08:00
2 changed files with 112 additions and 14 deletions
@@ -8,7 +8,7 @@
      },
      "outputs": [],
      "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
@@ -78,6 +78,7 @@
        "- Building KFP lightweight Python function components.\n",
        "- Assembling and compiling KFP components into a pipeline.\n",
        "- Executing a KFP pipeline using Vertex AI Pipelines.\n",
+        "- Loading component and pipeline definitions from a source code repository.\n",
        "- Building sequential, parallel, multiple output components.\n",
        "- Building control flow into pipelines."
      ]
@@ -114,7 +115,11 @@
        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
-        "    ! pip3 install --upgrade kfp $USER_FLAG"
+        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
+        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
+        "    ! pip3 install --upgrade rpy2 $USER_FLAG\n",
+        "    ! pip3 install --upgrade python-tabulate $USER_FLAG\n",
+        "    ! pip3 install -U opencv-python-headless==4.5.2.52 $USER_FLAG"
      ]
    },
    {
@@ -364,7 +369,7 @@
        "):\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud auth list 2>/dev/null\n",
-        "    SERVICE_ACCOUNT = shell_output[2].strip()\n",
+        "    SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
@@ -694,14 +699,29 @@
        "            + str(TASK_ID)\n",
        "            + \"/gcp_resources\"\n",
        "        )\n",
+        "        EVAL_METRICS = (\n",
+        "            PIPELINE_ROOT\n",
+        "            + \"/\"\n",
+        "            + PROJECT_NUMBER\n",
+        "            + \"/\"\n",
+        "            + JOB_ID\n",
+        "            + \"/\"\n",
+        "            + output_task_name\n",
+        "            + \"_\"\n",
+        "            + str(TASK_ID)\n",
+        "            + \"/evaluation_metrics\"\n",
+        "        )\n",
        "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
        "            ! gsutil cat $EXECUTE_OUTPUT\n",
-        "            break\n",
+        "            return EXECUTE_OUTPUT\n",
        "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
        "            ! gsutil cat $GCP_RESOURCES\n",
-        "            break\n",
+        "            return GCP_RESOURCES\n",
+        "        elif tf.io.gfile.exists(EVAL_METRICS):\n",
+        "            ! gsutil cat $EVAL_METRICS\n",
+        "            return EVAL_METRICS\n",
        "\n",
-        "    return EXECUTE_OUTPUT\n",
+        "    return None\n",
        "\n",
        "\n",
        "print_pipeline_output(pipeline, \"hello-world\")"
@@ -803,6 +823,73 @@
        "pipeline.delete()"
      ]
    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "load_component_pipeline_git:helloworld"
+      },
+      "source": [
+        "### Loading components and pipeline YAML definitions from source control\n",
+        "\n",
+        "By storing the component and pipeline definitions in a source repository, like Github, you can version control your components and pipelines, as follows:\n",
+        "\n",
+        "- Use the method `load_component_from_url()`.\n",
+        "\n",
+        "- Pull the raw file format version from the repo. For github, that will be in the form of:\n",
+        "\n",
+        "    https://raw.githubusercontent.com/\n",
+        "\n",
+        "- Specify the version of the component/pipeline. For github, that will be the branch."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "load_component_pipeline_git:helloworld"
+      },
+      "outputs": [],
+      "source": [
+        "VERSION = \"main\"\n",
+        "hello_world_op = components.load_component_from_url(\n",
+        "    f\"https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.yaml\"\n",
+        ")\n",
+        "\n",
+        "! wget https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/{VERSION}/notebooks/community/ml_ops/stage3/src/hello_world.json -O hello_git_example.json\n",
+        "\n",
+        "pipeline = aip.PipelineJob(\n",
+        "    display_name=\"hello_world-git\",\n",
+        "    template_path=\"hello_git_example.json\",\n",
+        "    pipeline_root=PIPELINE_ROOT,\n",
+        ")\n",
+        "\n",
+        "pipeline.run()\n",
+        "\n",
+        "! rm -f hello_git_example.json"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "source": [
+        "### Delete a pipeline job\n",
+        "\n",
+        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "delete_pipeline"
+      },
+      "outputs": [],
+      "source": [
+        "pipeline.delete()"
+      ]
+    },
    {
      "cell_type": "markdown",
      "metadata": {
@@ -750,14 +750,17 @@
        "    model_display_name: str = MODEL_DISPLAY_NAME,\n",
        "    serving_container_image_uri: str = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
        "):\n",
+        "    from google_cloud_pipeline_components.types import artifact_types\n",
+        "    from google_cloud_pipeline_components.v1.custom_job import \\\n",
+        "        CustomTrainingJobOp\n",
        "    from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
        "                                                              ModelDeployOp)\n",
        "    from google_cloud_pipeline_components.v1.model import ModelUploadOp\n",
-        "    from kfp.v2.google import experimental\n",
+        "    from kfp.v2.components import importer_node\n",
        "\n",
-        "    train_task = print_op(\"model training\")\n",
-        "    experimental.run_as_aiplatform_custom_job(\n",
-        "        train_task,\n",
+        "    custom_job_task = CustomTrainingJobOp(\n",
+        "        project=project,\n",
+        "        display_name=\"model-training\",\n",
        "        worker_pool_specs=[\n",
        "            {\n",
        "                \"containerSpec\": {\n",
@@ -775,14 +778,22 @@
        "        ],\n",
        "    )\n",
        "\n",
+        "    import_unmanaged_model_task = importer_node.importer(\n",
+        "        artifact_uri=WORKING_DIR,\n",
+        "        artifact_class=artifact_types.UnmanagedContainerModel,\n",
+        "        metadata={\n",
+        "            \"containerSpec\": {\n",
+        "                \"imageUri\": \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-3:latest\",\n",
+        "            },\n",
+        "        },\n",
+        "    ).after(custom_job_task)\n",
+        "\n",
        "    model_upload_op = ModelUploadOp(\n",
        "        project=project,\n",
        "        display_name=model_display_name,\n",
-        "        artifact_uri=WORKING_DIR,\n",
-        "        serving_container_image_uri=serving_container_image_uri,\n",
-        "        # serving_container_environment_variables={\"NOT_USED\": \"NO_VALUE\"},\n",
+        "        unmanaged_container_model=import_unmanaged_model_task.outputs[\"artifact\"],\n",
        "    )\n",
-        "    model_upload_op.after(train_task)\n",
+        "    model_upload_op.after(import_unmanaged_model_task)\n",
        "\n",
        "    endpoint_create_op = EndpointCreateOp(\n",
        "        project=project,\n",
Author	SHA1	Message	Date
Andrew Ferlitsch	d62b944f08	Merge branch 'ml_ops_6v3' of https://github.com/GoogleCloudPlatform/vertex-ai-samples into ml_ops_6v3	2022-03-03 23:56:46 +00:00
Andrew Ferlitsch	2325808fbf	fix: v1 upgrades	2022-03-03 23:56:27 +00:00
Andrew Ferlitsch	a7f088adc2	fix: v1 upgrades	2022-03-03 23:54:22 +00:00
Karl WeinmeisterandGitHub	5c54efb2d6	Merge branch 'main' into ml_ops_6v3	2022-03-03 10:20:29 -06:00
Andrew FerlitschandGitHub	6a94cf3fff	Merge branch 'main' into ml_ops_6v3	2022-03-02 12:25:03 -08:00
Andrew Ferlitsch	6cb168eddc	Merge branch 'ml_ops_6v3' of https://github.com/GoogleCloudPlatform/vertex-ai-samples into ml_ops_6v3	2022-03-02 20:23:52 +00:00
Andrew Ferlitsch	90d222ad2d	feat: loading from src control	2022-03-02 20:22:55 +00:00
Andrew Ferlitsch	989b59c21e	feat: loading from src control	2022-03-02 19:59:12 +00:00
Andrew FerlitschandGitHub	b11c5ca7c8	Merge branch 'main' into ml_ops_6v3	2022-03-02 10:38:32 -08:00
Andrew Ferlitsch	866ea1c4a2	feat: add to KFP get started	2022-03-02 18:37:05 +00:00
Andrew Ferlitsch	0bf9a924ed	fix: eval comp improvement	2022-02-25 20:05:16 +00:00
Andrew Ferlitsch	76c74ad4db	fix: eval comp improvement	2022-02-25 20:04:42 +00:00