test: debugging

Train tabular models with many frameworks and import to Vertex AI using Pipelines

.github/workflows/linter/requirements.txt

@@ -2,9 +2,9 @@ git+https://github.com/tensorflow/docs
 ipython
 jupyter
 nbconvert
-black==22.10.0
+black==22.12.0
 pyupgrade==2.38.4
-isort==5.10.1
+isort==5.12.0
 flake8==6.0.0
 nbqa==1.5.3
 

community-content/pytorch_image_classification_single_gpu_with_vertex_sdk_and_torchserve/trainer/requirements.txt

@@ -1,3 +1,3 @@
-torch==1.8.1
+torch==1.13.1
 torchvision==0.9.1
 tensorboard==2.5.0

community-content/pytorch_pre_built_images_deployment/pytorch-text-classification-vertex-ai-deploy.ipynb

@@ -31,17 +31,7 @@
       "source": [
         "# Deploying a PyTorch Text Classification Model on [Vertex AI](https://cloud.google.com/vertex-ai)\n",
         "\n",
-        "**This is an Experimental release**, covered by the Pre-GA Offerings Terms of your Google Cloud Platform [Terms of Service](https://cloud.google.com/terms).\n",
-        "\n",
-        "Experiments are focused on validating a prototype and are not guaranteed to be released. They are not intended for production use or covered by any SLA, support obligation, or deprecation policy and might be subject to backward-incompatible changes.\n",
-        "\n",
-        "**Kindly drop us a note before you run any scale tests.**\n",
-        "\n",
-        "**Do not hesitate to contact vertexai-prediction-preview-feedback@google.com if you have any questions or run into any issues.**\n",
-        "\n",
-        "The usage of the product is free during the Experimental release period: you will still incur charges for other GCP products usage, such as storage.\n",
-        "\n",
-        "The projects need to be added to the allowlist in order to deploy PyTorch models using Vertex AI Prediction pre-built PyTorch images. If you are interested in the feature, please send an email to vertexai-prediction-preview-feedback@google.com to provide your project numbers OR project ids."
+        "**Kindly reach out to Vertex AI before you run any scale tests or you have any questions.**\n"
       ]
     },
     {

notebooks/community/ml_ops/stage3/get_started_with_hpt_pipeline_components.ipynb

@@ -1333,7 +1333,6 @@
         "Next, you compile the pipeline and then exeute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:\n",
         "\n",
         "- `display_name`: A human readable name for the pipeline job.\n",
-        "- `import_file`: The Cloud Storage location to the dataset.\n",
         "- `worker_pool_specs`: The the machine and container, and auto-scaling requirements, as well as command line arguments.\n",
         "- `study_spec_metrics`: The metrics to optimize in the study trials.\n",
         "- `study_spec_parameters`: The parameters to tune."

notebooks/community/vertex_endpoints/torchserve/dreambooth_stablediffusion.ipynb

@@ -90,7 +90,7 @@
         "\n",
         "+ Create a `torchserve` handler for responding to prediction requests.\n",
         "+ Upload a Stable Diffusion 1.5 model on a prebuilt PyTorch container in Vertex AI.\n",
-        "+ Deploy a custom model to a Vertex AI Endpoint.\n",
+        "+ Deploy a model to a Vertex AI Endpoint.\n",
         "+ Send requests to the endpoint and parse the responses using Vertex AI Prediction service."
       ]
     },
@@ -100,7 +100,7 @@
         "id": "c6deba5a8557"
       },
       "source": [
-        "### Dataset\n",
+        "### Model\n",
         "\n",
         "This notebook uses a collection of model artifacts fine-tuned to generate images of a small dog. These are the same images used in the original [DreamBooth paper](https://dreambooth.github.io/)."
       ]
@@ -118,7 +118,7 @@
         "* Vertex AI models\n",
         "* Vertex AI endpoints\n",
         "* Vertex AI prediction\n",
-        "* Google Cloud Storage\n",
+        "* Cloud Storage\n",
         "* (Optionally) Vertex AI Workbench\n",
         "\n",
         "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
@@ -420,7 +420,6 @@
       "outputs": [],
       "source": [
         "import base64\n",
-        "import gc\n",
         "import math\n",
         "\n",
         "import torch\n",
@@ -437,9 +436,9 @@
         "id": "697566b5f660"
       },
       "source": [
-        "## View model inferences\n",
+        "## Optional: View model inferences\n",
         "\n",
-        "Before uploading the model to Vertex AI, you can review the expected output from the model. The model used in this notebook is available for your use and can be downloaded from Google Cloud Storage. This download may take a few minutes to complete."
+        "Before uploading the model to Vertex AI, you can review the expected output from the model. The model used in this notebook is available for your use and can be downloaded from Cloud Storage. This download may take a few minutes to complete."
       ]
     },
     {
@@ -574,31 +573,6 @@
         "grid"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "cf8d545a135b"
-      },
-      "source": [
-        "### Re-training the model\n",
-        "\n",
-        "If you aren't satisfied with the inference outputs from the model, you can retrain it. Before you can do that, you might need to release the GPU memory."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "5621b7a927f8"
-      },
-      "outputs": [],
-      "source": [
-        "# Run this cell _only_ if you want to release the GPU memory.\n",
-        "pipe = None\n",
-        "gc.collect()\n",
-        "torch.cuda.empty_cache()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -607,10 +581,10 @@
       "source": [
         "## Deploy the model to Vertex AI\n",
         "\n",
-        "You can host your Stable Diffusion 2.0 model on a Vertex AI endpoint where you can get inferences from it online. Uploading your model is a four step process: \n",
+        "You can host your Stable Diffusion 1.5 model on a Vertex AI endpoint where you can get inferences from it online. Uploading your model is a four step process: \n",
         "\n",
         "1. Create a custom TorchServe handler.\n",
-        "1. Upload the model artifacts onto Google Cloud Storage.\n",
+        "1. Upload the model artifacts onto Cloud Storage.\n",
         "2. Create a Vertex AI model with the model artifacts and a prebuilt PyTorch container image.\n",
         "3. Deploy the Vertex AI model onto an endpoint."
       ]
@@ -740,9 +714,9 @@
         "id": "ffab030f4bc8"
       },
       "source": [
-        "### Upload the model artifacts to Google Cloud Storage\n",
+        "### Upload the model artifacts to Cloud Storage\n",
         "\n",
-        "Create a new folder in your Google Cloud Storage bucket to hold the model artifacts"
+        "Create a new folder in your Cloud Storage bucket to hold the model artifacts"
       ]
     },
     {
@@ -838,6 +812,16 @@
         "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FAPoU8Sm5E6e"
+      },
+      "source": [
+        "<div style=\"background:#e3effe; padding:5px; color:#0000aa\">\n",
+        "<strong>Note:</strong> The next cell fails if you haven't <a href=\"https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com\"><u>enabled the Vertex API</u></a>.</div>"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -907,7 +891,7 @@
         "id": "c9fc560df0a9"
       },
       "source": [
-        "The previous cell, which deploys your model to the nedpoint, can take a while to complete. If the previous cell times out before returning, your endpoint might still be successfully deployed ot an end point. Check the [Cloud Console](https://console.cloud.google.com/vertex-ai/endpoints) to verify the results.\n",
+        "The previous cell, which deploys your model to the endpoint, can take a while to complete. If the previous cell times out before returning, your endpoint might still be successfully deployed to an endpoint. Check the [Cloud Console](https://console.cloud.google.com/vertex-ai/endpoints) to verify the results.\n",
         "\n",
         "You can also extend the time to wait for deployment by changing the `deploy_request_timeout` argument passed to `model.deploy()`."
       ]
@@ -920,7 +904,7 @@
       "source": [
         "## Get online predictions\n",
         "\n",
-        "Finally, with your Stable Diffusion 2.0 model deployed to a Vertex AI endpoint, you can now get online predictions from it. Using the Vertex AI SDK, you only need a few lines of code to get an inference."
+        "Finally, with your Stable Diffusion 1.5 model deployed to a Vertex AI endpoint, you can now get online predictions from it. Using the Vertex AI SDK, you only need a few lines of code to get an inference."
       ]
     },
     {

notebooks/notebook_template_review.py

@@ -1140,7 +1140,7 @@ def add_index(path: str,
             print(f'            {tag.strip()}<br/>\n')
         print('        </td>')
         print('        <td>')
-        print(f'            <b>{title}</b>.\n')
+        print(f'            <b>{title}</b>. ')
         if args.desc:
             desc = replace_cl(desc.replace('`', ''))
             print('<br/>')
@@ -1172,8 +1172,19 @@ def add_index(path: str,
                     print(f' Learn more about <a href="{linkbacks[_]}." target="_blank">{replace_cl(tags[_])}</a>.\n')
                     
         if args.steps:
-            steps = replace_cl(steps.replace('\n', '<br/>').replace('-', '&nbsp;&nbsp;-').replace('**', '').replace('*', '&nbsp;&nbsp;-').replace('`', ''))
-            print('<br/><br/>' + steps +  '<br/>')
+            print("<devsite-expandable>\n")
+            print('  <h6 class="showalways">Notebook steps</h6>\n')
+            print('  <ul>\n')
+            
+            if ":" in steps:
+                steps = steps.split(':')[1].replace('*', '').replace('-', '').strip().split('\n')
+            else:
+                steps = []
+              
+            for step in steps:
+                print(f'    <li>{replace_cl(step)}</li>\n')
+            print('  </ul>\n')
+            print("</devsite-expandable>\n")
                     
         print('        </td>')
         print('        <td>')

notebooks/official/CODEOWNERS

@@ -39,6 +39,4 @@
 /tabular_workflows/prophet_on_vertex_pipelines.ipynb @TheMichaelHu
 /model_evaluation/custom_tabular_classification_model_evaluation.ipynb @soheilazangeneh
 /sdk/SDK_FBProphet_Forecasting_Online.ipynb @brianchunkang
-/automl/sdk_automl_forecasting_hierarchical_batch.ipynb @ivanmkc
-/prediction/custom_batch_prediction_feature_filter.ipynb @soheilazangeneh
-/feature_store/feature_store_streaming_ingestion_sdk.ipynb @soheilazangeneh
+/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines @Ark-kun

notebooks/official/automl/automl_tabular_on_vertex_pipelines.ipynb

@@ -170,7 +170,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install -U google-cloud-pipeline-components -q"
+        "!pip install -U google-cloud-pipeline-components==1.0.25 -q"
       ]
     },
     {
@@ -333,7 +333,10 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"[your-region]\"  # @param {type: \"string\"}"
+        "REGION = \"us-central1\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
       ]
     },
     {
@@ -379,8 +382,11 @@
         "import os\n",
         "import sys\n",
         "\n",
-        "# If on Google Cloud Notebook, then don't execute this code\n",
-        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
+        "# If on Vertex AI Workbench, then don't execute this code\n",
+        "IS_COLAB = \"google.colab\" in sys.modules\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
+        "    \"DL_ANACONDA_HOME\"\n",
+        "):\n",
         "    if \"google.colab\" in sys.modules:\n",
         "        from google.colab import auth as google_auth\n",
         "\n",
@@ -388,10 +394,9 @@
         "\n",
         "    # If you are running this notebook locally, replace the string below with the\n",
         "    # path to your service account key and run this cell to authenticate your GCP\n",
-        "    # account. Alternatively, you may edit this notebook to authenticate using\n",
-        "    # gcloud.\n",
+        "    # account.\n",
         "    elif not os.getenv(\"IS_TESTING\"):\n",
-        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+        "        %env GOOGLE_APPLICATION_CREDENTIALS '[your-service-account-key-path]'"
       ]
     },
     {
@@ -478,6 +483,77 @@
         "! gsutil ls -al $BUCKET_URI"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "44accda192d5"
+      },
+      "source": [
+        "#### Service Account\n",
+        "\n",
+        "You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "e0c9c4f84849"
+      },
+      "outputs": [],
+      "source": [
+        "SERVICE_ACCOUNT = \"[your-service-account]\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "604ae09ab6d3"
+      },
+      "outputs": [],
+      "source": [
+        "if (\n",
+        "    SERVICE_ACCOUNT == \"\"\n",
+        "    or SERVICE_ACCOUNT is None\n",
+        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
+        "):\n",
+        "    # Get your service account from gcloud\n",
+        "    if not IS_COLAB:\n",
+        "        shell_output = !gcloud auth list 2>/dev/null\n",
+        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
+        "\n",
+        "    else:  # IS_COLAB:\n",
+        "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
+        "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
+        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
+        "\n",
+        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d1ecb60964d5"
+      },
+      "source": [
+        "#### Set service account access for Vertex AI Pipelines\n",
+        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "a592f0a380c2"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
+        "\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -836,6 +912,7 @@
         "\n",
         "job.run()\n",
         "\n",
+        "\n",
         "pipeline_task_details = job.gca_resource.job_detail.task_details\n",
         "\n",
         "if export_additional_model_without_custom_ops:\n",
@@ -877,6 +954,7 @@
         "stage_1_tuner_task = get_task_detail(\n",
         "    pipeline_task_details, \"automl-tabular-stage-1-tuner\"\n",
         ")\n",
+        "\n",
         "stage_1_tuning_result_artifact_uri = (\n",
         "    stage_1_tuner_task.outputs[\"tuning_result_output\"].artifacts[0].uri\n",
         ")"

notebooks/official/model_monitoring/test.ipynb

@@ -0,0 +1,103 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "415646a6-b435-409f-856b-ad06676c8dd2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33m  WARNING: The script google-oauthlib-tool is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
+      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+      "pandas-gbq 0.19.1 requires google-api-core<3.0.0dev,>=2.10.2, but you have google-api-core 2.10.0 which is incompatible.\n",
+      "pandas-gbq 0.19.1 requires google-auth-oauthlib>=0.7.0, but you have google-auth-oauthlib 0.4.6 which is incompatible.\n",
+      "google-cloud-vision 3.3.1 requires google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0, but you have google-api-core 2.10.0 which is incompatible.\n",
+      "google-cloud-notebooks 1.6.1 requires google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0, but you have google-api-core 2.10.0 which is incompatible.\n",
+      "google-cloud-bigquery-storage 2.18.1 requires google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0, but you have google-api-core 2.10.0 which is incompatible.\n",
+      "tf-models-official 2.11.3 requires numpy>=1.20, but you have numpy 1.18.5 which is incompatible.\n",
+      "tf-models-official 2.11.3 requires tensorflow~=2.11.0, but you have tensorflow 2.7.0 which is incompatible.\n",
+      "google-vizier 0.0.15 requires google-api-python-client>=2.65.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
+      "google-vizier 0.0.15 requires numpy>=1.21.5, but you have numpy 1.18.5 which is incompatible.\n",
+      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\u001b[0m\u001b[31m\n",
+      "\u001b[0m"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "\n",
+    "# The Vertex AI Workbench Notebook product has specific requirements\n",
+    "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
+    "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
+    "    \"/opt/deeplearning/metadata/env_version\"\n",
+    ")\n",
+    "\n",
+    "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
+    "USER_FLAG = \"\"\n",
+    "if IS_WORKBENCH_NOTEBOOK:\n",
+    "    USER_FLAG = \"--user\"\n",
+    "\n",
+    "# Install the packages\n",
+    "! pip3 install {USER_FLAG} -q --upgrade google-cloud-aiplatform \\\n",
+    "                                        google-cloud-storage \\\n",
+    "                                        pandas-gbq\n",
+    "\n",
+    "! pip3 install {USER_FLAG} -q -U google-cloud-aiplatform \"shapely<2\" \\\n",
+    "                                 tensorflow==2.7 \\\n",
+    "                                 google-api-core==2.10 \\\n",
+    "                                 protobuf==3.20.3"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7018941a-8076-4dec-85cd-ad35394ad1ef",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import google.cloud.aiplatform as aiplatform\n",
+    "import pandas as pd\n",
+    "import tensorflow"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "995dd18e-7fcc-4cd6-a0cc-ae8ab093c24e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "environment": {
+   "kernel": "python3",
+   "name": "common-cu110.m102",
+   "type": "gcloud",
+   "uri": "gcr.io/deeplearning-platform-release/base-cu110:m102"
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/official/pipelines/README.md

@@ -304,5 +304,29 @@ The steps performed include:
 
    Learn more about [AutoML components](https://cloud.google.com/vertex-ai/docs/pipelines/vertex-automl-component).
 
-   Learn more about [BigQuery ML components](https://cloud.google.com/vertex-ai/docs/pipelines/bigqueryml-component).
+[Train custom tabular ML models with many frameworks and import to Vertex AI using Vertex Pipelines](https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines)
 
+Learn how to build a pipeline that does the following:
+
+*   Ingest data
+*   Transform data
+*   Clean up data
+*   Split data into train/test subsets
+*   Configure model
+*   Train model using multiple ML frameworks
+*   Import model into Vertex Model Registry
+*   [Optional] Deploy model to Vertex Endpoints for serving
+
+Included pipelines:
+
+*   Train ML model
+*   *   Tabular classification
+*   *   *   TensorFlow
+*   *   *   PyTorch
+*   *   *   XGBoost
+*   *   *   Scikit-learn
+*   *   Tabular regression
+*   *   *   TensorFlow
+*   *   *   PyTorch
+*   *   *   XGBoost
+*   *   *   Scikit-learn

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_classification_logistic_regression_model_using_Scikit_learn_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,73 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+binarize_column_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Binarize_column/in_CSV_format/component.yaml")
+train_logistic_regression_model_using_scikit_learn_from_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/ML_frameworks/Scikit_learn/Train_logistic_regression_model/from_CSV/component.yaml")
+upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/1f5cf6e06409b704064b2086c0a705e4e6b4fcde/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Scikit-learn_pickle_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_classification_logistic_regression_model_using_Scikit_learn_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    classification_label_column = "class"
+    all_columns = [label_column] + feature_columns
+
+    training_data = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    training_data = select_columns_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    # Cleaning the NaN values.
+    training_data = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        replacement_value="0",
+        #replacement_type_name="float",
+    ).outputs["transformed_table"]
+
+    classification_training_data = binarize_column_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_name=label_column,
+        predicate="> 0",
+        new_column_name=classification_label_column,
+    ).outputs["transformed_table"]
+
+    model = train_logistic_regression_model_using_scikit_learn_from_CSV_op(
+        dataset=classification_training_data,
+        label_column_name=classification_label_column,
+        # Optional:
+        #penalty="l2",
+        #solver="lbfgs",
+        #max_iterations=100,
+        #multi_class_mode="auto",
+        #random_seed=0,
+    ).outputs["model"]
+
+    vertex_model_name = upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        sklearn_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func = train_tabular_classification_logistic_regression_model_using_Scikit_learn_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_classification_model_using_PyTorch_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,95 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+binarize_column_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Binarize_column/in_CSV_format/component.yaml")
+create_fully_connected_pytorch_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_fully_connected_network/component.yaml")
+train_pytorch_model_from_csv_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Train_PyTorch_model/from_CSV/component.yaml")
+create_pytorch_model_archive_with_base_handler_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_PyTorch_Model_Archive/with_base_handler/component.yaml")
+upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_PyTorch_model_archive/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_classification_model_using_PyTorch_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    classification_label_column = "class"
+    all_columns = [label_column] + feature_columns
+
+    training_data = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    training_data = select_columns_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    # Cleaning the NaN values.
+    training_data = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        replacement_value="0",
+        #replacement_type_name="float",
+    ).outputs["transformed_table"]
+
+    classification_training_data = binarize_column_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_name=label_column,
+        predicate=" > 0",
+        new_column_name=classification_label_column,
+    ).outputs["transformed_table"]
+
+    network = create_fully_connected_pytorch_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        output_activation_name="sigmoid",
+        # output_size=1,
+    ).outputs["model"]
+
+    model = train_pytorch_model_from_csv_op(
+        model=network,
+        training_data=classification_training_data,
+        label_column_name=classification_label_column,
+        loss_function_name="binary_cross_entropy",
+        # Optional:
+        #number_of_epochs=1,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #batch_log_interval=100,
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    model_archive = create_pytorch_model_archive_with_base_handler_op(
+        model=model,
+        # Optional:
+        # model_name="model",
+        # model_version="1.0",
+    ).outputs["Model archive"]
+
+    vertex_model_name = upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op(
+        model_archive=model_archive,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func=train_tabular_classification_model_using_PyTorch_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_classification_model_using_TensorFlow_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,106 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+binarize_column_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Binarize_column/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+create_fully_connected_tensorflow_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Create_fully_connected_network/component.yaml")
+train_model_using_Keras_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Train_model_using_Keras/on_CSV/component.yaml")
+predict_with_TensorFlow_model_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Predict/on_CSV/component.yaml")
+upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Tensorflow_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_classification_model_using_TensorFlow_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    classification_label_column = "class"
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    classification_dataset = binarize_column_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_name=label_column,
+        predicate=" > 0",
+        new_column_name=classification_label_column,
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=classification_dataset,
+        fraction_1=training_set_fraction,
+    )
+    classification_training_data = split_task.outputs["split_1"]
+    classification_testing_data = split_task.outputs["split_2"]
+
+    network = create_fully_connected_tensorflow_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        output_activation_name="sigmoid",
+        # output_size=1,
+    ).outputs["model"]
+
+    model = train_model_using_Keras_on_CSV_op(
+        training_data=classification_training_data,
+        model=network,
+        label_column_name=classification_label_column,
+        # Optional:
+        loss_function_name="binary_crossentropy",
+        number_of_epochs=10,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #metric_names=["mean_absolute_error"],
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    predictions = predict_with_TensorFlow_model_on_CSV_data_op(
+        dataset=classification_testing_data,
+        model=model,
+        # label_column_name needs to be set when doing prediction on a dataset that has labels
+        label_column_name=classification_label_column,
+        # Optional:
+        # batch_size=1000,
+    ).outputs["predictions"]
+
+    vertex_model_name = upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func = train_tabular_classification_model_using_TensorFlow_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_classification_model_using_XGBoost_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,94 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+binarize_column_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Binarize_column/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+train_XGBoost_model_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Train/component.yaml")
+xgboost_predict_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Predict/component.yaml")
+upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_XGBoost_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_classification_model_using_XGBoost_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    classification_label_column = "class"
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    classification_dataset = binarize_column_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_name=label_column,
+        predicate="> 0",
+        new_column_name=classification_label_column,
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=classification_dataset,
+        fraction_1=training_set_fraction,
+    )
+    classification_training_data = split_task.outputs["split_1"]
+    classification_testing_data = split_task.outputs["split_2"]
+
+    model = train_XGBoost_model_on_CSV_op(
+        training_data=classification_training_data,
+        label_column_name=classification_label_column,
+        objective="binary:logistic",
+        # Optional:
+        #starting_model=None,
+        #num_iterations=10,
+        #booster_params={},
+        #booster="gbtree",
+        #learning_rate=0.3,
+        #min_split_loss=0,
+        #max_depth=6,
+    ).outputs["model"]
+
+    # Predicting on the testing data
+    predictions = xgboost_predict_on_CSV_op(
+        data=classification_testing_data,
+        model=model,
+        # label_column needs to be set when doing prediction on a dataset that has labels
+        label_column_name=classification_label_column,
+    ).outputs["predictions"]
+
+    vertex_model_name = upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func = train_tabular_classification_model_using_XGBoost_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_classification_model_using_all_frameworks_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,224 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+binarize_column_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Binarize_column/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+
+# TensorFlow
+create_fully_connected_tensorflow_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Create_fully_connected_network/component.yaml")
+train_model_using_Keras_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Train_model_using_Keras/on_CSV/component.yaml")
+predict_with_TensorFlow_model_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Predict/on_CSV/component.yaml")
+upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Tensorflow_model/component.yaml")
+
+# PyTorch
+create_fully_connected_pytorch_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_fully_connected_network/component.yaml")
+train_pytorch_model_from_csv_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Train_PyTorch_model/from_CSV/component.yaml")
+create_pytorch_model_archive_with_base_handler_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_PyTorch_Model_Archive/with_base_handler/component.yaml")
+upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_PyTorch_model_archive/component.yaml")
+
+# XGBoost
+train_XGBoost_model_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Train/component.yaml")
+xgboost_predict_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Predict/component.yaml")
+upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_XGBoost_model/component.yaml")
+
+# Scikit-learn
+#train_linear_regression_model_using_scikit_learn_from_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/1f5cf6e06409b704064b2086c0a705e4e6b4fcde/community-content/pipeline_components/ML_frameworks/Scikit_learn/Train_linear_regression_model/from_CSV/component.yaml")
+train_logistic_regression_model_using_scikit_learn_from_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/1f5cf6e06409b704064b2086c0a705e4e6b4fcde/community-content/pipeline_components/ML_frameworks/Scikit_learn/Train_logistic_regression_model/from_CSV/component.yaml")
+upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Scikit-learn_pickle_model/component.yaml")
+
+# Vertex AI
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_classification_model_using_all_frameworks_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    classification_label_column = "class"
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    classification_dataset = binarize_column_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_name=label_column,
+        predicate=" > 0",
+        new_column_name=classification_label_column,
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=classification_dataset,
+        fraction_1=training_set_fraction,
+    )
+    classification_training_data = split_task.outputs["split_1"]
+    classification_testing_data = split_task.outputs["split_2"]
+
+    # TensorFlow
+    tensorflow_network = create_fully_connected_tensorflow_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        output_activation_name="sigmoid",
+        # output_size=1,
+    ).outputs["model"]
+
+    tensorflow_model = train_model_using_Keras_on_CSV_op(
+        training_data=classification_training_data,
+        model=tensorflow_network,
+        label_column_name=classification_label_column,
+        # Optional:
+        loss_function_name="binary_crossentropy",
+        number_of_epochs=10,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #metric_names=["mean_absolute_error"],
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    tensorflow_predictions = predict_with_TensorFlow_model_on_CSV_data_op(
+        dataset=classification_testing_data,
+        model=tensorflow_model,
+        # label_column_name needs to be set when doing prediction on a dataset that has labels
+        label_column_name=classification_label_column,
+        # Optional:
+        # batch_size=1000,
+    ).outputs["predictions"]
+
+    tensorflow_vertex_model_name = upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op(
+        model=tensorflow_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        tensorflow_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=tensorflow_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # PyTorch
+    pytorch_network = create_fully_connected_pytorch_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        output_activation_name="sigmoid",
+        # output_size=1,
+    ).outputs["model"]
+
+    pytorch_model = train_pytorch_model_from_csv_op(
+        model=pytorch_network,
+        training_data=classification_training_data,
+        label_column_name=classification_label_column,
+        loss_function_name="binary_cross_entropy",
+        # Optional:
+        #number_of_epochs=1,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #batch_log_interval=100,
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    pytorch_model_archive = create_pytorch_model_archive_with_base_handler_op(
+        model=pytorch_model,
+        # Optional:
+        # model_name="model",
+        # model_version="1.0",
+    ).outputs["Model archive"]
+
+    pytorch_vertex_model_name = upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op(
+        model_archive=pytorch_model_archive,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        pytorch_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=pytorch_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # XGBoost
+    xgboost_model = train_XGBoost_model_on_CSV_op(
+        training_data=classification_training_data,
+        label_column_name=classification_label_column,
+        objective="binary:logistic",
+        # Optional:
+        #starting_model=None,
+        #num_iterations=10,
+        #booster_params={},
+        #booster="gbtree",
+        #learning_rate=0.3,
+        #min_split_loss=0,
+        #max_depth=6,
+    ).outputs["model"]
+
+    # Predicting on the testing data
+    xgboost_predictions = xgboost_predict_on_CSV_op(
+        data=classification_testing_data,
+        model=xgboost_model,
+        # label_column needs to be set when doing prediction on a dataset that has labels
+        label_column_name=classification_label_column,
+    ).outputs["predictions"]
+
+    xgboost_vertex_model_name = upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op(
+        model=xgboost_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        xgboost_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=xgboost_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # Scikit-learn
+    sklearn_model = train_logistic_regression_model_using_scikit_learn_from_CSV_op(
+        dataset=classification_training_data,
+        label_column_name=classification_label_column,
+        # Optional:
+        #penalty="l2",
+        #solver="lbfgs",
+        #max_iterations=100,
+        #multi_class_mode="auto",
+        #random_seed=0,
+    ).outputs["model"]
+
+    sklearn_vertex_model_name = upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op(
+        model=sklearn_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        sklearn_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=sklearn_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func=train_tabular_classification_model_using_all_frameworks_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_regression_linear_model_using_Scikit_learn_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,57 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+train_linear_regression_model_using_scikit_learn_from_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/1f5cf6e06409b704064b2086c0a705e4e6b4fcde/community-content/pipeline_components/ML_frameworks/Scikit_learn/Train_linear_regression_model/from_CSV/component.yaml")
+upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Scikit-learn_pickle_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_regression_linear_model_using_Scikit_learn_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    all_columns = [label_column] + feature_columns
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    training_data = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    training_data = select_columns_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    # Cleaning the NaN values.
+    training_data = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        replacement_value="0",
+        #replacement_type_name="float",
+    ).outputs["transformed_table"]
+
+    model = train_linear_regression_model_using_scikit_learn_from_CSV_op(
+        dataset=training_data,
+        label_column_name=label_column,
+    ).outputs["model"]
+
+    vertex_model_name = upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        sklearn_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func = train_tabular_regression_linear_model_using_Scikit_learn_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_regression_model_using_PyTorch_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,85 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+create_fully_connected_pytorch_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_fully_connected_network/component.yaml")
+train_pytorch_model_from_csv_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Train_PyTorch_model/from_CSV/component.yaml")
+create_pytorch_model_archive_with_base_handler_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_PyTorch_Model_Archive/with_base_handler/component.yaml")
+upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_PyTorch_model_archive/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_regression_model_using_PyTorch_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    all_columns = [label_column] + feature_columns
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    training_data = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    training_data = select_columns_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    # Cleaning the NaN values.
+    training_data = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=training_data,
+        replacement_value="0",
+        #replacement_type_name="float",
+    ).outputs["transformed_table"]
+
+    network = create_fully_connected_pytorch_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        # output_activation_name=None,
+        # output_size=1,
+    ).outputs["model"]
+
+    model = train_pytorch_model_from_csv_op(
+        model=network,
+        training_data=training_data,
+        label_column_name=label_column,
+        # Optional:
+        #loss_function_name="mse_loss",
+        #number_of_epochs=1,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #batch_log_interval=100,
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    model_archive = create_pytorch_model_archive_with_base_handler_op(
+        model=model,
+        # Optional:
+        # model_name="model",
+        # model_version="1.0",
+    ).outputs["Model archive"]
+
+    vertex_model_name = upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op(
+        model_archive=model_archive,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func=train_tabular_regression_model_using_PyTorch_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_regression_model_using_TensorFlow_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,97 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+create_fully_connected_tensorflow_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Create_fully_connected_network/component.yaml")
+train_model_using_Keras_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Train_model_using_Keras/on_CSV/component.yaml")
+predict_with_TensorFlow_model_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Predict/on_CSV/component.yaml")
+upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Tensorflow_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_regression_model_using_Tensorflow_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=dataset,
+        fraction_1=training_set_fraction,
+    )
+    training_data = split_task.outputs["split_1"]
+    testing_data = split_task.outputs["split_2"]
+
+    network = create_fully_connected_tensorflow_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        # output_activation_name=None,
+        # output_size=1,
+    ).outputs["model"]
+
+    model = train_model_using_Keras_on_CSV_op(
+        training_data=training_data,
+        model=network,
+        label_column_name=label_column,
+        # Optional:
+        #loss_function_name="mean_squared_error",
+        number_of_epochs=10,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        metric_names=["mean_absolute_error"],
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    predictions = predict_with_TensorFlow_model_on_CSV_data_op(
+        dataset=testing_data,
+        model=model,
+        # label_column_name needs to be set when doing prediction on a dataset that has labels
+        label_column_name=label_column,
+        # Optional:
+        # batch_size=1000,
+    ).outputs["predictions"]
+
+    vertex_model_name = upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func=train_tabular_regression_model_using_Tensorflow_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_regression_model_using_XGBoost_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,85 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+train_XGBoost_model_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Train/component.yaml")
+xgboost_predict_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Predict/component.yaml")
+upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_XGBoost_model/component.yaml")
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_regression_model_using_XGBoost_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=dataset,
+        fraction_1=training_set_fraction,
+    )
+    training_data = split_task.outputs["split_1"]
+    testing_data = split_task.outputs["split_2"]
+
+    model = train_XGBoost_model_on_CSV_op(
+        training_data=training_data,
+        label_column_name=label_column,
+        # Optional:
+        #starting_model=None,
+        #num_iterations=10,
+        #booster_params={},
+        #objective="reg:squarederror",
+        #booster="gbtree",
+        #learning_rate=0.3,
+        #min_split_loss=0,
+        #max_depth=6,
+    ).outputs["model"]
+
+    # Predicting on the testing data
+    predictions = xgboost_predict_on_CSV_op(
+        data=testing_data,
+        model=model,
+        # label_column needs to be set when doing prediction on a dataset that has labels
+        label_column_name=label_column,
+    ).outputs["predictions"]
+
+    vertex_model_name = upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op(
+        model=model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func = train_tabular_regression_model_using_XGBoost_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/pipelines/Train_tabular_models_with_many_frameworks_and_import_to_Vertex_AI_using_Pipelines/Train_tabular_regression_model_using_all_frameworks_and_import_to_Vertex_AI/pipeline.py

@@ -0,0 +1,208 @@
+# python3 -m pip install "kfp<2.0.0" "google-cloud-aiplatform>=1.16.0" --upgrade --quiet
+from kfp import components
+
+# %% Loading components
+download_from_gcs_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/storage/download/component.yaml")
+select_columns_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Select_columns/in_CSV_format/component.yaml")
+fill_all_missing_values_using_Pandas_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/pandas/Fill_all_missing_values/in_CSV_format/component.yaml")
+split_rows_into_subsets_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/dataset_manipulation/Split_rows_into_subsets/in_CSV/component.yaml")
+
+# TensorFlow
+create_fully_connected_tensorflow_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Create_fully_connected_network/component.yaml")
+train_model_using_Keras_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Train_model_using_Keras/on_CSV/component.yaml")
+predict_with_TensorFlow_model_on_CSV_data_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/tensorflow/Predict/on_CSV/component.yaml")
+upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Tensorflow_model/component.yaml")
+
+# PyTorch
+create_fully_connected_pytorch_network_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_fully_connected_network/component.yaml")
+train_pytorch_model_from_csv_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Train_PyTorch_model/from_CSV/component.yaml")
+create_pytorch_model_archive_with_base_handler_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/PyTorch/Create_PyTorch_Model_Archive/with_base_handler/component.yaml")
+upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_PyTorch_model_archive/component.yaml")
+
+# XGBoost
+train_XGBoost_model_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Train/component.yaml")
+xgboost_predict_on_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/XGBoost/Predict/component.yaml")
+upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_XGBoost_model/component.yaml")
+
+# Scikit-learn
+train_linear_regression_model_using_scikit_learn_from_CSV_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/1f5cf6e06409b704064b2086c0a705e4e6b4fcde/community-content/pipeline_components/ML_frameworks/Scikit_learn/Train_linear_regression_model/from_CSV/component.yaml")
+upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Upload_Scikit-learn_pickle_model/component.yaml")
+
+# Vertex AI
+deploy_model_to_endpoint_op = components.load_component_from_url("https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/399405402d95f4a011e2d2e967c96f8508ba5688/community-content/pipeline_components/google-cloud/Vertex_AI/Models/Deploy_to_endpoint/component.yaml")
+
+# %% Pipeline definition
+def train_tabular_regression_model_using_all_frameworks_pipeline():
+    dataset_gcs_uri = "gs://ml-pipeline-dataset/Chicago_taxi_trips/chicago_taxi_trips_2019-01-01_-_2019-02-01_limit=10000.csv"
+    feature_columns = ["trip_seconds", "trip_miles", "pickup_community_area", "dropoff_community_area", "fare", "tolls", "extras"]  # Excluded "trip_total"
+    label_column = "tips"
+    training_set_fraction = 0.8
+    # Deploying the model might incur additional costs over time
+    deploy_model = False
+
+    all_columns = [label_column] + feature_columns
+
+    dataset = download_from_gcs_op(
+        gcs_path=dataset_gcs_uri
+    ).outputs["Data"]
+
+    dataset = select_columns_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        column_names=all_columns,
+    ).outputs["transformed_table"]
+
+    dataset = fill_all_missing_values_using_Pandas_on_CSV_data_op(
+        table=dataset,
+        replacement_value="0",
+        # # Optional:
+        # column_names=None,  # =[...]
+    ).outputs["transformed_table"]
+
+    split_task = split_rows_into_subsets_op(
+        table=dataset,
+        fraction_1=training_set_fraction,
+    )
+    training_data = split_task.outputs["split_1"]
+    testing_data = split_task.outputs["split_2"]
+
+    # TensorFlow
+    tensorflow_network = create_fully_connected_tensorflow_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        # output_activation_name=None,
+        # output_size=1,
+    ).outputs["model"]
+
+    tensorflow_model = train_model_using_Keras_on_CSV_op(
+        training_data=training_data,
+        model=tensorflow_network,
+        label_column_name=label_column,
+        # Optional:
+        #loss_function_name="mean_squared_error",
+        number_of_epochs=10,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        metric_names=["mean_absolute_error"],
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    tensorflow_predictions = predict_with_TensorFlow_model_on_CSV_data_op(
+        dataset=testing_data,
+        model=tensorflow_model,
+        # label_column_name needs to be set when doing prediction on a dataset that has labels
+        label_column_name=label_column,
+        # Optional:
+        # batch_size=1000,
+    ).outputs["predictions"]
+
+    tensorflow_vertex_model_name = upload_Tensorflow_model_to_Google_Cloud_Vertex_AI_op(
+        model=tensorflow_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        tensorflow_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=tensorflow_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # PyTorch
+    pytorch_network = create_fully_connected_pytorch_network_op(
+        input_size=len(feature_columns),
+        # Optional:
+        hidden_layer_sizes=[10],
+        activation_name="elu",
+        # output_activation_name=None,
+        # output_size=1,
+    ).outputs["model"]
+
+    pytorch_model = train_pytorch_model_from_csv_op(
+        model=pytorch_network,
+        training_data=training_data,
+        label_column_name=label_column,
+        # Optional:
+        #loss_function_name="mse_loss",
+        #number_of_epochs=1,
+        #learning_rate=0.1,
+        #optimizer_name="Adadelta",
+        #optimizer_parameters={},
+        #batch_size=32,
+        #batch_log_interval=100,
+        #random_seed=0,
+    ).outputs["trained_model"]
+
+    pytorch_model_archive = create_pytorch_model_archive_with_base_handler_op(
+        model=pytorch_model,
+        # Optional:
+        # model_name="model",
+        # model_version="1.0",
+    ).outputs["Model archive"]
+
+    pytorch_vertex_model_name = upload_PyTorch_model_archive_to_Google_Cloud_Vertex_AI_op(
+        model_archive=pytorch_model_archive,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        pytorch_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=pytorch_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # XGBoost
+    xgboost_model = train_XGBoost_model_on_CSV_op(
+        training_data=training_data,
+        label_column_name=label_column,
+        # Optional:
+        #starting_model=None,
+        #num_iterations=10,
+        #booster_params={},
+        #objective="reg:squarederror",
+        #booster="gbtree",
+        #learning_rate=0.3,
+        #min_split_loss=0,
+        #max_depth=6,
+    ).outputs["model"]
+
+    # Predicting on the testing data
+    xgboost_predictions = xgboost_predict_on_CSV_op(
+        data=testing_data,
+        model=xgboost_model,
+        # label_column needs to be set when doing prediction on a dataset that has labels
+        label_column_name=label_column,
+    ).outputs["predictions"]
+
+    xgboost_vertex_model_name = upload_XGBoost_model_to_Google_Cloud_Vertex_AI_op(
+        model=xgboost_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        xgboost_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=xgboost_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+    # Scikit-learn
+    sklearn_model = train_linear_regression_model_using_scikit_learn_from_CSV_op(
+        dataset=training_data,
+        label_column_name=label_column,
+    ).outputs["model"]
+
+    sklearn_vertex_model_name = upload_Scikit_learn_pickle_model_to_Google_Cloud_Vertex_AI_op(
+        model=sklearn_model,
+    ).outputs["model_name"]
+
+    # Deploying the model might incur additional costs over time
+    if deploy_model:
+        sklearn_vertex_endpoint_name = deploy_model_to_endpoint_op(
+            model_name=sklearn_vertex_model_name,
+        ).outputs["endpoint_name"]
+
+pipeline_func=train_tabular_regression_model_using_all_frameworks_pipeline
+
+# %% Pipeline submission
+if __name__ == '__main__':
+    from google.cloud import aiplatform
+    aiplatform.PipelineJob.from_pipeline_func(pipeline_func=pipeline_func).submit()

notebooks/official/tabular_workflows/prophet_on_vertex_pipelines.ipynb

@@ -8,7 +8,7 @@
       },
       "outputs": [],
       "source": [
-        "# Copyright 2022 Google LLC\n",
+        "# Copyright 2023 Google LLC\n",
         "#\n",
         "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
         "# you may not use this file except in compliance with the License.\n",
@@ -904,23 +904,8 @@
       "source": [
         "### Create the training job\n",
         "\n",
-        "Use the `get_prophet_train_pipeline_and_parameters` function to configure your training job. Here, you can configure the [data splitting](https://cloud.google.com/vertex-ai/docs/tabular-data/data-splits#forecasting) and [windowing](https://cloud.google.com/vertex-ai/docs/tabular-data/bp-tabular#context-window) strategies, hyperparameter tuning trials, and the Dataflow job for parallelizing model training.\n",
+        "Use the `get_prophet_train_pipeline_and_parameters` function to configure your training job.\n",
         "\n",
-        "The following parameters are used to configure splitting:\n",
-        "-  `predefined_split_key` (str): The predefined_split column name. Other splitting options include fractional splitting on a time column.\n",
-        "- `training_fraction` (float): The percentage of the data that belongs to the training set. Set this value if you are using a fraction split or a timestamp split.\n",
-        "- `training_fraction` (float): The percentage of the data that belongs to the validation set. Set this value if you are using a fraction split or a timestamp split.\n",
-        "- `training_fraction` (float): The percentage of the data that belongs to the test set. Set this value if you are using a fraction split or a timestamp split.\n",
-        "- `timestamp_split_key` (str): The name of the column containing the timestamps for the data split. Set this value if you are using a timestamp split.\n",
-        "\n",
-        "The following parameters are used to configure windowing:\n",
-        "- `window_column` (str): Name of the column that should be used to filter input rows. The column should contain either booleans or string booleans; if the value of the row is True, generate a sliding window from that row.\n",
-        "-  `window_stride_length` (int): Step length used to generate input examples. Every window_stride_length rows is used to generate a sliding window. Other windowing options include using a predefined column or setting a maximum number to downsample to.\n",
-        "- `window_max_count` (int): Number of rows that should be used to generate input examples. If the total row count is larger than this number, the input data will be randomly sampled to hit the count.\n",
-        "\n",
-        "For more information on how the pipeline parameters relate to splitting and windowing, see the [Prophet on Vertex AI documentation](https://cloud.google.com/vertex-ai/docs/tabular-data/forecasting-prophet).\n",
-        "\n",
-        "Other parameters used by the training job below include:\n",
         "-  `project` (str): The GCP project that runs the pipeline components.\n",
         "-  `location` (str): The GCP region for Vertex AI.\n",
         "-  `root_dir` (str): The Cloud Storage location to store the output.\n",
@@ -931,10 +916,16 @@
         "-  `optimization_objective` (str): Optimization objective for tuning. Supported metrics come from Prophet's performance_metrics function. These are mse, rmse, mae, mape, mdape, smape, and coverage.\n",
         "-  `data_granularity_unit` (str): String representing the units of time for the time column.\n",
         "- `data_source_bigquery_table_path` (str): The BigQuery table path of format bq://bq_project.bq_dataset.bq_table\n",
-        "- `max_num_trials` (int): Maximum number of tuning trials to perform per time series.\n",
-        "-  `trainer_dataflow_machine_type` (str): The dataflow machine type used for training.\n",
-        "- `trainer_dataflow_max_num_workers` (int): The maximum number of Dataflow workers used for training.\n",
         "- `dataflow_service_account` (str): Custom service account to run dataflow jobs. The dataflow job can also be configured to use private IPs and a specific VPC subnet.\n",
+        "- `max_num_trials` (int): Maximum number of tuning trials to perform per time series.\n",
+        "-  `predefined_split_key` (str): The name of the predefined_split column. Vertex AI splits the data using the TRAIN, VALIDATE, or TEST values in this column. To learn more about data splits and for the full set of data split parameters, refer to the public documentation page [Forecasting with Prophet](https://cloud.google.com/vertex-ai/docs/tabular-data/forecasting-prophet).\n",
+        "-  `window_stride_length` (int): Step length used to generate input examples. Every window_stride_length rows is used to generate a sliding window. To learn more about windowing and for the full set of windowing parameters, refer to the public documentation page [Forecasting with Prophet](https://cloud.google.com/vertex-ai/docs/tabular-data/forecasting-prophet).\n",
+        "-  `trainer_dataflow_machine_type` (str): The Dataflow machine type used for training.\n",
+        "- `trainer_dataflow_max_num_workers` (int): The maximum number of Dataflow workers used for training.\n",
+        "- `evaluation_dataflow_machine_type` (int): The Dataflow machine type used for evaluation.\n",
+        "- `evaluation_dataflow_max_num_workers` (int): The max number of Dataflow workers used for evaluation.\n",
+        "\n",
+        "Prophet training jobs run on Dataflow, meaning there is an initial startup time of 5-7 minutes. Any runtime past this startup time can be improved by scaling vertically (ex. changing the machine type from `n1-standard-2` to `e2-highcpu-8`) or horizontally (ex. increasing the number of workers from `5` to `100`).\n",
         "\n",
         "For a full list of parameters, see the GCPC SDK [documentation](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-1.0.33/google_cloud_pipeline_components.experimental.automl.forecasting.html#google_cloud_pipeline_components.experimental.automl.forecasting.utils.get_prophet_train_pipeline_and_parameters).\n"
       ]
@@ -974,7 +965,9 @@
         "    window_stride_length=window_stride_length,\n",
         "    max_num_trials=2,\n",
         "    trainer_dataflow_machine_type=\"n1-standard-2\",\n",
-        "    trainer_dataflow_max_num_workers=50,\n",
+        "    trainer_dataflow_max_num_workers=5,\n",
+        "    evaluation_dataflow_machine_type=\"n1-standard-1\",\n",
+        "    evaluation_dataflow_max_num_workers=1,\n",
         "    dataflow_service_account=SERVICE_ACCOUNT,\n",
         ")"
       ]
@@ -1118,7 +1111,7 @@
         "    data_source_bigquery_table_path=PREDICTION_DATASET_BQ_PATH,\n",
         "    bigquery_destination_uri=dataset_path,\n",
         "    machine_type=\"n1-standard-2\",\n",
-        "    max_num_workers=50,\n",
+        "    max_num_workers=5,\n",
         ")"
       ]
     },