per linter

consolidated imports into a single cell
per linter
2026-09-27 07:31:58 +00:00 · 2023-02-07 21:08:06 +00:00 · 2023-02-07 21:03:27 +00:00 · 2023-02-07 20:55:12 +00:00 · 2023-02-07 20:45:25 +00:00 · 2023-02-07 20:28:52 +00:00
13 changed files with 6727 additions and 79 deletions
@@ -5,6 +5,6 @@ nbconvert
 black==22.10.0
 pyupgrade==2.38.4
 isort==5.10.1
-flake8==4.0.1
+flake8==6.0.0
 nbqa==1.5.3

@@ -40,4 +40,4 @@
 /notebooks/community/pipelines/google_cloud_pipeline_components_bqml_pipeline_anomaly_detection.ipynb @inardini
 /notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb @Narwhalprime
 /notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb @Narwhalprime
-
+/notebooks/community/feature_store/get_started_vertex_feature_store.ipynb @junkourata
@@ -54,18 +54,20 @@
    {
      "cell_type": "markdown",
      "metadata": {
-        "id": "tvgnzT1CKxrO"
+        "id": "239ba71252d3"
      },
      "source": [
        "## Overview\n",
        "\n",
-        "This notebook shows how to use `Vertex AI Pipelines` and `BigQuery ML pipeline components` to train and evaluate a demand forecasting model.\n",
-        "\n",
-        "### Dataset\n",
-        "\n",
-        "The dataset is a modified version of the dataset in [Build and visualize demand forecast predictions using Datastream, Dataflow, BigQuery ML, and Looker\n",
-        "](https://cloud.google.com/architecture/build-visualize-demand-forecast-prediction-datastream-dataflow-bigqueryml-looker) solution architecture\n",
-        "\n",
+        "This notebook shows how to use `Vertex AI Pipelines` and `BigQuery ML pipeline components` to train and evaluate a demand forecasting model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "25c28706c23e"
+      },
+      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to train and evaluate a BigQuery ML model using Vertex AI Pipelines and BigQuery ML pipeline components. \n",
@@ -87,8 +89,27 @@
        "  - Generate the ARIMA Plus forecasts\n",
        "  - Generate the ARIMA PLUS forecast explainations\n",
        "- Compile the pipeline.\n",
-        "- Execute the pipeline.\n",
+        "- Execute the pipeline."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "586acfa9b502"
+      },
+      "source": [
+        "### Dataset\n",
        "\n",
+        "The dataset is a modified version of the dataset in [Build and visualize demand forecast predictions using Datastream, Dataflow, BigQuery ML, and Looker\n",
+        "](https://cloud.google.com/architecture/build-visualize-demand-forecast-prediction-datastream-dataflow-bigqueryml-looker) solution architecture\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tvgnzT1CKxrO"
+      },
+      "source": [
        "### Costs \n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
@@ -352,9 +373,8 @@
        "id": "06571eb4063b"
      },
      "source": [
-        "#### Timestamp\n",
-        "\n",
-        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
+        "#### UUID\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
      ]
    },
    {
@@ -365,9 +385,16 @@
      },
      "outputs": [],
      "source": [
-        "from datetime import datetime\n",
+        "import random\n",
+        "import string\n",
        "\n",
-        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+        "\n",
+        "# Generate a uuid of a specifed length(default=8)\n",
+        "def generate_uuid(length: int = 8) -> str:\n",
+        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
+        "\n",
+        "\n",
+        "UUID = generate_uuid()"
      ]
    },
    {
@@ -485,7 +512,7 @@
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
-        "    BUCKET_NAME = PROJECT_ID + \"-aip-\" + TIMESTAMP\n",
+        "    BUCKET_NAME = PROJECT_ID + \"-aip-\" + UUID\n",
        "    BUCKET_URI = f\"gs://{BUCKET_NAME}\""
      ]
    },
@@ -706,6 +733,7 @@
        "KFP_COMPONENTS_PATH = \"components\"\n",
        "PIPELINES_PATH = \"pipelines\"\n",
        "\n",
+        "! mkdir -m 777 -p {DATA_PATH}\n",
        "! mkdir -m 777 -p {KFP_COMPONENTS_PATH}\n",
        "! mkdir -m 777 -p {PIPELINES_PATH}"
      ]
@@ -771,7 +799,7 @@
        "  --location={LOCATION} \\\n",
        "  --source_format=CSV \\\n",
        "  --skip_leading_rows=1\\\n",
-        "  fast_fresh.orders_{TIMESTAMP} \\\n",
+        "  fast_fresh.orders_{UUID} \\\n",
        "  {RAW_DATA_URI} \\\n",
        "  time_of_sale:DATETIME,order_id:INTEGER,product_name:STRING,price:NUMERIC,quantity:NUMERIC,payment_method:STRING,store_id:INTEGER,user_id:INTEGER"
      ]
@@ -782,7 +810,7 @@
        "id": "ZrgOD30o7HcL"
      },
      "source": [
-        "## BQML Training Formalization\n",
+        "## BigQuery ML Training Formalization\n",
        "\n",
        "In the next cells, you build the components and pipeline to train and evaluate the BQML demand forecasting model."
      ]
@@ -820,13 +848,13 @@
        "BQ_EVALUATE_MODEL_TABLE_PREFIX = \"orders_arima_model_evaluate\"\n",
        "BQ_FORECAST_TABLE_PREFIX = \"orders_arima_forecast\"\n",
        "BQ_EXPLAIN_FORECAST_TABLE_PREFIX = \"orders_arima_explain_forecast\"\n",
-        "BQ_ORDERS_TABLE = f\"{BQ_ORDERS_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_TRAINING_TABLE = f\"{BQ_TRAINING_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_MODEL_TABLE = f\"{BQ_MODEL_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_EVALUATE_TS_TABLE = f\"{BQ_EVALUATE_TS_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_EVALUATE_MODEL_TABLE = f\"{BQ_EVALUATE_MODEL_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_FORECAST_TABLE = f\"{BQ_FORECAST_TABLE_PREFIX}_{TIMESTAMP}\"\n",
-        "BQ_EXPLAIN_FORECAST_TABLE = f\"{BQ_EXPLAIN_FORECAST_TABLE_PREFIX}_{TIMESTAMP}\"\n",
+        "BQ_ORDERS_TABLE = f\"{BQ_ORDERS_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_TRAINING_TABLE = f\"{BQ_TRAINING_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_MODEL_TABLE = f\"{BQ_MODEL_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_EVALUATE_TS_TABLE = f\"{BQ_EVALUATE_TS_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_EVALUATE_MODEL_TABLE = f\"{BQ_EVALUATE_MODEL_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_FORECAST_TABLE = f\"{BQ_FORECAST_TABLE_PREFIX}_{UUID}\"\n",
+        "BQ_EXPLAIN_FORECAST_TABLE = f\"{BQ_EXPLAIN_FORECAST_TABLE_PREFIX}_{UUID}\"\n",
        "\n",
        "BQ_TRAIN_CONFIGURATION = {\n",
        "    \"destinationTable\": {\n",
@@ -1022,7 +1050,7 @@
        "id": "pcSL1FHk69KT"
      },
      "source": [
-        "### Build the BQML training pipeline\n",
+        "### Build the BigQuery ML training pipeline\n",
        "\n",
        "Define your workflow using Kubeflow Pipelines DSL package. \n",
        "\n",
@@ -1094,8 +1122,8 @@
        "        location=location,\n",
        "    ).set_display_name(\"get train data\")\n",
        "\n",
-        "    # Train the ARIMA PLUS model\n",
-        "    bq_arima_model_op = (\n",
+        "    # Run an ARIMA PLUS experiment\n",
+        "    bq_arima_model_exp_op = (\n",
        "        BigqueryCreateModelJobOp(\n",
        "            query=f\"\"\"\n",
        "        -- create model table\n",
@@ -1104,10 +1132,7 @@
        "        MODEL_TYPE = \\'ARIMA_PLUS\\',\n",
        "        TIME_SERIES_TIMESTAMP_COL = \\'hourly_timestamp\\',\n",
        "        TIME_SERIES_DATA_COL = \\'total_sold\\',\n",
-        "        TIME_SERIES_ID_COL = [\\'product_name\\'],\n",
-        "        MODEL_REGISTRY = \\'vertex_ai\\',\n",
-        "        VERTEX_AI_MODEL_ID = \\'order_demand_forecasting\\',\n",
-        "        VERTEX_AI_MODEL_VERSION_ALIASES = [\\'staging\\']\n",
+        "        TIME_SERIES_ID_COL = [\\'product_name\\']\n",
        "        ) AS\n",
        "        SELECT\n",
        "          hourly_timestamp,\n",
@@ -1119,7 +1144,7 @@
        "            project=project,\n",
        "            location=location,\n",
        "        )\n",
-        "        .set_display_name(\"train arima plus model\")\n",
+        "        .set_display_name(\"run arima+ model experiment\")\n",
        "        .after(create_training_dataset_op)\n",
        "    )\n",
        "\n",
@@ -1128,12 +1153,12 @@
        "        BigqueryMLArimaEvaluateJobOp(\n",
        "            project=project,\n",
        "            location=location,\n",
-        "            model=bq_arima_model_op.outputs[\"model\"],\n",
+        "            model=bq_arima_model_exp_op.outputs[\"model\"],\n",
        "            show_all_candidate_models=False,\n",
        "            job_configuration_query=bq_evaluate_time_series_configuration,\n",
        "        )\n",
        "        .set_display_name(\"evaluate arima plus time series\")\n",
-        "        .after(bq_arima_model_op)\n",
+        "        .after(bq_arima_model_exp_op)\n",
        "    )\n",
        "\n",
        "    # Evaluate ARIMA Plus model\n",
@@ -1141,12 +1166,12 @@
        "        BigqueryEvaluateModelJobOp(\n",
        "            project=project,\n",
        "            location=location,\n",
-        "            model=bq_arima_model_op.outputs[\"model\"],\n",
+        "            model=bq_arima_model_exp_op.outputs[\"model\"],\n",
        "            query_statement=f\"\"\"SELECT * FROM `{project}.{bq_dataset}.{bq_training_table}` WHERE split='TEST'\"\"\",\n",
        "            job_configuration_query=bq_evaluate_model_configuration,\n",
        "        )\n",
        "        .set_display_name(\"evaluate arima plus model\")\n",
-        "        .after(bq_arima_model_op)\n",
+        "        .after(bq_arima_model_exp_op)\n",
        "    )\n",
        "\n",
        "    # Plot model metrics\n",
@@ -1164,6 +1189,34 @@
        "        < PERF_THRESHOLD,\n",
        "        name=\"avg. mae good\",\n",
        "    ):\n",
+        "        # Train the ARIMA PLUS model\n",
+        "        bq_arima_model_op = (\n",
+        "            BigqueryCreateModelJobOp(\n",
+        "                query=f\"\"\"\n",
+        "        -- create model table\n",
+        "        CREATE OR REPLACE MODEL `{project}.{bq_dataset}.{bq_model_table}`\n",
+        "        OPTIONS(\n",
+        "        MODEL_TYPE = \\'ARIMA_PLUS\\',\n",
+        "        TIME_SERIES_TIMESTAMP_COL = \\'hourly_timestamp\\',\n",
+        "        TIME_SERIES_DATA_COL = \\'total_sold\\',\n",
+        "        TIME_SERIES_ID_COL = [\\'product_name\\'],\n",
+        "        MODEL_REGISTRY = \\'vertex_ai\\',\n",
+        "        VERTEX_AI_MODEL_ID = \\'order_demand_forecasting\\',\n",
+        "        VERTEX_AI_MODEL_VERSION_ALIASES = [\\'staging\\']\n",
+        "        ) AS\n",
+        "        SELECT\n",
+        "          DATETIME_TRUNC(time_of_sale, HOUR) as hourly_timestamp,\n",
+        "          product_name,\n",
+        "          SUM(quantity) AS total_sold,\n",
+        "          FROM `{project}.{bq_dataset}.{bq_orders_table}`\n",
+        "        GROUP BY hourly_timestamp, product_name;\n",
+        "        \"\"\",\n",
+        "                project=project,\n",
+        "                location=location,\n",
+        "            )\n",
+        "            .set_display_name(\"train arima+ model\")\n",
+        "            .after(get_evaluation_model_metrics_op)\n",
+        "        )\n",
        "\n",
        "        # Generate the ARIMA PLUS forecasts\n",
        "        bq_arima_forecast_op = (\n",
@@ -1224,7 +1277,7 @@
      "source": [
        "### Execute your pipeline\n",
        "\n",
-        "Next, you execute the pipeline. It takes the following parameters which we set as default:\n",
+        "Next, we execute the pipeline. It takes the following parameters which we set as default:\n",
        "\n",
        "- `bq_dataset`: The BigQuery dataset to train on.\n",
        "- `bq_orders_table` : The BigQuery table of raw data.\n",
@@ -1266,7 +1319,7 @@
      "source": [
        "### View BigQuery ML training pipeline results\n",
        "\n",
-        "Finally, you will view the artifact outputs of each task in the pipeline."
+        "Finally, you view the artifact outputs of each task in the pipeline."
      ]
    },
    {
@@ -1342,8 +1395,8 @@
        "print(\"bigquery-ml-arima-evaluate-job\")\n",
        "artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-ml-arima-evaluate-job\")\n",
        "print(\"\\n\\n\")\n",
-        "print(\"get-model-evaluation-metrics\")\n",
-        "artifacts = print_pipeline_output(bqml_pipeline, \"get-model-evaluation-metrics\")\n",
+        "print(\"bigquery-evaluate-model-job\")\n",
+        "artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-evaluate-model-job\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"bigquery-forecast-model-job\")\n",
        "artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-forecast-model-job\")\n",
@@ -1407,7 +1460,8 @@
        "\n",
        "# Remove local resorces\n",
        "! rm -rf {KFP_COMPONENTS_PATH}\n",
-        "! rm -rf {PIPELINES_PATH}"
+        "! rm -rf {PIPELINES_PATH}\n",
+        "! rm -rf {DATA_PATH}"
      ]
    }
  ],
@@ -42,12 +42,12 @@
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/natural_language/cloud_natural_language_pipeline.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/natural_language/cloud_natural_language_pipeline.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
@@ -328,7 +328,7 @@
      },
      "outputs": [],
      "source": [
-        "PROJECT_ID = \"cloud-ml-language-test\"  # @param {type:\"string\"}\n",
+        "PROJECT_ID = \"your-project-id\"  # @param {type:\"string\"}\n",
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
@@ -368,7 +368,7 @@
      "source": [
        "REGION = \"us\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
-        "TRAINING_DATA_LOCATION = \"gs://dougchen-20221130-pipeline-colab-test/data-00001-of-00001.jsonl\"  # @param {type:\"string\"}\n",
+        "TRAINING_DATA_LOCATION = \"gs://your-training-data-location\"  # @param {type:\"string\"}\n",
        "TASK_TYPE = \"CLASSIFICATION\"  # @param [\"CLASSIFICATION\", \"MULTILABEL_CLASSIFICATION\"]"
      ]
    },
@@ -740,7 +740,7 @@
        "\n",
        "This sends a create pipeline job request to Vertex Pipelines. Note that this  task run synchronously and may take a while to complete.\n",
        "\n",
-        "You may view the progress of the job at any time by clicking on the generated links (after \"View Pipeline Job\" in the console output of the cell below). Once the pipeline finishes, you may examine the artifacts produced from this pipeline. See "
+        "You may view the progress of the job at any time by clicking on the generated links (after \"View Pipeline Job\" in the console output of the cell below). Once the pipeline finishes, you may examine the artifacts produced from this pipeline."
      ]
    },
    {
@@ -42,12 +42,12 @@
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
-        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/community-content/ready_to_go_text_classification_pipeline/ready_to_go_text_classification_pipeline.ipynb\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
-        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/community-content/ready_to_go_text_classification_pipeline/ready_to_go_text_classification_pipeline.ipynb\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
@@ -706,7 +706,7 @@
        ")\n",
        "\n",
        "# The GCS directory for keeping staging files for model evaluation.\n",
-        "ROOT_DIR = 'f\"{BASE_OUTPUT_DIR}/root\"'  # @param {type:\"string\"}"
+        "ROOT_DIR = \"'f\\\"{BASE_OUTPUT_DIR}/root\\\"'\"  # @param {type:\"string\"}"
      ]
    },
    {
@@ -30,6 +30,7 @@
 /automl/automl_forecasting_bqml_arima_plus_comparison.ipynb @TheMichaelHu
 /automl/automl_tabular_on_vertex_pipelines.ipynb @helinwang
 /custom/custom_training_tensorboard_profiler.ipynb @itseric
+/custom/get_started_with_vertex_endpoint_and_shared_vm.ipynb @andrewferlitsch @xingziye
 /workbench/spark/spark_sample_notebook.ipynb @bradmiro
 /workbench/spark/spark_ml.ipynb @bradmiro
 /model_registry/bqml_vertexai_model_registry.ipynb @soheilazangeneh
@@ -352,7 +352,10 @@
      },
      "outputs": [],
      "source": [
-        "EMAIL = \"[your-email-address]\"  # @param {type: \"string\"}"
+        "EMAIL = \"[your-email-address]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    EMAIL = \"noreply@google.com\""
      ]
    },
    {
@@ -31,7 +31,7 @@
        "id": "fsv4jGuU89rX"
      },
      "source": [
-        "# Vertex AI Model Monitoring for AutoML tabular models\n",
+        "# E2E ML on GCP: MLOps stage 7 : monitoring: Vertex AI Model Monitoring for AutoML tabular models\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
@@ -91,6 +91,7 @@
        "- Deploy the `Model` resource to the `Endpoint` resource.\n",
        "- Configure the `Endpoint` resource for model monitoring.\n",
        "- Generate synthetic prediction requests for skew.\n",
+        "- Wait for email alert notification.\n",
        "- Generate synthetic prediction requests for drift.\n",
        "- Wait for email alert notification.\n",
        "\n",
@@ -105,7 +106,7 @@
      "source": [
        "### Dataset\n",
        "\n",
-        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). In this notebook, you use only the fields year, month and day from the dataset to predict the value of mean daily temperature (mean_temp)."
+        "The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of this dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
      ]
    },
    {
@@ -562,7 +563,7 @@
      "source": [
        "### Create BigQuery client\n",
        "\n",
-        "In this tutorial, you explore the monitoring data stored in BigQuery. You create a client interface, which you subsequently use to access the data."
+        "In this tutorial, you use data from the same public BigQuery table that was used to train the pre-trained model. You create a client interface, which you subsequently use to access the data."
      ]
    },
    {
@@ -667,6 +668,23 @@
        "print(dataset.resource_name)"
      ]
    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_transformations:gsod"
+      },
+      "outputs": [],
+      "source": [
+        "TRANSFORMATIONS = [\n",
+        "    {\"auto\": {\"column_name\": \"year\"}},\n",
+        "    {\"auto\": {\"column_name\": \"month\"}},\n",
+        "    {\"auto\": {\"column_name\": \"day\"}},\n",
+        "]\n",
+        "\n",
+        "label_column = \"mean_temp\""
+      ]
+    },
    {
      "cell_type": "markdown",
      "metadata": {
@@ -685,7 +703,7 @@
        "- `optimization_prediction_type`: The type task to train the model for.\n",
        "  - `classification`: A tabuar classification model.\n",
        "  - `regression`: A tabular regression model.\n",
-        "- `column_transformations`: (Optional): Transformations to apply to the input columns. In this example, you set the column transformations to use the default transformation based on their data type.\n",
+        "- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
        "- `optimization_objective`: The optimization objective to minimize or maximize.\n",
        "  - binary classification:\n",
        "    - `minimize-log-loss`\n",
@@ -701,23 +719,6 @@
        "    - `minimize-rmsle`"
      ]
    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "set_transformations:gsod"
-      },
-      "outputs": [],
-      "source": [
-        "TRANSFORMATIONS = [\n",
-        "    {\"auto\": {\"column_name\": \"year\"}},\n",
-        "    {\"auto\": {\"column_name\": \"month\"}},\n",
-        "    {\"auto\": {\"column_name\": \"day\"}},\n",
-        "]\n",
-        "\n",
-        "label_column = \"mean_temp\""
-      ]
-    },
    {
      "cell_type": "code",
      "execution_count": null,
@@ -1064,9 +1065,7 @@
        "You are receiving this mail because you are using the Vertex AI Model Monitoring service.\n",
        "This mail is to inform you that we received your request to set up drift or skew detection for the Prediction Endpoint listed below. Starting from now, incoming prediction requests will be sampled and logged for analysis.\n",
        "Raw requests and responses will be collected from prediction service and saved in bq://[your-project-id].model_deployment_monitoring_[endpoint-id].serving_predict .\n",
-        "</blockquote>\n",
-        "\n",
-        "*Note:* You do not need to wait for the email notification to continue to the next step."
+        "</blockquote>"
      ]
    },
    {
@@ -1131,7 +1130,7 @@
        "\n",
        "Next, you extract the first 1000 instances from the BigQuery training table to use for prediction requests. You modify the data (synthetic) to trigger the skew detection in the prediction requests from the training distribution versus serving distribution, as follows:\n",
        "\n",
-        "- `year`: Set all values to 3."
+        "- `year`: Set all values to 3 (was 2)."
      ]
    },
    {
@@ -1276,7 +1275,7 @@
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
-        "id": "2e64ffaae2de"
+        "id": "b91a0e19ff8b"
      },
      "outputs": [],
      "source": [
@@ -1284,6 +1283,41 @@
        "    time.sleep(60 * 45)"
      ]
    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2b5859ea4ae9"
+      },
+      "source": [
+        "### Logging sampled requests\n",
+        "\n",
+        "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
+        "\n",
+        "Next, you wait for the logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bd177a8decbb"
+      },
+      "outputs": [],
+      "source": [
+        "while True:\n",
+        "    time.sleep(180)\n",
+        "\n",
+        "    ENDPOINT_ID = endpoint.resource_name.split(\"/\")[-1]\n",
+        "\n",
+        "    table = bigquery.TableReference.from_string(\n",
+        "        f\"{PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",
+        "    )\n",
+        "    rows = bqclient.list_rows(table)\n",
+        "    print(rows.total_rows)\n",
+        "    if rows.total_rows > 505:\n",
+        "        break"
+      ]
+    },
    {
      "cell_type": "markdown",
      "metadata": {
@@ -1385,7 +1419,7 @@
        "    )\n",
        "    rows = bqclient.list_rows(table)\n",
        "    print(rows.total_rows)\n",
-        "    if rows.total_rows > 505:\n",
+        "    if rows.total_rows > 1050:\n",
        "        break"
      ]
    },
@@ -0,0 +1,18 @@
+
+[Training, tuning and deploying a PyTorch stable diffusion model on Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/stable_diffusion/dreambooth_stablediffusion.ipynb)
+
+```
+Learn to build, train, tune and deploy a stable diffusion 2.0 model using HuggingFace and PyTorch on
+[Vertex AI](https://cloud.google.com/vertex/).
+
+The steps performed include:
+
+- Fine tune the stable diffusion 2.0 model with your own images.
+- Upload the model artifacts to Google Cloud Storage.
+- Check the created model artifacts.
+- Upload the model in the PyTorch prebuilt container.
+- Create an endpoint on Vertex AI.
+- Deploy the model to the endpoint.
+- Generate a new image.
+
+```
Author	SHA1	Message	Date
Eric Schmidt	6072a3cf99	per linter	2023-02-07 21:08:06 +00:00
Eric Schmidt	9cc7194303	consolidated imports into a single cell	2023-02-07 21:03:27 +00:00
Eric Schmidt	1e8f7a27df	per linter	2023-02-07 20:55:12 +00:00
Eric Schmidt	7fbe8a5355	per reviewer	2023-02-07 20:45:25 +00:00
Eric Schmidt	73e83c5c2b	per reviewer	2023-02-07 20:28:52 +00:00
Eric Schmidt	5b51fc774f	per reviewer	2023-02-07 19:30:26 +00:00
Eric Schmidt	7a10a7274e	per reviewer	2023-02-07 18:20:06 +00:00
Eric Schmidt	c4d85daa97	revisions; deleted deployment-only nb	2023-02-03 23:37:40 +00:00
Eric Schmidt	5d2b3d11d9	revisions	2023-02-03 22:40:55 +00:00
Eric Schmidt	aaf02717dd	added clean up section	2023-02-03 19:23:14 +00:00
Eric Schmidt	cba37ca50d	iter	2023-02-03 19:20:13 +00:00
Eric Schmidt	a0d66836bf	iter	2023-02-02 00:17:23 +00:00
Eric Schmidt	62b37435c2	revisions	2023-02-01 19:36:26 +00:00
Eric Schmidt	5dd8484697	success!	2023-02-01 18:54:43 +00:00
Eric Schmidt	f190fc6281	feat: added README, dreambooth notebook	2023-01-30 23:21:47 +00:00
Eric Schmidt	b3367cd946	feat: stable diffusion 2.0 with PyTorch	2023-01-24 20:35:22 +00:00
Mend RenovateandGitHub	9335ea3591	chore(deps): update dependency flake8 to v6 (#1298 )	2023-01-24 11:31:29 -08:00
Andrew FerlitschandGitHub	0f7343feee	migration: experiments (#1487 ) * migration: experiments * fix: review	2023-01-24 12:27:19 -05:00
Ivan NardiniandGitHub	12cd965ce6	new demand forecasting pipeline notebook (#1439 ) * new demand forecasting pipeline notebook * linter passed * review notebook * linter passed * review notebook * linter passed * andy review * linter passed	2023-01-24 08:44:14 -08:00
Ziye Xing GitHub Andrew Ferlitsch Eric Schmidt	32632711ff	Add co-hosting model notebook (#713 ) * Add notebook for co-hosting model * Add notebook for co-hosting model * Change co-hosting model notebook inline link to officical Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com> Co-authored-by: Eric Schmidt <em.schmidt78@gmail.com>	2023-01-23 10:29:13 -08:00
junkourataandGitHub	d676d87cce	feat: Add E2E notebook featuring Vertex Feature Store, Training and Prediction (#1398 ) * Add E2E tutorial for Feature Store * Add Codeowner and fix the formatting and lining. * Fixed lint	2023-01-23 10:24:51 -08:00
Douglass ChenandGitHub	153a8044b8	Update Colab notebooks' default fields and URLs (#1485 ) * Add Cloud natural language pipeline colab notebook * Add ready-to-go text classification pipeline colab notebook * Ran reformatting scripts on text classification pipeline colab notebooks * Update CODEOWNERS files * Fix order of cells in cloud_natural_language_pipeline.ipynb * Remove unused variables via linter for text classification colabs; fix classification variable for preprocessing component * Minor fix: remove GCPC version requirement * Minor fix: remove outputs * fix formatting with nbfmt * move ready-to-go pipeline to notebooks/community * fix link * update CODEOWNERS * move text classification colabs to notebooks/community/pipelines * Address initial comments on NL notebook * Remove commented lines in NL notebook * minor cell formatting * clear outputs * minor changes to NL notebook * address comments for ready-to-go pipeline * run linter locally * add pipeline description to NL pipeline * run linter locally (PR check could not lint) * Add cell to examine metrics, update kernel restart cell from official template * lint * Update default fields and URLs in NL notebook * Fix URLs in ready to go notebook * run linter	2023-01-23 10:00:25 -08:00
Andrew FerlitschandGitHub	c0d9250416	migration: labeling (#1479 )	2023-01-23 08:58:28 -08:00