feat: add metadata notebook

notebooks/community/ml_ops/stage3/get_started_with_bqml_pipeline_components.ipynb

@@ -700,7 +700,7 @@
         "    bq_model = gcc_bq.BigqueryCreateModelJobOp(\n",
         "        project=project,\n",
         "        location=location,\n",
-        "        query=f\"CREATE OR REPLACE MODEL {dataset}.{model} OPTIONS (model_type='dnn_classifier', labels=['{label}'], min_trials={min_trials}) AS SELECT * FROM `{bq_table}` WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL\",\n",
+        "        query=f\"CREATE OR REPLACE MODEL {dataset}.{model} OPTIONS (model_type='dnn_classifier', labels=['{label}'], num_trials={min_trials}) AS SELECT * FROM `{bq_table}` WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL\",\n",
         "    ).after(bq_dataset)\n",
         "\n",
         "    # bq_eval = gcc_bq.BigqueryEvaluateModelJobOp(\n",
@@ -723,8 +723,8 @@
         "        # query_statement=f\"SELECT * EXCEPT ({label}) FROM {bq_table} WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL\"\n",
         "        job_configuration_query={\n",
         "            \"destinationTable\": {\n",
-        "                \"projectId\": f\"`{project}`\",\n",
-        "                \"datasetId\": f\"{dataset}\",\n",
+        "                \"projectId\": PROJECT_ID,\n",
+        "                \"datasetId\": \"bqml_tutorial\",\n",
         "                \"tableId\": \"results_1\",\n",
         "            }\n",
         "        },\n",
@@ -1077,7 +1077,7 @@
         "    job = bqclient.delete_model(\"bqml_tutorial.penguins_model\")\n",
         "except:\n",
         "    pass\n",
-        "job = bqclient.delete_dataset(\"bqml_tutorial\")"
+        "job = bqclient.delete_dataset(\"bqml_tutorial\", delete_contents=True)"
       ]
     },
     {

notebooks/community/ml_ops/stage4/get_started_with_google_artifact_registry.ipynb

@@ -0,0 +1,482 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 4 : formalization: get started with Google Artifact Registry\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage4/get_started_with_google_artifact_registry.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage4/get_started_with_google_artifact_registry.ipynb\">\n",
+        "      Open in Google Cloud Notebooks\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 4 : formalization: get started with Google Artifact Registry."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage4,get_started_google_artifact_registry"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use `Google Artifact Registry`.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Google Artifact Registry`\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Creating a private Dockerrepository.\n",
+        "- Tagging a container image, specific to the private Dockerrepository.\n",
+        "- Pushing a container image to the private Dockerrepository.\n",
+        "- Pulling a container image from the private Dockerrepository.\n",
+        "- Deleting a private Dockerrepository."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "source": [
+        "## Installations\n",
+        "\n",
+        "Install *one time* the packages for executing the MLOps notebooks."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "outputs": [],
+      "source": [
+        "ONCE_ONLY = False\n",
+        "if ONCE_ONLY:\n",
+        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
+        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
+        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
+        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
+        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
+        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
+        "    ! pip3 install --upgrade kfp $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "project_id"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "intro_gar"
+      },
+      "source": [
+        "## Introduction to Google Artifact Registry\n",
+        "\n",
+        "The `Google Artifact Registry` is a service for storing and managing artifacts in private repositories, including container images, Helm charts, and language packages. It is the recommended container image registry for Google Cloud.\n",
+        "\n",
+        "Learn more about [Quick start for Docker](https://cloud.google.com/artifact-registry/docs/docker/quickstart)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_enable_api"
+      },
+      "source": [
+        "### Enable Artifact Registry API\n",
+        "\n",
+        "First, you must enable the Artifact Registry API service for your project.\n",
+        "\n",
+        "Learn more about [Enabling service](https://cloud.google.com/artifact-registry/docs/enable-service)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_enable_api"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud services enable artifactregistry.googleapis.com"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_create_repo"
+      },
+      "source": [
+        "## Create a private Docker repository\n",
+        "\n",
+        "Your first step is to create your own Docker repository in Google Artifact Registry.\n",
+        "\n",
+        "1. Run the `gcloud artifacts repositories create` command to create a new Docker repository with your region with the description \"docker repository\".\n",
+        "\n",
+        "2. Run the `gcloud artifacts repositories list` command to verify that your repository was created."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_create_repo"
+      },
+      "outputs": [],
+      "source": [
+        "PRIVATE_REPO = \"my-docker-repo\"\n",
+        "\n",
+        "! gcloud artifacts repositories create {PRIVATE_REPO} --repository-format=docker --location={REGION} --description=\"Docker repository\"\n",
+        "\n",
+        "! gcloud artifacts repositories list"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_auth"
+      },
+      "source": [
+        "### Configure authentication to your private repo\n",
+        "\n",
+        "Before you push or pull container images, configure Docker to use the `gcloud` command-line tool to authenticate requests to `Artifact Registry` for your region."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_auth"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud auth configure-docker {REGION}-docker.pkg.dev --quiet"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_get_example"
+      },
+      "source": [
+        "### Obtain an example container image\n",
+        "\n",
+        "For demonstration purposes, you obtain (pull) a local copy of our demonstration container image: `hello-app:1.0`"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_get_example"
+      },
+      "outputs": [],
+      "source": [
+        "! docker pull us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_tag_image"
+      },
+      "source": [
+        "## Tagging your container image\n",
+        "\n",
+        "Now that you have your own container image, the first step is to tag your image.\n",
+        "\n",
+        "- Tagging the Docker image with a repository name configures the docker push command to push the image to a specific location, e.g., us-central1-docker.pkg.dev.\n",
+        "\n",
+        "- `:my-tag` is a tag you're adding to the Docker image. If a tag is not specified, it defaults to `:latest`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_tag_image"
+      },
+      "outputs": [],
+      "source": [
+        "CONTAINER_NAME = \"my-image:my-tag\"\n",
+        "\n",
+        "! docker tag us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0 us-central1-docker.pkg.dev/{PROJECT_ID}/{PRIVATE_REPO}/{CONTAINER_NAME}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_push_image"
+      },
+      "source": [
+        "## Push your image to your private Docker repository\n",
+        "\n",
+        "Next, push your container to your private Docker repository."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_push_image"
+      },
+      "outputs": [],
+      "source": [
+        "! docker push {REGION}-docker.pkg.dev/{PROJECT_ID}/{PRIVATE_REPO}/{CONTAINER_NAME}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_pull_image"
+      },
+      "source": [
+        "## Pull your image from your private Docker repostory\n",
+        "\n",
+        "Now pull your container from your private Docker repository."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_pull_image"
+      },
+      "outputs": [],
+      "source": [
+        "! docker pull {REGION}-docker.pkg.dev/{PROJECT_ID}/{PRIVATE_REPO}/{CONTAINER_NAME}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gar_delete_repo"
+      },
+      "source": [
+        "### Deleting your private Docker repostory\n",
+        "\n",
+        "Finally, once your private repository becomes obsolete, use the command `gcloud artifacts repositories delete` to delete it `Google Artifact Registry`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gar_delete_repo"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud artifacts repositories delete {PRIVATE_REPO} --location={REGION} --quiet"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "get_started_with_google_artifact_registry.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}