Merge branch 'main' into ml_ops_6v6

notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb

@@ -0,0 +1,874 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VBOfRw7ifk8w"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "title:generic,gcp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 2 : AutoML Image Classfication Training with Customer Managed Encryption Keys (CMEK)\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb\">\n",
+        "      Open in Google Cloud Notebooks\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "overview:mlops"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 2 : experimentation: get started with AutoML training with a customer managed encyrption key CMEK."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dataset:flowers,icn"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public #(GCS) bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to use a customer managed encryption key (CMEK) for `Vertex AI AutoML` training.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI AutoML`\n",
+        "- Customer managed encryption key.\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Creating a customer managed encryption key.\n",
+        "- Creating an image dataset with CMEK encryption.\n",
+        "- Train an AutoML model with CMEK encryption."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "install_mlops"
+      },
+      "source": [
+        "## Installations\n",
+        "\n",
+        "Install the Vertex AI SDK and the KMS package for CMEK encryption."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sBfZtR4X1Dr_"
+      },
+      "outputs": [],
+      "source": [
+        "USER_FLAG = \"--user\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
+        "! pip3 install --upgrade google-cloud-kms $USER_FLAG"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "restart"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "restart"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "project_id"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "set_gcloud_project_id"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable, which is used for operations\n",
+        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "timestamp"
+      },
+      "source": [
+        "#### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "timestamp"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:mbsdk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
+        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "setup_vars"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_aip:mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aip\n",
+        "from google.cloud import kms"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mRk9eoTm6Pyi"
+      },
+      "source": [
+        "## Setting up Customer Managed Encryption Keys\n",
+        "\n",
+        "By default, Google Cloud automatically encrypts data when it is stored in Cloud Storage using encryption keys managed by Google. If you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) for your training jobs.\n",
+        "\n",
+        "### Enable KMS API\n",
+        "\n",
+        "First, you enble the [Cloud Key Management Service (KMS)](https://console.cloud.google.com/flows/enableapi?apiid=cloudkms.googleapis.com)\n",
+        "\n",
+        "Learn more about [Customer managed encryption keys (CMEK)](https://cloud.google.com/vertex-ai/docs/general/cmek)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RD_Pvrg584X3"
+      },
+      "source": [
+        "### Create a key ring\n",
+        "\n",
+        "After you have enabled the KMS API, you create a key ring and a key. Use the helper function `create_key_ring()` to create a key ring, with the following parameters:\n",
+        "\n",
+        "- `project_id`: Your project ID.\n",
+        "- `location`: Your region.\n",
+        "- `key_ring_id`: The unique identifier for your key ring.\n",
+        "\n",
+        "The helper function calls the KMS client method `create_key_ring()` to create your key ring.\n",
+        "\n",
+        "Learn more about [KMS: Create a key ring](https://cloud.google.com/kms/docs/samples/kms-create-key-ring)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dxRZzbvQnZC7"
+      },
+      "outputs": [],
+      "source": [
+        "KEY_RING_ID = \"your_cmek_key_ring_id\"\n",
+        "\n",
+        "\n",
+        "def create_key_ring(project_id, location, key_ring_id):\n",
+        "    \"\"\"\n",
+        "    Creates a new key ring in Cloud KMS\n",
+        "\n",
+        "    Args:\n",
+        "        project_id (string): Google Cloud project ID (e.g. 'my-project').\n",
+        "        location (string): Cloud KMS location (e.g. 'us-east1').\n",
+        "        id (string): ID of the key ring to create (e.g. 'my-key-ring').\n",
+        "\n",
+        "    Returns:\n",
+        "        KeyRing: Cloud KMS key ring.\n",
+        "\n",
+        "    \"\"\"\n",
+        "\n",
+        "    # Create the client.\n",
+        "    client = kms.KeyManagementServiceClient()\n",
+        "\n",
+        "    # Build the parent location name.\n",
+        "    location_name = f\"projects/{project_id}/locations/{location}\"\n",
+        "\n",
+        "    # Build the key ring.\n",
+        "    key_ring = {}\n",
+        "\n",
+        "    # Call the API.\n",
+        "    created_key_ring = client.create_key_ring(\n",
+        "        request={\n",
+        "            \"parent\": location_name,\n",
+        "            \"key_ring_id\": key_ring_id,\n",
+        "            \"key_ring\": key_ring,\n",
+        "        }\n",
+        "    )\n",
+        "    print(\"Created key ring: {}\".format(created_key_ring.name))\n",
+        "    return created_key_ring\n",
+        "\n",
+        "\n",
+        "key_ring = create_key_ring(\n",
+        "    project_id=PROJECT_ID, location=REGION, key_ring_id=KEY_RING_ID\n",
+        ")\n",
+        "print(key_ring)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gCL1-IfFtWXl"
+      },
+      "source": [
+        "### Create a key\n",
+        "\n",
+        "Next, you create your key. Use the helper function `create_key()` with the following parameters:\n",
+        "\n",
+        "- `project_id`: Your project ID.\n",
+        "- `location`: Your region.\n",
+        "- `key_ring_id`: The unique identifier for your key ring.\n",
+        "- `key_id`: The unique identifier for your key.\n",
+        "\n",
+        "The helper function calls the KMS client method `create_cryto_key()` to create your key.\n",
+        "\n",
+        "Learn more about [](https://cloud.google.com/kms/docs/samples/kms-create-key-symmetric-encrypt-decrypt)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "LXcagdmSnYYW"
+      },
+      "outputs": [],
+      "source": [
+        "KEY_ID = \"your_cmek_key_id\"\n",
+        "\n",
+        "\n",
+        "def create_key(project_id, location, key_ring_id, key_id):\n",
+        "    \"\"\"\n",
+        "    Creates a new symmetric encryption/decryption key in Cloud KMS.\n",
+        "\n",
+        "    Args:\n",
+        "        project_id (string): Google Cloud project ID (e.g. 'my-project').\n",
+        "        location (string): Cloud KMS location (e.g. 'us-east1').\n",
+        "        key_ring_id (string): ID of the Cloud KMS key ring (e.g. 'my-key-ring').\n",
+        "        key_id (string): ID of the key to create (e.g. 'my-symmetric-key').\n",
+        "\n",
+        "    Returns:\n",
+        "        CryptoKey: Cloud KMS key.\n",
+        "\n",
+        "    \"\"\"\n",
+        "\n",
+        "    # Create the client.\n",
+        "    client = kms.KeyManagementServiceClient()\n",
+        "\n",
+        "    # Build the parent key ring name.\n",
+        "    key_ring_name = client.key_ring_path(project_id, location, key_ring_id)\n",
+        "\n",
+        "    # Build the key.\n",
+        "    purpose = kms.CryptoKey.CryptoKeyPurpose.ENCRYPT_DECRYPT\n",
+        "    algorithm = (\n",
+        "        kms.CryptoKeyVersion.CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION\n",
+        "    )\n",
+        "    key = {\n",
+        "        \"purpose\": purpose,\n",
+        "        \"version_template\": {\n",
+        "            \"algorithm\": algorithm,\n",
+        "        },\n",
+        "    }\n",
+        "\n",
+        "    # Call the API.\n",
+        "    created_key = client.create_crypto_key(\n",
+        "        request={\"parent\": key_ring_name, \"crypto_key_id\": key_id, \"crypto_key\": key}\n",
+        "    )\n",
+        "    print(\"Created symmetric key: {}\".format(created_key.name))\n",
+        "    return created_key\n",
+        "\n",
+        "\n",
+        "key_id = create_key(\n",
+        "    project_id=PROJECT_ID, location=REGION, key_ring_id=KEY_RING_ID, key_id=KEY_ID\n",
+        ")\n",
+        "\n",
+        "print(key_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3gKDBOqC8Gl5"
+      },
+      "source": [
+        "### Set service account permissions\n",
+        "\n",
+        "Next, you set permissions for your Vertex AI service account to encrypt and decrypt resources using your key.\n",
+        "\n",
+        "Learn more about [Grant Vertex AI permissions](https://cloud.google.com/vertex-ai/docs/general/cmek#grant_permissions)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6QrRg08Vqfru"
+      },
+      "outputs": [],
+      "source": [
+        "# Reference: https://cloud.google.com/vertex-ai/docs/general/cmek#granting_permissions\n",
+        "# Get the service account\n",
+        "SERVICE_ACCOUNT = ! gcloud projects get-iam-policy {PROJECT_ID} \\\n",
+        "  --flatten=\"bindings[].members\" \\\n",
+        "  --format=\"table(bindings.members)\" \\\n",
+        "  --filter=\"bindings.role:roles/aiplatform.serviceAgent\" \\\n",
+        "  | grep -oP \"service-.+?@gcp-sa-aiplatform.iam.gserviceaccount.com\"\n",
+        "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
+        "\n",
+        "print(f\"Service account is: {SERVICE_ACCOUNT}\")\n",
+        "\n",
+        "# Give permissions\n",
+        "! gcloud kms keys add-iam-policy-binding {KEY_ID} \\\n",
+        "  --keyring={KEY_RING_ID} \\\n",
+        "  --location={REGION} \\\n",
+        "  --project={PROJECT_ID} \\\n",
+        "  --member=serviceAccount:{SERVICE_ACCOUNT} \\\n",
+        "  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1e8cd37e5f99"
+      },
+      "source": [
+        "Create the full resource identifier for the created key"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ebAHZg2vlhXL"
+      },
+      "outputs": [],
+      "source": [
+        "ENCRYPTION_SPEC_KEY_NAME = key_id.name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Aa_8wrqSkamz"
+      },
+      "source": [
+        "## Initialize Vertex SDK for Python\n",
+        "\n",
+        "Initialize the *client* for Vertex AI\n",
+        "\n",
+        "All resources created during this Notebook run will encrypted with the encryption key created above.\n",
+        "\n",
+        "You can override the encryption key at each function call."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project, bucket, and corresponding encryption key.\n",
+        "\n",
+        "All resources created during this session are encrypted with the encryption key you created.\n",
+        "\n",
+        "*Note:* You can override the encryption key at each function call."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ohdgOs69kGNU"
+      },
+      "outputs": [],
+      "source": [
+        "aip.init(\n",
+        "    project=PROJECT_ID,\n",
+        "    staging_bucket=BUCKET_NAME,\n",
+        "    location=REGION,\n",
+        "    encryption_spec_key_name=ENCRYPTION_SPEC_KEY_NAME,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "import_file:u_dataset,csv"
+      },
+      "source": [
+        "#### Location of Cloud Storage training data.\n",
+        "\n",
+        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "import_file:flowers,csv,icn"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE = (\n",
+        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\"\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "35QVNhACqcTJ"
+      },
+      "source": [
+        "# Create `Vertex AI ImageDataset` resource\n",
+        "\n",
+        "Next, you create an `ImageDataset` resource, which will be encrypted using your encryption key."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4OfCqaYRqcTJ"
+      },
+      "outputs": [],
+      "source": [
+        "dataset = aip.ImageDataset.create(\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    gcs_source=[IMPORT_FILE],\n",
+        "    import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
+        ")\n",
+        "\n",
+        "print(dataset.resource_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6-bBqipfqcTS"
+      },
+      "source": [
+        "# Launch a Training Job to Create a Model\n",
+        "\n",
+        "Train an AutoML Image Classification model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aA41rT_mb-rV"
+      },
+      "outputs": [],
+      "source": [
+        "job = aiplatform.AutoMLImageTrainingJob(\n",
+        "    display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    prediction_type=\"classification\",\n",
+        "    multi_label=False,\n",
+        "    model_type=\"CLOUD\",\n",
+        "    base_model=None,\n",
+        ")\n",
+        "\n",
+        "# This will take around half an hour to run\n",
+        "model = job.run(\n",
+        "    dataset=ds,\n",
+        "    model_display_name=\"flowers_\" + TIMESTAMP,\n",
+        "    training_fraction_split=0.6,\n",
+        "    validation_fraction_split=0.2,\n",
+        "    test_fraction_split=0.2,\n",
+        "    budget_milli_node_hours=8000,\n",
+        "    disable_early_stopping=False,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5vhDsMJNqcTW"
+      },
+      "source": [
+        "# Deploy Your Model\n",
+        "\n",
+        "Deploy your model, then wait until the model FINISHES deployment before proceeding to prediction."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Y9GH72wWqcTX"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = model.deploy()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nIw1ifPuqcTb"
+      },
+      "source": [
+        "# Predict on Endpoint\n",
+        "- Take one sample from the data imported to the dataset\n",
+        "- This sample will be encoded to base64 and passed to the endpoint for prediction"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "H23ISHdHVIZM"
+      },
+      "outputs": [],
+      "source": [
+        "test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
+        "test_item, test_label = str(test_item[0]).split(\",\")\n",
+        "\n",
+        "print(test_item, test_label)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TF_N0kqZU768"
+      },
+      "outputs": [],
+      "source": [
+        "import base64\n",
+        "\n",
+        "import tensorflow as tf\n",
+        "\n",
+        "with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
+        "    content = f.read()\n",
+        "\n",
+        "# The format of each instance should conform to the deployed model's prediction input schema.\n",
+        "instances_list = [{\"content\": base64.b64encode(content).decode(\"utf-8\")}]\n",
+        "\n",
+        "prediction = endpoint.predict(instances=instances_list)\n",
+        "\n",
+        "print(prediction)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nWA3qocXfk82"
+      },
+      "source": [
+        "# Undeploy Model from Endpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "V1brMaO_fk82"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.undeploy_all()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "e00750837ca8"
+      },
+      "outputs": [],
+      "source": [
+        "# missing\n",
+        "endpoint.delete()\n",
+        "model.delete()\n",
+        "dataset.delete()\n",
+        "\n",
+        "! gcloud kms keys versions destroy key-version \\\n",
+        "  --key key {KEY_ID} \\\n",
+        "  --keyring={KEY_RING_ID} \\\n",
+        "  --location={REGION} "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aa95b7fff9b5"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud kms keys list --location {REGION} --keyring {KEY_RING_ID}"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "get_started_with_cmek_training.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}