fix: install issue

fix: reorg

notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb

@@ -0,0 +1,865 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ur8xi4C7S06n"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2023 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JAPoU8Sm5E6e"
+      },
+      "source": [
+        "# Get started with importing a custom model evaluation to the Vertex AI Model Registry\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/get_started_with_custom_model_evaluation_import.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>                                                                                               \n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "24743cf4a1e1"
+      },
+      "source": [
+        "**_NOTE_**: This notebook has been tested in the following environment:\n",
+        "\n",
+        "* Python version = 3.9"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tvgnzT1CKxrO"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This tutorial shows how to use Vertex AI Model Evaluation to import a custom model evaluation to an existing Vertex AI Model Registry entry."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d975e698c9a4"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn how to construct and upload a custom model evaluation, and upload the custom model evaluation to a Model resource entry in Vertex AI Model Registry.\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services and resources:\n",
+        "\n",
+        "- Vertex AI Model Evaluation\n",
+        "- Vertex AI Model Registry\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Import a pretrained (blessed) model to the Vertex AI Model Registry.\n",
+        "- Construct a custom model evaluation.\n",
+        "- Import the model evaluation metrics to the corresponding model in the Vertex AI Model Registry.\n",
+        "- List the model evaluation for the corresponding model in the Vertex AI Model Registry.\n",
+        "- Construct a second custom model evaluation.\n",
+        "- Import the second model evaluation metrics to the corresponding model in the Vertex AI Model Registry.\n",
+        "- List the second model evaluation for the corresponding model in the Vertex AI Model Registry.\n",
+        "\n",
+        "Learn more about [Model Evaluation in Vertex AI](https://cloud.google.com/vertex-ai/docs/evaluation/introduction)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "08d289fa873f"
+      },
+      "source": [
+        "### Model\n",
+        "\n",
+        "This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.\n",
+        "\n",
+        "Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "aed92deeb4a0"
+      },
+      "source": [
+        "### Costs \n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
+        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
+        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "i7EUnXsZhAGF"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the following packages required to execute this notebook. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2b4ef9b72d43"
+      },
+      "outputs": [],
+      "source": [
+        "# Install the packages\n",
+        "USER=''\n",
+        "! pip3 install {USER} --upgrade google-cloud-aiplatform \\\n",
+        "                                tensorflow==2.5 \\\n",
+        "                                tensorflow-hub"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "58707a750154"
+      },
+      "source": [
+        "### Colab only: Uncomment the following cell to restart the kernel."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f200f10a1da3"
+      },
+      "outputs": [],
+      "source": [
+        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
+        "# import IPython\n",
+        "\n",
+        "# app = IPython.Application.instance()\n",
+        "# app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BF1j6f9HApxa"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
+        "\n",
+        "3. [Enable the Vertex AI API]\n",
+        "\n",
+        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WReHDGG5g0XY"
+      },
+      "source": [
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oM1iC_MfAts1"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "\n",
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "region"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "region"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "sBCra4QMA2wR"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "74ccc9e52986"
+      },
+      "source": [
+        "**1. Vertex AI Workbench**\n",
+        "* Do nothing as you are already authenticated."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "de775a3773ba"
+      },
+      "source": [
+        "**2. Local JupyterLab instance, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "254614fa0c46"
+      },
+      "outputs": [],
+      "source": [
+        "# ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ef21552ccea8"
+      },
+      "source": [
+        "**3. Colab, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "603adbbf0532"
+      },
+      "outputs": [],
+      "source": [
+        "# from google.colab import auth\n",
+        "# auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f6b2ccc891ed"
+      },
+      "source": [
+        "**4. Service account or other**\n",
+        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zgPO1eR3CYjk"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "Create a storage bucket to store intermediate artifacts such as datasets.\n",
+        "\n",
+        "- *{Note to notebook author: For any user-provided strings that need to be unique (like bucket names or model ID's), append \"-unique\" to the end so proper testing can occur}*"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MzGDU7TWdts_"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_URI = f\"gs://your-bucket-name-unique-{PROJECT_ID}\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-EcIXiGsCePi"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NIq7R4HZCfIc"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "960505627ddf"
+      },
+      "source": [
+        "### Import libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PyQmSRbKA8r-"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import tensorflow_hub as hub\n",
+        "from google.cloud import aiplatform\n",
+        "from google.cloud.aiplatform import gapic"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk,all"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
+      },
+      "source": [
+        "#### Set hardware accelerators\n",
+        "\n",
+        "You can set hardware accelerators for prediction.\n",
+        "\n",
+        "Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
+        "\n",
+        "    (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
+        "\n",
+        "\n",
+        "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
+        "\n",
+        "Learn more about [hardware accelerator support for your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators), and [GPU pricing](https://cloud.google.com/compute/gpus-pricing)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "accelerators:training,cpu,prediction,cpu,mbsdk"
+      },
+      "outputs": [],
+      "source": [
+        "DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "container:training,prediction"
+      },
+      "source": [
+        "#### Set pre-built containers\n",
+        "\n",
+        "Set the pre-built Docker container image for prediction.\n",
+        "\n",
+        "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "container:training,prediction"
+      },
+      "outputs": [],
+      "source": [
+        "TF = \"2.5\".replace(\".\", \"-\")\n",
+        "\n",
+        "if DEPLOY_GPU:\n",
+        "    DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n",
+        "else:\n",
+        "    DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n",
+        "\n",
+        "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
+        "    REGION.split(\"-\")[0], DEPLOY_VERSION\n",
+        ")\n",
+        "\n",
+        "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d8128b8ff025"
+      },
+      "source": [
+        "## Get pretrained model from TensorFlow Hub\n",
+        "\n",
+        "For demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a `Vertex AI Model` resource. Once you have a `Vertex AI Model` resource, the model can be deployed to a `Vertex AI Endpoint` resource.\n",
+        "\n",
+        "### Download the pretrained model\n",
+        "\n",
+        "First, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c55fa4c826f7"
+      },
+      "outputs": [],
+      "source": [
+        "tfhub_model = tf.keras.Sequential(\n",
+        "    [hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5\")]\n",
+        ")\n",
+        "\n",
+        "tfhub_model.build([None, 32, 32, 3])\n",
+        "\n",
+        "tfhub_model.summary()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "64618c713db9"
+      },
+      "outputs": [],
+      "source": [
+        "MODEL_DIR = BUCKET_URI + \"/model\"\n",
+        "tfhub_model.save(MODEL_DIR)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e8ce91147c93"
+      },
+      "source": [
+        "### Upload the TensorFlow Hub model to a `Vertex AI Model` resource\n",
+        "\n",
+        "Finally, you upload the model artifacts from the TFHub model into a `Vertex AI Model` resource using the method `upload()`, with the following parameters:\n",
+        "\n",
+        "- `display_name`: A human readable name for the `Model` resource.\n",
+        "- `artifact_uri`: The Cloud Storage location of the model package.\n",
+        "- `serving_container_image_uri`: The serving container image.\n",
+        "\n",
+        "Uploading a model into a Vertex AI Model resource returns a long running operation, since it may take a few moments. \n",
+        "\n",
+        "*Note:* When you upload the model artifacts to a `Vertex AI Model` resource, you specify the corresponding deployment container image."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ad61e1429512"
+      },
+      "outputs": [],
+      "source": [
+        "model = aiplatform.Model.upload(\n",
+        "    display_name=\"resnet\",\n",
+        "    artifact_uri=MODEL_DIR,\n",
+        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
+        "    is_default_version=True,\n",
+        "    version_aliases=[\"v1\"],\n",
+        ")\n",
+        "\n",
+        "print(model)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "c11e98ef5391"
+      },
+      "source": [
+        "## Introduction to custom model evaluations\n",
+        "\n",
+        "When training a custom model, one generally performs some form of an evaluation of the trained model. Your custom model evaluation can then be imported to the corresponding model in the Vertex AI Model Registry using the `import_model_evaluation()` method. Once imported, the custom model evaluation can be subsequently retreived with the `list_model_evaluations()` method. \n",
+        "\n",
+        "The Vertex AI Model Registry supports importing multiple model evaluations for a model where each evaluation is distinquished by a unique `display_name`.\n",
+        "\n",
+        "\n",
+        "### Create a model evaluation\n",
+        "\n",
+        "First, you create a model evaluation in a format that corresponds to one of the predefined schemas for model evaluations. In this example, you use the schema for a classification metric, and specify the following subset of evaluation metrics as a dictionary:\n",
+        "\n",
+        "- `logLoss`: The log loss.\n",
+        "- `auPrc`: The accuracy.\n",
+        "\n",
+        "You then construct the `ModelEvaluation` object with the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the evaluation metric.\n",
+        "- `metrics_schema_uri`: The schema for the specific type of evaluation metrics.\n",
+        "- `metrics`: The dictionary with the evaluation metrics.\n",
+        "\n",
+        "Learn more about [Schemas for evaluation metrics](https://cloud.google.com/vertex-ai/docs/evaluation/introduction#features)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "e9af222db292"
+      },
+      "outputs": [],
+      "source": [
+        "metrics = {\"logLoss\": 1.4, \"auPrc\": 0.85}\n",
+        "print(metrics)\n",
+        "\n",
+        "model_eval = gapic.ModelEvaluation(\n",
+        "    display_name=\"eval\",\n",
+        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
+        "    metrics=metrics,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "68870bd8194d"
+      },
+      "source": [
+        "### Upload the evaluation metrics to the Model Registry\n",
+        "\n",
+        "Next, upload the model's evaluation from the custom training job to the corresponding entry in the Vertex AI Model Registry.\n",
+        "\n",
+        "Currently, there is not yet support for this method in the SDK. Instead, you use the lower level GAPIC API interface."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3044788848af"
+      },
+      "outputs": [],
+      "source": [
+        "API_ENDPOINT = f\"{REGION}-aiplatform.googleapis.com\"\n",
+        "client = gapic.ModelServiceClient(client_options={\"api_endpoint\": API_ENDPOINT})\n",
+        "\n",
+        "client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_eval)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "337319a19efe"
+      },
+      "source": [
+        "### List the custom model evaluation\n",
+        "\n",
+        "Now that your custom metric has been uploaded to the corresponding model in the Vertex AI Model Registry, you can retriv"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1b3374e988cf"
+      },
+      "outputs": [],
+      "source": [
+        "evaluation = model.list_model_evaluations()[0]\n",
+        "print(evaluation.gca_resource)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "68870bd8194d"
+      },
+      "source": [
+        "### Upload a second evaluation metrics to the Model Registry\n",
+        "\n",
+        "Next, upload a second model evaluation to the corresponding entry in the Vertex AI Model Registry. In this example, we refer to first evaluation metric as `eval` (from training) and the second as `prod` (from production data)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9b864c0ba07f"
+      },
+      "outputs": [],
+      "source": [
+        "metrics = {\"logLoss\": 1.2, \"auPrc\": 0.87}\n",
+        "print(metrics)\n",
+        "\n",
+        "model_prod = gapic.ModelEvaluation(\n",
+        "    display_name=\"prod\",\n",
+        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
+        "    metrics=metrics,\n",
+        ")\n",
+        "\n",
+        "client.import_model_evaluation(parent=model.resource_name, model_evaluation=model_prod)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "337319a19efe"
+      },
+      "source": [
+        "### List a specific custom model evaluation\n",
+        "\n",
+        "Now that your custom second metric has been uploaded to the corresponding model in the Vertex AI Model Registry, you can retrieve this specific evaluation by filtering using the `display_name`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1b3374e988cf"
+      },
+      "outputs": [],
+      "source": [
+        "evaluations = model.list_model_evaluations()\n",
+        "for evaluation in evaluations:\n",
+        "    if evaluation.display_name == \"prod\":\n",
+        "        print(evaluation.gca_resource)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b3a3993dc813"
+      },
+      "source": [
+        "### Upload version 2 of the TFHub model to the `Vertex AI Model Registry`\n",
+        "\n",
+        "Next, you upload the second version of the TFHub model as a `Model` resource in the `Vertex AI Model Registry`, with the additional following parameters:\n",
+        "\n",
+        "- `parent_model`: The existing `Model` resource for which to add this model as the next model version.\n",
+        "- `is_default_version`: Whether this will be the default version for the `Model` resource. In this example, you change from the default from the first version to the second version of the model.\n",
+        "- `version_ailiases`: User defined list of alternative alias names for the model version, such as `production`.\n",
+        "- `version_description`: User description of the model version.\n",
+        "\n",
+        "When a subsequent model version is created in the `Vertex AI Model Registry`, the property `version_id` will automatically be incremented. In this example, it will be set to 2 (2nd version)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "852be5e5f964"
+      },
+      "outputs": [],
+      "source": [
+        "model_v2 = aiplatform.Model.upload(\n",
+        "    display_name=\"resnet\",\n",
+        "    artifact_uri=MODEL_DIR,\n",
+        "    serving_container_image_uri=DEPLOY_IMAGE,\n",
+        "    parent_model=model.resource_name,\n",
+        "    is_default_version=True,\n",
+        "    version_aliases=[\"v2\"],\n",
+        "    version_description=\"This is the second version of the model\",\n",
+        ")\n",
+        "\n",
+        "print(model_v2)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "68870bd8194d"
+      },
+      "source": [
+        "### Upload an evaluation metrics for version 2 of the model to the Model Registry\n",
+        "\n",
+        "Next, upload a model evaluation to the corresponding model version in the Vertex AI Model Registry. Note, you referenced `model_v2.resource_name` to refer to version 2 of this model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9b864c0ba07f"
+      },
+      "outputs": [],
+      "source": [
+        "metrics = {\"logLoss\": 1.0, \"auPrc\": 0.91}\n",
+        "print(metrics)\n",
+        "\n",
+        "model_eval = gapic.ModelEvaluation(\n",
+        "    display_name=\"eval\",\n",
+        "    metrics_schema_uri=\"gs://google-cloud-aiplatform/schema/modelevaluation/classification_metrics_1.0.0.yaml\",\n",
+        "    metrics=metrics,\n",
+        ")\n",
+        "\n",
+        "client.import_model_evaluation(\n",
+        "    parent=model_v2.resource_name, model_evaluation=model_eval\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d4e72b21b06b"
+      },
+      "source": [
+        "### List the evaluations for both versions of the model\n",
+        "\n",
+        "Finally, list the number of evaluations for both versions of the model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "369a23d392a4"
+      },
+      "outputs": [],
+      "source": [
+        "evaluations = model.list_model_evaluations()\n",
+        "print(\"Model v1 no. of evaluations\", len(evaluations))\n",
+        "evaluations = model_v2.list_model_evaluations()\n",
+        "print(\"Model v2 no. of evaluations\", len(evaluations))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TpV-iwP9qw9c"
+      },
+      "source": [
+        "## Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial:\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "sx_vKniMq9ZX"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# Delete model resource\n",
+        "model.delete()\n",
+        "\n",
+        "# Delete Cloud Storage objects that were created\n",
+        "delete_bucket = False\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil -m rm -r $BUCKET_URI"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "get_started_with_custom_model_evaluation_import.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}