fix: review comments

notebooks/official/generative_ai/tune_peft.ipynb

@@ -0,0 +1,744 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "9f0d0f32-23b4-41a6-b364-579da297c326",
+      "metadata": {
+        "id": "9f0d0f32-23b4-41a6-b364-579da297c326"
+      },
+      "outputs": [],
+      "source": [
+        "# @title Copyright & License (click to expand)\n",
+        "# Copyright 2023 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dd53d60c-97eb-4c72-91ea-f274a753ab34",
+      "metadata": {
+        "id": "dd53d60c-97eb-4c72-91ea-f274a753ab34"
+      },
+      "source": [
+        "# Vertex AI Tuning a PEFT model\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/tune_peft.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/tune_peft.ipynb\">\n",
+        "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "    <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/generative_ai/tune_peft.ipynb\">\n",
+        "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "9ef820fb-1203-4cab-965f-17093a4ba25e",
+      "metadata": {
+        "id": "9ef820fb-1203-4cab-965f-17093a4ba25e"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This tutorial demonstrates how to use Vertex AI to tune a PEFT large-language model (LLM) and make a prediction. This workflow improves a model's accuracy by fine-tuning a base model with a training dataset."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "74b00940-376c-4056-90fb-d22c1ce6eedf",
+      "metadata": {
+        "id": "74b00940-376c-4056-90fb-d22c1ce6eedf"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn to use `Vertex AI LLM` to tune and deploy a PEFT large language model.\n",
+        "\n",
+        "\n",
+        "This tutorial uses the following Google Cloud ML services:\n",
+        "\n",
+        "- `Vertex AI LLM`\n",
+        "- `Vertex AI Model Garden`\n",
+        "- `Vertex AI Prediction`\n",
+        "\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "- Get the Vertex AI LLM model.\n",
+        "- Tune the model.\n",
+        "  - This will automatically create a Vertex AI endpoint and deploy the model to it.\n",
+        "- Make a prediction using `Vertex AI LLM`.\n",
+        "- Make a prediction using `Vertex AI Prediction`"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "c6b43693-b20a-41bd-b5b8-5ad414517162",
+      "metadata": {
+        "id": "c6b43693-b20a-41bd-b5b8-5ad414517162"
+      },
+      "source": [
+        "### Model\n",
+        "\n",
+        "The pre-trained LLM model is a BISON (Decoder only) model for text generation."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6d7b5435-e947-49bb-9ce3-aa8a42c30118",
+      "metadata": {
+        "id": "6d7b5435-e947-49bb-9ce3-aa8a42c30118"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0cbf01f0-5f6e-4bcd-903f-84ccaad5332c",
+      "metadata": {
+        "id": "0cbf01f0-5f6e-4bcd-903f-84ccaad5332c"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the following packages required to execute this notebook."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "d1f49723-9cbc-4fac-8e6d-48f33db68255",
+      "metadata": {
+        "id": "d1f49723-9cbc-4fac-8e6d-48f33db68255"
+      },
+      "outputs": [],
+      "source": [
+        "! pip3 install --upgrade --quiet google-cloud-aiplatform  \"shapely<2.0.0\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ff7f7e74-60d7-4032-b335-216564f43e35",
+      "metadata": {
+        "id": "ff7f7e74-60d7-4032-b335-216564f43e35"
+      },
+      "source": [
+        "### Colab only: Uncomment the following cell to restart the kernel"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5c10aa63-17a3-4162-a397-fe570a962cb3",
+      "metadata": {
+        "id": "5c10aa63-17a3-4162-a397-fe570a962cb3"
+      },
+      "outputs": [],
+      "source": [
+        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
+        "# import IPython\n",
+        "\n",
+        "# app = IPython.Application.instance()\n",
+        "# app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b37d4259-7e39-417b-8879-24f7575732c8",
+      "metadata": {
+        "id": "b37d4259-7e39-417b-8879-24f7575732c8"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "caaf0d7e-c6cb-4e56-af5c-553db5180e00",
+      "metadata": {
+        "id": "caaf0d7e-c6cb-4e56-af5c-553db5180e00"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "\n",
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "054d794d-cd2e-4280-95ac-859b264ea2d6",
+      "metadata": {
+        "id": "054d794d-cd2e-4280-95ac-859b264ea2d6"
+      },
+      "source": [
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "0121bf60-1acd-4272-afaf-aa54b4ded263",
+      "metadata": {
+        "id": "0121bf60-1acd-4272-afaf-aa54b4ded263"
+      },
+      "outputs": [],
+      "source": [
+        "REGION = \"us-central1\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "eac9e842-d225-4876-836f-afdb1937d800",
+      "metadata": {
+        "id": "eac9e842-d225-4876-836f-afdb1937d800"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
+        "\n",
+        "**1. Vertex AI Workbench**\n",
+        "* Do nothing as you are already authenticated.\n",
+        "\n",
+        "**2. Local JupyterLab instance, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "23082eec-b1bd-4594-b5b5-56fe2b74db6f",
+      "metadata": {
+        "id": "23082eec-b1bd-4594-b5b5-56fe2b74db6f"
+      },
+      "outputs": [],
+      "source": [
+        "# ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "3c20f923-3c46-4d6d-80d2-d7cb22b1a8da",
+      "metadata": {
+        "id": "3c20f923-3c46-4d6d-80d2-d7cb22b1a8da"
+      },
+      "source": [
+        "**3. Colab, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "60302a3f-fad9-452c-8998-a9c9822d2732",
+      "metadata": {
+        "id": "60302a3f-fad9-452c-8998-a9c9822d2732"
+      },
+      "outputs": [],
+      "source": [
+        "# from google.colab import auth\n",
+        "# auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ac33116d-b079-46cb-9614-86326c211e00",
+      "metadata": {
+        "id": "ac33116d-b079-46cb-9614-86326c211e00"
+      },
+      "source": [
+        "**4. Service account or other**\n",
+        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "52c100d7-172f-4578-a3bf-f6e6d193ee6b",
+      "metadata": {
+        "id": "52c100d7-172f-4578-a3bf-f6e6d193ee6b"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "Create a storage bucket to store intermediate artifacts such as datasets."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "638142db-edea-47c2-a4be-f43e6ff4c6f0",
+      "metadata": {
+        "id": "638142db-edea-47c2-a4be-f43e6ff4c6f0"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "bf01f385-3c69-45ca-b72b-84fb45b15f25",
+      "metadata": {
+        "id": "bf01f385-3c69-45ca-b72b-84fb45b15f25"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "a0eabc6a-c7cd-4964-8d27-a020a526b0d9",
+      "metadata": {
+        "id": "a0eabc6a-c7cd-4964-8d27-a020a526b0d9"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "42eb59a2-a959-4cf2-b312-77c726baa361",
+      "metadata": {
+        "id": "42eb59a2-a959-4cf2-b312-77c726baa361"
+      },
+      "source": [
+        "#### Service Account\n",
+        "\n",
+        "You use a service account to create Vertex AI Pipeline jobs."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "b357b4b6-eb6b-404e-b5a8-2771519ce569",
+      "metadata": {
+        "id": "b357b4b6-eb6b-404e-b5a8-2771519ce569"
+      },
+      "outputs": [],
+      "source": [
+        "SERVICE_ACCOUNT = \"\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "0f85a0f2-4ffe-4679-9a23-bc3c755e896a",
+      "metadata": {
+        "id": "0f85a0f2-4ffe-4679-9a23-bc3c755e896a"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "IS_COLAB = \"google.colab\" in sys.modules\n",
+        "if (\n",
+        "    SERVICE_ACCOUNT == \"\"\n",
+        "    or SERVICE_ACCOUNT is None\n",
+        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
+        "):\n",
+        "    # Get your service account from gcloud\n",
+        "    if not IS_COLAB:\n",
+        "        shell_output = !gcloud auth list 2>/dev/null\n",
+        "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
+        "\n",
+        "    else:  # IS_COLAB:\n",
+        "        shell_output = ! gcloud projects describe $PROJECT_ID --format=\"value(projectNumber)\"\n",
+        "        project_number = shell_output[0]\n",
+        "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
+        "\n",
+        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "f64a1440-c23c-4b7b-9b8a-e8a4ce600265",
+      "metadata": {
+        "id": "f64a1440-c23c-4b7b-9b8a-e8a4ce600265"
+      },
+      "source": [
+        "#### Set service account access for Vertex AI Pipelines\n",
+        "\n",
+        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step. You only need to run this step once per service account."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "072a6ea1-c58a-4f9b-9569-6d68631dfed5",
+      "metadata": {
+        "id": "072a6ea1-c58a-4f9b-9569-6d68631dfed5"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
+        "\n",
+        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "e6a924d0-a034-4e53-b240-03d356c7b7a6",
+      "metadata": {
+        "id": "e6a924d0-a034-4e53-b240-03d356c7b7a6"
+      },
+      "source": [
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "463729ba-ec3c-4302-95bf-80207b0f9e2d",
+      "metadata": {
+        "id": "463729ba-ec3c-4302-95bf-80207b0f9e2d"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aiplatform\n",
+        "from vertexai.preview.language_models import (TextGenerationModel,\n",
+        "                                              TuningEvaluationSpec)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a522acfe-d0b6-4b4e-b201-0a4ccf59b133",
+      "metadata": {
+        "id": "a522acfe-d0b6-4b4e-b201-0a4ccf59b133"
+      },
+      "source": [
+        "## Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c845aca6-4f72-4d3b-b9ed-de4a18fcbbf8",
+      "metadata": {
+        "id": "c845aca6-4f72-4d3b-b9ed-de4a18fcbbf8"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "7ec1bf44-dc64-47e0-9bd7-c2d5fc3d0851",
+      "metadata": {
+        "id": "7ec1bf44-dc64-47e0-9bd7-c2d5fc3d0851"
+      },
+      "source": [
+        "### Load pretrained model\n",
+        "\n",
+        "Load the pretrained BISON model from Vertex AI LLM Model Garden."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "22c4fec6-9c10-4f08-80b1-b6e457453103",
+      "metadata": {
+        "id": "22c4fec6-9c10-4f08-80b1-b6e457453103"
+      },
+      "outputs": [],
+      "source": [
+        "model = TextGenerationModel.from_pretrained(\"google/text-bison@001\")\n",
+        "\n",
+        "model.list_tuned_model_names()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "8f9a48e3-8464-41de-93d7-c451824d0ece",
+      "metadata": {
+        "id": "8f9a48e3-8464-41de-93d7-c451824d0ece"
+      },
+      "source": [
+        "### Tune the model\n",
+        "\n",
+        "Next, you tune the model using the `tune_model()` method, with the following parameters:\n",
+        "\n",
+        "`training_data`: A pandas Dataframe or Cloud Storage location of the training data for tuning the model.<br>\n",
+        "`learning_rate_multiplier`: A multiplier to apply to the recommended learning rate. To use the recommended learning rate, use 1.0. <br>\n",
+        "`train_steps`: The number of steps to run for model tuning. The batch size varies by tuning location:<br>\n",
+        "- us-central1 has a batch size of 8.\n",
+        "- europe-west4 has a batch size of 24.<br>\n",
+        "\n",
+        "If there are 240 examples in a training dataset, in europe-west4, it takes 240 / 24 = 10 steps to process the entire dataset once. In us-central1, it takes 240 / 8 = 30 steps to process the entire dataset once. The default value is 300.<br>\n",
+        "\n",
+        "`tuning_job_location`: The region where the tuning job should be run. Supported regions are: `us-central1` and `europe-west4`.<br>\n",
+        "`tuned_model_location`: The region where the tuned model should be deployed."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "8ab391f7-9229-491e-9eff-6e80e9e1d2a7",
+      "metadata": {
+        "id": "8ab391f7-9229-491e-9eff-6e80e9e1d2a7"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil cp gs://cloud-samples-data/vertex-ai/model-evaluation/peft_eval_sample.jsonl {BUCKET_URI}/peft_eval_sample.jsonl\n",
+        "! gsutil cp gs://cloud-samples-data/vertex-ai/model-evaluation/peft_train_sample.jsonl {BUCKET_URI}/peft_train_sample.jsonl\n",
+        "\n",
+        "tuning_evaluation_spec = TuningEvaluationSpec(\n",
+        "    evaluation_data=f\"{BUCKET_URI}/peft_eval_sample.jsonl\",\n",
+        "    evaluation_interval=20,\n",
+        "    enable_early_stopping=True,\n",
+        ")\n",
+        "model.tune_model(\n",
+        "    training_data=f\"{BUCKET_URI}/peft_train_sample.jsonl\",\n",
+        "    # set to 1 for fast iteration and demo purpose. For this sample training dataset, we recommend at least 100 steps\n",
+        "    train_steps=100,\n",
+        "    learning_rate_multiplier=1.0,\n",
+        "    tuning_job_location=\"europe-west4\",\n",
+        "    tuned_model_location=\"us-central1\",  # Only us-central1 is supported\n",
+        "    model_display_name=\"test_model\",\n",
+        "    tuning_evaluation_spec=tuning_evaluation_spec,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "5680557f-67bd-4e8c-a383-02ab655246c5",
+      "metadata": {
+        "id": "5680557f-67bd-4e8c-a383-02ab655246c5"
+      },
+      "source": [
+        "### Make a prediction with Vertex AI LLM\n",
+        "\n",
+        "Now, make a prediction using the `predict()` method from the Vertex AI LLM interface."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "r_0HHwrj96f8",
+      "metadata": {
+        "id": "r_0HHwrj96f8"
+      },
+      "outputs": [],
+      "source": [
+        "prompt = \"TRANSCRIPT: \\nPROCEDURE PERFORMED: , Umbilical hernia repair.,PROCEDURE:,  After informed consent was obtained, the patient was brought to the operative suite and placed supine on the operating table.  The patient was sedated, and an adequate local anesthetic was administered using 1% lidocaine without epinephrine.  The patient was prepped and draped in the usual sterile manner.,A standard curvilinear umbilical incision was made, and dissection was carried down to the hernia sac using a combination of Metzenbaum scissors and Bovie electrocautery.  The sac was cleared of overlying adherent tissue, and the fascial defect was delineated.  The fascia was cleared of any adherent tissue for a distance of 1.5 cm from the defect.  The sac was then placed into the abdominal cavity and the defect was closed primarily using simple interrupted 0 Vicryl sutures.  The umbilicus was then re-formed using 4-0 Vicryl to tack the umbilical skin to the fascia.,The wound was then irrigated using sterile saline, and hemostasis was obtained using Bovie electrocautery.  The skin was approximated with 4-0 Vicryl in a subcuticular fashion.  The skin was prepped with benzoin, and Steri-Strips were applied.  A dressing was then applied.  All surgical counts were reported as correct.,Having tolerated the procedure well, the patient was subsequently taken to the recovery room in good and stable condition.\\n\\n LABEL: \""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "b8c48d2b-bca0-44f3-96ab-0fdde26dd2a9",
+      "metadata": {
+        "id": "b8c48d2b-bca0-44f3-96ab-0fdde26dd2a9"
+      },
+      "outputs": [],
+      "source": [
+        "print(model.predict(prompt))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ae39e95e-9553-4cd8-98cc-602a0d70e940",
+      "metadata": {
+        "id": "ae39e95e-9553-4cd8-98cc-602a0d70e940"
+      },
+      "source": [
+        "### Get the deployed Vertex AI Endpoint resource\n",
+        "\n",
+        "Next, get the Vertex AI Endpoint resource that the model was automatically deployed to."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ade86880-a392-4db6-8152-8e58b50d376b",
+      "metadata": {
+        "id": "ade86880-a392-4db6-8152-8e58b50d376b"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint = aiplatform.Endpoint(model._endpoint.resource_name)\n",
+        "print(endpoint)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "9795902e-e124-48f3-9951-02deb6f85fff",
+      "metadata": {
+        "id": "9795902e-e124-48f3-9951-02deb6f85fff"
+      },
+      "source": [
+        "### Make a prediction using Vertex AI Prediction\n",
+        "\n",
+        "Now, make a prediction using the `predict()` method from the Vertex AI Prediction interface, with the following parameters:\n",
+        "\n",
+        "- `instances`: A list of one or more instances for prediction. Each instance has the format:\n",
+        "    - { \"content\": the_text_input }\n",
+        "- `parameters`: Parameters passed to the model for the model's predict method. The corresponding examples is default values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "35615fa2-c923-4bfd-b194-5265bbe81ce1",
+      "metadata": {
+        "id": "35615fa2-c923-4bfd-b194-5265bbe81ce1"
+      },
+      "outputs": [],
+      "source": [
+        "endpoint.predict(\n",
+        "    instances=[{\"prompt\": prompt}],\n",
+        "    parameters={\n",
+        "        \"temperature\": 0.0,\n",
+        "        \"maxDecodeSteps\": 128,\n",
+        "        \"topP\": 0.95,\n",
+        "        \"topK\": 40,\n",
+        "    },\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0xlMGasOaBJ6",
+      "metadata": {
+        "id": "0xlMGasOaBJ6"
+      },
+      "source": [
+        "### Run Post-Tuning Evaluation\n",
+        "\n",
+        "Note that the format of `ground_truth_data` should be a JSONL file where each line is a json of the following format:\n",
+        "\n",
+        "```\n",
+        "{\n",
+        "  \"prompt\": \"your input/prompt text\",\n",
+        "  \"ground_truth\": \"your ground truth output text\"\n",
+        "}\n",
+        "```\n",
+        "\n",
+        "* \"prompt\" corresponds to the \"input_text\" in the train dataset. This is needed for batch prediction\n",
+        "* \"ground_truth\" corresponds to the \"output_text\" in the train dataset. This is needed for evaluation.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "G55d36wwaI6X",
+      "metadata": {
+        "id": "G55d36wwaI6X"
+      },
+      "outputs": [],
+      "source": [
+        "from vertexai.preview.language_models import EvaluationTextGenerationSpec\n",
+        "\n",
+        "tuned_model = model\n",
+        "\n",
+        "# Uncomment the following to load a tuned model if the tuning session is broken\n",
+        "# tuned_model = TextGenerationModel.from_pretrained(\"google/text-bison@001\")\n",
+        "# tuned_model.get_tuned_model(f'projects/{PROJECT_ID}/locations/us-central1/models/3890975937629519872')\n",
+        "\n",
+        "# Text generation example\n",
+        "evaluation_task_spec = EvaluationTextGenerationSpec(\n",
+        "    ground_truth_data=[\n",
+        "        \"gs://cloud-samples-data/vertex-ai/model-evaluation/peft_test_sample.jsonl\"\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "tuned_model.evaluate(task_spec=evaluation_task_spec)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "99c3c746-4f85-4fd9-8467-d5017477c012",
+      "metadata": {
+        "id": "99c3c746-4f85-4fd9-8467-d5017477c012"
+      },
+      "source": [
+        "## Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "0f467b0a-07c5-4c49-b6cd-7588c9e3985b",
+      "metadata": {
+        "id": "0f467b0a-07c5-4c49-b6cd-7588c9e3985b"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "delete_bucket = False\n",
+        "\n",
+        "endpoint.undeploy_all()\n",
+        "endpoint.delete()\n",
+        "\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -rf {BUCKET_URI}"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "tune_peft.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}