mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3071)
* Fixed code issue template issue in distillation file * Did required changes in notebook template * Did minor changes * Added execption handling at cleanup step to handle error while performing cleanup * Fixed issue based on feedback given on feedback * fix, chore, refactor: removes hard-coded project-id, remove future tense and reorganizes the sections, refactors the cleaning up section * fix, refactor, chore: cleans up the resources using display name rather than resource name, adds wait step to wait until the pipeline job is finished, updates the overview section to remove 'we' * fix: runs _job.wait() instead of .wait() method for waiting, updates the var pipeline_job to pipeline --------- Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
This commit is contained in:
nileshspringml
co-authored by
krishr2d2
krishr2d2
parent
50d70a3037
commit
575d2f9479
@@ -29,41 +29,32 @@
|
||||
"id": "JAPoU8Sm5E6e"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI Distill a model\n",
|
||||
"# Vertex AI: Distill a large language model\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/distillation.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/distillation.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fgenerative_ai%2Fdistillation.ipynb\">\n",
|
||||
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" </td> \n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/generative_ai/distillation.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/distillation.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "24743cf4a1e1"
|
||||
},
|
||||
"source": [
|
||||
"**_NOTE_**: This notebook has been tested in the following environment:\n",
|
||||
"\n",
|
||||
"* Python version = 3.9"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -72,11 +63,15 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use the Distilling Step by Step on the Vertex AI.\n",
|
||||
"This tutorial demonstrates how to use the distilling Step by Step on the Vertex AI.\n",
|
||||
"\n",
|
||||
"We developed the distilling step-by-step (DSS) method ([paper](https://arxiv.org/abs/2305.02301v1)) that can enrich customer’s data by eliciting the reasoning process (rationales) from a large language model (LLM). This new mechanism has shown to be able to (a) train smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by fine-tuning or distillation. Our method extracts LLM rationales as additional supervision within a multi-task training framework.\n",
|
||||
"The distilling step-by-step (DSS) method ([paper](https://arxiv.org/abs/2305.02301v1)) can enrich customer’s data by eliciting the reasoning process (rationales) from a large language model (LLM). This new mechanism has shown to be able to (a) train smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by fine-tuning or distillation. This method extracts LLM rationales as additional supervision within a multi-task training framework.\n",
|
||||
"\n",
|
||||
"Learn more about [distill-text-models](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models)."
|
||||
"Learn more about [distill-text-models](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models).\n",
|
||||
"\n",
|
||||
"**_NOTE_**: This notebook is tested in the following environment:\n",
|
||||
"\n",
|
||||
"* Python version = 3.9"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -87,23 +82,60 @@
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn to use `Vertex AI LLM` to distill and deploy a large language model.\n",
|
||||
"In this tutorial, you learn how to distill and deploy a large language model using Vertex AI LLM.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Vertex AI services:\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
"- `Vertex AI LLM`\n",
|
||||
"- `Vertex AI Model Garden`\n",
|
||||
"- `Vertex AI Prediction`\n",
|
||||
"- Vertex AI LLM\n",
|
||||
"- Vertex AI Model Garden\n",
|
||||
"- Vertex AI Online prediction\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Get the Vertex AI LLM model.\n",
|
||||
"- Distill the model.\n",
|
||||
" - This will automatically create a Vertex AI endpoint and deploy the model to it.\n",
|
||||
"- Make a prediction using `Vertex AI LLM`.\n",
|
||||
"- Make a prediction using `Vertex AI Prediction`"
|
||||
"- Distill the model(this automatically creates a Vertex AI endpoint and deploys the model to the endpoint). \n",
|
||||
"- Make a prediction using Vertex AI LLM."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3d269b76353d"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"Distillation works on a labeled or an unlabeled dataset. If you have a high quality labeled dataset with hundreds of examples, then it's recommended that you use the labeled dataset. Otherwise, you can use an unlabeled prompt dataset. If you use an unlabeled dataset, then the teacher model generates the labels and the rationale for distillation. More than 1,000 examples are recommended if you use an unlabeled dataset.\n",
|
||||
"\n",
|
||||
"For this tutorial, you use a dataset stored in a public Cloud Storage bucket at the below paths. \n",
|
||||
"- Train sample: `gs://cloud-samples-data/vertex-ai/model-evaluation/peft_train_sample.jsonl`\n",
|
||||
"- Validation sample: `gs://cloud-samples-data/vertex-ai/model-evaluation/peft_eval_sample.jsonl`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e7c0aa7c769f"
|
||||
},
|
||||
"source": [
|
||||
"#### Input format requirement\n",
|
||||
"\n",
|
||||
"The labeled or unlabeled distillation dataset must be in JSON Lines (JSONL) format where each line contains a single tuning example. Before you distill your model, upload your dataset to a Cloud Storage bucket.\n",
|
||||
"\n",
|
||||
"Each dataset example contains an `input_text` field with the model prompt and an optional `output_text` field that contains an example response that the distilled model is expected to produce.\n",
|
||||
"\n",
|
||||
"The maximum token length for `input_text` is 7,168 and the maximum token length for `output_text` is 1,024. If either field exceeds the maximum token length, the excess tokens are truncated.\n",
|
||||
"\n",
|
||||
"The maximum number of examples that a dataset for a text generation model can contain is 10,000.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Example:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\"input_text\": \"question: How many people live in Beijing? context: With over 21 million residents, Beijing is the world's most populous national capital city and is China's second largest city after Shanghai. It is located in Northern China, and is governed as a municipality under the direct administration of the State Council with 16 urban, suburban, and rural districts.[14] Beijing is mostly surrounded by Hebei Province with the exception of neighboring Tianjin to the southeast; together, the three divisions form the Jingjinji megalopolis and the national capital region of China.\", \"output_text\": \"over 21 million people\"}\n",
|
||||
"{\"input_text\": \"question: How many parishes are there in Louisiana? context: The U.S. state of Louisiana is divided into 64 parishes (French: paroisses) in the same manner that 48 other states of the United States are divided into counties, and Alaska is divided into boroughs.\", \"output_text\": \"64\"}\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -129,189 +161,127 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "i7EUnXsZhAGF"
|
||||
"id": "f0316df526f8"
|
||||
},
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the following packages required to execute this notebook. "
|
||||
"## Get started"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a2c2cb2109a0"
|
||||
},
|
||||
"source": [
|
||||
"### Install Vertex AI SDK for Python and other required packages\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2b4ef9b72d43"
|
||||
"id": "89d404f6cc9d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade --quiet google-cloud-aiplatform \"shapely<2.0.0\""
|
||||
"! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
|
||||
" \"shapely<2.0.0\" \\\n",
|
||||
" PyYAML"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "58707a750154"
|
||||
"id": "ff555b32bab8"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only: Uncomment the following cell to restart the kernel."
|
||||
"### Restart runtime (Colab only)\n",
|
||||
"\n",
|
||||
"To use the newly installed packages, you must restart the runtime on Google Colab."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "f200f10a1da3"
|
||||
"id": "f09b4dff629a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# import IPython\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# app = IPython.Application.instance()\n",
|
||||
"# app.kernel.do_shutdown(True)"
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "BF1j6f9HApxa"
|
||||
"id": "54c5ef8a8f43"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
|
||||
"<div class=\"alert alert-block alert-warning\">\n",
|
||||
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
||||
"</div>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WReHDGG5g0XY"
|
||||
"id": "92e68cfc3a90"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"### Authenticate your notebook environment (Colab only)\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, try the following:\n",
|
||||
"* Run `gcloud config list`.\n",
|
||||
"* Run `gcloud projects list`.\n",
|
||||
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
|
||||
"Authenticate your environment on Google Colab.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "oM1iC_MfAts1"
|
||||
"id": "46604f70e831"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" from google.colab import auth\n",
|
||||
"\n",
|
||||
" auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "4f872cd812d0"
|
||||
},
|
||||
"source": [
|
||||
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "5ff1a3cc4e1d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Set the project id\n",
|
||||
"! gcloud config set project {PROJECT_ID}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"import vertexai\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "sBCra4QMA2wR"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "74ccc9e52986"
|
||||
},
|
||||
"source": [
|
||||
"**1. Vertex AI Workbench**\n",
|
||||
"* Do nothing as you are already authenticated."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "de775a3773ba"
|
||||
},
|
||||
"source": [
|
||||
"**2. Local JupyterLab instance, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "254614fa0c46"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! gcloud auth login"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ef21552ccea8"
|
||||
},
|
||||
"source": [
|
||||
"**3. Colab, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "603adbbf0532"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# from google.colab import auth\n",
|
||||
"# auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f6b2ccc891ed"
|
||||
},
|
||||
"source": [
|
||||
"**4. Service account or other**\n",
|
||||
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
|
||||
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -322,9 +292,7 @@
|
||||
"source": [
|
||||
"### Create a Cloud Storage bucket\n",
|
||||
"\n",
|
||||
"Create a storage bucket to store intermediate artifacts such as datasets.\n",
|
||||
"\n",
|
||||
"- *{Note to notebook author: For any user-provided strings that need to be unique (like bucket names or model ID's), append \"-unique\" to the end so proper testing can occur}*"
|
||||
"Create a storage bucket to store intermediate artifacts such as datasets."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -344,7 +312,7 @@
|
||||
"id": "-EcIXiGsCePi"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -355,7 +323,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -364,9 +332,11 @@
|
||||
"id": "08d289fa873f"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"#### Copy the dataset to your bucket\n",
|
||||
"\n",
|
||||
"We've provided the below sample data for you to get started.\n"
|
||||
"Before you start the distillation, copy the dataset from the source to your Cloud Storage bucket.\n",
|
||||
"\n",
|
||||
"**Note**: Alternatively, you can directly specify the source path for the data when you perform distillation. Copying the data to your Google Cloud project is only optional."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -381,33 +351,6 @@
|
||||
"! gsutil cp gs://cloud-samples-data/vertex-ai/model-evaluation/peft_train_sample.jsonl {BUCKET_URI}/peft_train_sample.jsonl"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0e6f90042af6"
|
||||
},
|
||||
"source": [
|
||||
"#### Data Input format requirement\n",
|
||||
"\n",
|
||||
"Distillation works on a labeled or an unlabeled dataset. If you have a high quality labeled dataset with hundreds of examples, then we recommend that you use that. Otherwise, you can use an unlabeled prompt dataset. If you use an unlabeled dataset, then the teacher model generates the labels and the rationale for distillation. More than 1,000 examples are recommended if you use an unlabeled dataset.\n",
|
||||
"\n",
|
||||
"The labeled or unlabeled distillation dataset must be in JSON Lines (JSONL) format where each line contains a single tuning example. Before you distill your model, you upload your dataset to a Cloud Storage bucket.\n",
|
||||
"\n",
|
||||
"Each dataset example contains an `input_text` field with the model prompt and an optional `output_text` field that contains an example response that the distilled model is expected to produce.\n",
|
||||
"\n",
|
||||
"The maximum token length for `input_text` is 7,168 and the maximum token length for `output_text` is 1,024. If either field exceeds the maximum token length, the excess tokens are truncated.\n",
|
||||
"\n",
|
||||
"The maximum number of examples that a dataset for a text generation model can contain is 10,000.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Sample dataset:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"{\"input_text\": \"question: How many people live in Beijing? context: With over 21 million residents, Beijing is the world's most populous national capital city and is China's second largest city after Shanghai. It is located in Northern China, and is governed as a municipality under the direct administration of the State Council with 16 urban, suburban, and rural districts.[14] Beijing is mostly surrounded by Hebei Province with the exception of neighboring Tianjin to the southeast; together, the three divisions form the Jingjinji megalopolis and the national capital region of China.\", \"output_text\": \"over 21 million people\"}\n",
|
||||
"{\"input_text\": \"question: How many parishes are there in Louisiana? context: The U.S. state of Louisiana is divided into 64 parishes (French: paroisses) in the same manner that 48 other states of the United States are divided into counties, and Alaska is divided into boroughs.\", \"output_text\": \"64\"}\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -425,44 +368,21 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import vertexai\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from vertexai.preview.language_models import (TextGenerationModel,\n",
|
||||
" TuningEvaluationSpec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk,all"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk,all"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vertexai.init(project=PROJECT_ID, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a0275c76a30f"
|
||||
},
|
||||
"source": [
|
||||
"### Load pretrained model\n",
|
||||
"## Load pretrained model\n",
|
||||
"\n",
|
||||
"Load the pretrained BISON model from Vertex AI LLM Model Garden.\n",
|
||||
"See the models that supports distillation in [here](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models#supported_models)."
|
||||
"See the [list of models that support distillation](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models#supported_models)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -485,20 +405,19 @@
|
||||
"id": "3f35db20ac38"
|
||||
},
|
||||
"source": [
|
||||
"### Distill the model\n",
|
||||
"## Distill the model\n",
|
||||
"\n",
|
||||
"Next, you distill the model using the `distill_from()` method, with the following parameters:\n",
|
||||
"\n",
|
||||
"`teacher_model`: The teacher model that you would like to distill the knowledge from.\n",
|
||||
"`dataset`: A pandas Dataframe or Cloud Storage location of the training data for tuning the model.<br>\n",
|
||||
"`learning_rate_multiplier`: A multiplier to apply to the recommended learning rate. To use the recommended learning rate, use 1.0. <br>\n",
|
||||
"`train_steps`: The number of steps to run for model tuning. The batch size varies by tuning location:<br>\n",
|
||||
"- us-central1 has a batch size of 8.\n",
|
||||
"- europe-west4 has a batch size of 24.<br>\n",
|
||||
"- `teacher_model`: The teacher model that you would like to distill the knowledge from.\n",
|
||||
"- `dataset`: A pandas Dataframe or Cloud Storage location of the training data for tuning the model.\n",
|
||||
"- `learning_rate_multiplier`: A multiplier to apply to the recommended learning rate. To use the recommended learning rate, use 1.0.\n",
|
||||
"- `train_steps`: The number of steps to run for model tuning. The default value is 300. The batch size varies by tuning location as below for 8k models such as `text-bison@002`:\n",
|
||||
" \n",
|
||||
" - us-central1 has a batch size of 8.\n",
|
||||
" - europe-west4 has a batch size of 24.\n",
|
||||
"\n",
|
||||
"If there are 240 examples in a training dataset, in europe-west4, it takes 240 / 24 = 10 steps to process the entire dataset once. In us-central1, it takes 240 / 8 = 30 steps to process the entire dataset once. The default value is 300.<br>\n",
|
||||
"\n",
|
||||
"For more context, see this [doc](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models#create_a_text_model_distilling_job) for definition of the parameters. "
|
||||
"For parameter definitions and further context, see [Create a text model distilling job](https://cloud.google.com/vertex-ai/docs/generative-ai/models/distill-text-models#create_a_text_model_distilling_job). "
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -517,6 +436,17 @@
|
||||
"eval_spec.evaluation_interval = 20"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "495abb2c72c8"
|
||||
},
|
||||
"source": [
|
||||
"Set a display name for your model resource and the endpoint resource using the `DISPLAY_NAME` parameter.\n",
|
||||
"\n",
|
||||
"**Note**: In the tuning pipeline, the model and endpoint share the same display name."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -525,15 +455,22 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"student_model.distill_from(\n",
|
||||
"# Set the display name\n",
|
||||
"DISPLAY_NAME = \"vertex-distillation-model-unique\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Create the tuning pipeline job\n",
|
||||
"pipeline = student_model.distill_from(\n",
|
||||
" teacher_model=teacher_model,\n",
|
||||
" dataset=f\"{BUCKET_URI}/peft_train_sample.jsonl\",\n",
|
||||
" train_steps=200,\n",
|
||||
" learning_rate_multiplier=1,\n",
|
||||
" accelerator_type=\"TPU\",\n",
|
||||
" model_display_name=\"test-vertex-distillation\",\n",
|
||||
" model_display_name=DISPLAY_NAME,\n",
|
||||
" evaluation_spec=eval_spec,\n",
|
||||
")"
|
||||
")\n",
|
||||
"\n",
|
||||
"# Wait until the tuning pipeline job finishes\n",
|
||||
"pipeline._job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -542,7 +479,7 @@
|
||||
"id": "d432a5238785"
|
||||
},
|
||||
"source": [
|
||||
"### Make a prediction with Vertex AI LLM\n",
|
||||
"## Make a prediction with Vertex AI LLM\n",
|
||||
"\n",
|
||||
"Now, make a prediction using the `predict()` method from the Vertex AI LLM interface."
|
||||
]
|
||||
@@ -555,6 +492,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Define the prompt\n",
|
||||
"prompt = \"TRANSCRIPT: \\nPROCEDURE PERFORMED: , Umbilical hernia repair.,PROCEDURE:, After informed consent was obtained, the patient was brought to the operative suite and placed supine on the operating table. The patient was sedated, and an adequate local anesthetic was administered using 1% lidocaine without epinephrine. The patient was prepped and draped in the usual sterile manner.,A standard curvilinear umbilical incision was made, and dissection was carried down to the hernia sac using a combination of Metzenbaum scissors and Bovie electrocautery. The sac was cleared of overlying adherent tissue, and the fascial defect was delineated. The fascia was cleared of any adherent tissue for a distance of 1.5 cm from the defect. The sac was then placed into the abdominal cavity and the defect was closed primarily using simple interrupted 0 Vicryl sutures. The umbilicus was then re-formed using 4-0 Vicryl to tack the umbilical skin to the fascia.,The wound was then irrigated using sterile saline, and hemostasis was obtained using Bovie electrocautery. The skin was approximated with 4-0 Vicryl in a subcuticular fashion. The skin was prepped with benzoin, and Steri-Strips were applied. A dressing was then applied. All surgical counts were reported as correct.,Having tolerated the procedure well, the patient was subsequently taken to the recovery room in good and stable condition.\\n\\n LABEL: \""
|
||||
]
|
||||
},
|
||||
@@ -566,6 +504,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Print the prompt\n",
|
||||
"print(student_model.predict(prompt))"
|
||||
]
|
||||
},
|
||||
@@ -591,27 +530,30 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"# Fetch the endpoint resource using the display name and create time\n",
|
||||
"endpoints = aiplatform.Endpoint.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
")\n",
|
||||
"if len(endpoints) > 0:\n",
|
||||
" # Undeploy the model from the endpoint\n",
|
||||
" endpoints[0].undeploy_all()\n",
|
||||
" # Delete the endpoint\n",
|
||||
" endpoints[0].delete()\n",
|
||||
"\n",
|
||||
"delete_bucket = False\n",
|
||||
"# Fetch the model resource using the display name and create time\n",
|
||||
"models = aiplatform.Model.list(\n",
|
||||
" filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
|
||||
")\n",
|
||||
"if len(models) > 0:\n",
|
||||
" # Delete the model\n",
|
||||
" models[0].delete()\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Delete endpoint resource\n",
|
||||
" endpoint = aiplatform.Endpoint(student_model._endpoint.resource_name)\n",
|
||||
" endpoint.undeploy_all()\n",
|
||||
" endpoint.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "d473314cae55"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = False\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
"# Delete the pipeline job\n",
|
||||
"pipeline._job.delete()\n",
|
||||
"\n",
|
||||
"# Delete the Cloud Storage bucket\n",
|
||||
"delete_bucket = True\n",
|
||||
"if delete_bucket:\n",
|
||||
" ! gsutil rm -rf {BUCKET_URI}"
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user