xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add a notebook for llama2 serving and finetuning with peft, and analyze with vertex text moderation API (#2170)

Browse Source 
* Add a notebook for llama2 serving and finetuning with peft, and analyze with vertex text moderation API

* Fix comments
This commit is contained in:
genquan9
2023-08-14 15:57:31 +00:00
committed by GitHub
parent 0b13475152
commit 734836b928
2 changed files with 762 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/community/CODEOWNERS
+1 -1
View File
@@ -82,4 +82,4 @@
/notebooks/community/model_garden/model_garden_pytorch_falcon_instruct_peft.ipynb @genquan9
/notebooks/community/model_garden/model_garden_movinet_clip_classification.ipynb @KCFindstr
/notebooks/community/model_garden/model_garden_pytorch_open_clip.ipynb @lydhr
/notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb @genquan9
notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb
+761
View File
@@ -0,0 +1,761 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7d9bbf86da5e"
},
"outputs": [],
"source": [
"# Copyright 2023 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "99c1c3fc2ca5"
},
"source": [
"# Vertex AI Model Garden - LLaMA2 (PEFT)\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
"Open in Vertex AI Workbench\n",
" </a> (A Python-3 CPU notebook is recommended)\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3de7470326a2"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates deploying prebuilt LLaMA2 models, and also finetuning and deploying LLaMA2 models with performance efficient finetuning libraries ([PEFT](https://github.com/huggingface/peft)) in Vertex AI. This notebook also uses [Text moderation APIs](https://cloud.google.com/natural-language/docs/moderating-text) to analyze predictions against a list of safety attributes\n",
"\n",
"### Objective\n",
"\n",
"- Deploy prebuilt LLaMA2 models\n",
"- Finetune and deploy LLaMA2 models with PEFT\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "264c07757582"
},
"source": [
"## Before you begin"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ioensNKM8ned"
},
"source": [
"### Colab only\n",
"Run the following commands for Colab and skip this section if you are using Workbench."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2707b02ef5df"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" ! pip3 install --upgrade google-cloud-aiplatform\n",
" ! pip3 install ipython pandas[output_formatting] google-cloud-language==2.10.0\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
" # Install gdown for downloading example training images.\n",
" ! pip3 install gdown\n",
"\n",
" # Restart the notebook kernel after installs.\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bb7adab99e41"
},
"source": [
"### Setup Google Cloud project\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
"\n",
"1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs.\n",
"\n",
"1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console) with `Vertex AI User` and `Storage Object Admin` roles for deploying fine tuned model to Vertex AI endpoint."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6c460088b873"
},
"source": [
"Fill following variables for experiments environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "855d6b96f291"
},
"outputs": [],
"source": [
"# Cloud project id.\n",
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
"\n",
"# The region you want to launch jobs in.\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"\n",
"# The Cloud Storage bucket for storing experiments output.\n",
"BUCKET_URI = \"\" # @param {type:\"string\"}\n",
"\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud services enable language.googleapis.com\n",
"\n",
"import os\n",
"\n",
"STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n",
"EXPERIMENT_BUCKET = os.path.join(BUCKET_URI, \"peft\")\n",
"DATA_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"data\")\n",
"BASE_MODEL_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"base_model\")\n",
"MODEL_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"model\")\n",
"PREDICTION_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"prediction\")\n",
"\n",
"# The service account looks like:\n",
"# '@.iam.gserviceaccount.com'\n",
"# Please go to https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console\n",
"# and create service account with `Vertex AI User` and `Storage Object Admin` roles.\n",
"# The service account for deploying fine tuned model.\n",
"SERVICE_ACCOUNT = \"\" # @param {type:\"string\"}\n",
"\n",
"from google.colab import auth\n",
"\n",
"auth.authenticate_user(project_id=PROJECT_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e828eb320337"
},
"source": [
"### Initialize Vertex AI API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "12cd25839741"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2cc825514deb"
},
"source": [
"### Define constants"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b42bd4fa2b2d"
},
"outputs": [],
"source": [
"# The pre-built training and serving docker images.\n",
"TRAIN_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-peft-train\"\n",
"PREDICTION_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-peft-serve\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0c250872074f"
},
"source": [
"### Define common functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "354da31189dc"
},
"outputs": [],
"source": [
"import os\n",
"from datetime import datetime\n",
"\n",
"from google.cloud import aiplatform, language\n",
"\n",
"\n",
"def get_job_name_with_datetime(prefix: str):\n",
" \"\"\"Gets the job name with date time when triggering training or deployment\n",
" jobs in Vertex AI.\n",
" \"\"\"\n",
" return prefix + datetime.now().strftime(\"_%Y%m%d_%H%M%S\")\n",
"\n",
"\n",
"def deploy_model(\n",
" model_name,\n",
" base_model_id,\n",
" finetuned_lora_model_path,\n",
" service_account,\n",
" task,\n",
" machine_type=\"n1-standard-8\",\n",
" accelerator_type=\"NVIDIA_TESLA_V100\",\n",
" accelerator_count=1,\n",
"):\n",
" \"\"\"Deploys trained models into Vertex AI.\"\"\"\n",
" endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
" serving_env = {\n",
" \"BASE_MODEL_ID\": base_model_id,\n",
" \"TASK\": task,\n",
" }\n",
" if finetuned_lora_model_path:\n",
" serving_env[\"FINETUNED_LORA_MODEL_PATH\"] = finetuned_lora_model_path\n",
" model = aiplatform.Model.upload(\n",
" display_name=model_name,\n",
" serving_container_image_uri=PREDICTION_DOCKER_URI,\n",
" serving_container_ports=[7080],\n",
" serving_container_predict_route=\"/predictions/peft_serving\",\n",
" serving_container_health_route=\"/ping\",\n",
" serving_container_environment_variables=serving_env,\n",
" )\n",
" model.deploy(\n",
" endpoint=endpoint,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" deploy_request_timeout=1800,\n",
" service_account=service_account,\n",
" )\n",
" return model, endpoint\n",
"\n",
"\n",
"def moderate_text(text: str) -> language.ModerateTextResponse:\n",
" \"\"\"Calls Vertex AI APIs to analyze text moderations.\"\"\"\n",
" client = language.LanguageServiceClient()\n",
" document = language.Document(\n",
" content=text,\n",
" type_=language.Document.Type.PLAIN_TEXT,\n",
" )\n",
" return client.moderate_text(document=document)\n",
"\n",
"\n",
"def show_text_moderation(text: str, response: language.ModerateTextResponse):\n",
" \"\"\"Shows text moderation results.\"\"\"\n",
" import pandas as pd\n",
"\n",
" def confidence(category: language.ClassificationCategory) -> float:\n",
" return category.confidence\n",
"\n",
" columns = [\"category\", \"confidence\"]\n",
" categories = sorted(response.moderation_categories, key=confidence, reverse=True)\n",
" data = ((category.name, category.confidence) for category in categories)\n",
" df = pd.DataFrame(columns=columns, data=data)\n",
"\n",
" print(f\"Text analyzed:\\n{text}\")\n",
" print(df.to_markdown(index=False, tablefmt=\"presto\", floatfmt=\".0%\"))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ivs2RK093c8X"
},
"source": [
"## Access LLaMA2 pretrained and finetuned models\n",
"The original models from Meta are converted into the Hugging Face format for finetuning and serving in Vertex AI.\n",
"After clicking the agreement of LLaMA2 in Vertex AI Model Garden, a Cloud Storage bucket will be shared to access LLaMA2 pretrained and finetuned models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jwn4PcTf4EMt"
},
"outputs": [],
"source": [
"VERTEX_AI_MODEL_GARDEN_LLAMA2 = \"\" # This will be shared once click the agreement of LLaMA2 in Vertex AI Model Garden.\n",
"! gsutil cp -R $VERTEX_AI_MODEL_GARDEN_LLAMA2 $BASE_MODEL_BUCKET"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2MjaORIIFDVu"
},
"source": [
"Set the base model id."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E8OiHHNNE_wj"
},
"outputs": [],
"source": [
"base_model_name = \"llama2-7b-chat-hf\" # @param [\"llama2-7b-hf\", \"llama2-7b-chat-hf\", \"llama2-13b-hf\", \"llama2-13b-chat-hf\", \"llama2-70b-hf\", \"llama2-70b-chat-hf\"]\n",
"base_model_id = os.path.join(BASE_MODEL_BUCKET, base_model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8neJc8CnDDpu"
},
"source": [
"## Deploy prebuilt LLaMA2 models\n",
"\n",
"This section deploys prebuilt LLaMA2 models on Vertex AI."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dHFW7yvjaVFV"
},
"source": [
"You use the PEFT serving images to deploy prebuilt LLaMA2 models, by setting finetuning LoRA model paths as empty. The model deployment step will take ~15 minutes to complete."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Uak1pyEeExYM"
},
"outputs": [],
"source": [
"# Sets V100 to deploy LLaMA2 7B.\n",
"machine_type = \"n1-standard-8\"\n",
"accelerator_type = \"NVIDIA_TESLA_V100\"\n",
"accelerator_count = 1\n",
"\n",
"# Sets A100 (40G) to deploy LLaMA2 13B.\n",
"# machine_type = \"a2-highgpu-1g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 1\n",
"\n",
"# Sets 4 A100 (40G) to deploy LLaMA2 70B models.\n",
"# machine_type = \"a2-highgpu-4g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 4\n",
"\n",
"# The supported precision loading types are \"4bit\", \"8bit\", \"float16\" and \"float32\".\n",
"precision_loading_type = \"float16\"\n",
"\n",
"model_without_peft, endpoint_without_peft = deploy_model(\n",
" model_name=get_job_name_with_datetime(prefix=\"llama2-serve\"),\n",
" base_model_id=base_model_id,\n",
" finetuned_lora_model_path=\"\", # This will avoid override finetuning models.\n",
" service_account=SERVICE_ACCOUNT,\n",
" task=\"causal-language-modeling-lora\",\n",
" precision_loading_type=precision_loading_type,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=1,\n",
")\n",
"print(\"endpoint_name:\", endpoint_without_peft.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sGKIjgmDFRW2"
},
"source": [
"NOTE: The prebuilt model weights will be downloaded on the fly from $BASE_MODEL_BUCKET after the deployment succeeds. Thus additional 5 minutes of waiting time is needed **after** the above model deployment step succeeds and before you run the next step below. Otherwise you might see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint.\n",
"\n",
"Once deployment succeeds, you can send requests to the endpoint with text prompts.\n",
"\n",
"Example:\n",
"\n",
"```\n",
"Human: What is a car?\n",
"Assistant: A car, or a motor car, is a road-connected human-transportation system used to move people or goods from one place to another. The term also encompasses a wide range of vehicles, including motorboats, trains, and aircrafts. Cars typically have four wheels, a cabin for passengers, and an engine or motor. They have been around since the early 19th century and are now one of the most popular forms of transportation, used for daily commuting, shopping, and other purposes.\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rDHsCOqvFYBi"
},
"outputs": [],
"source": [
"# Loads an existing endpoint as below.\n",
"# endpoint_name = endpoint_without_peft.name\n",
"# aip_endpoint_name = (\n",
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
"# )\n",
"# endpoint_without_peft = aiplatform.Endpoint(aip_endpoint_name)\n",
"instances = [\n",
" {\"prompt\": \"Write a poem about Valencia.\"},\n",
"]\n",
"response = endpoint_without_peft.predict(instances=instances)\n",
"\n",
"for prediction in response.predictions[0]:\n",
" print(prediction[\"generated_text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aUMsewPDj_pS"
},
"source": [
"Text moderation analyzes a document against a list of safety attributes, which include \"harmful categories\" and topics that may be considered sensitive."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EE_GCSVVkBWj"
},
"outputs": [],
"source": [
"for prediction in response.predictions[0]:\n",
" generated_text = prediction[\"generated_text\"]\n",
" # Send a request to the API.\n",
" response = moderate_text(generated_text)\n",
"\n",
" # Show the results.\n",
" show_text_moderation(generated_text, response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e70e3519ff8b"
},
"source": [
"## Finetune and deploy LLaMA2 models with PEFT\n",
"\n",
"This section demonstrates how to finetune and deploy LLaMA2 models with PEFT LoRA."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iWGwJHqI7LMs"
},
"source": [
"### Finetune"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KKEYoRfiHDVv"
},
"source": [
"Use the Vertex AI SDK to create and run the custom training jobs with Vertex AI Model Garden training images.\n",
"\n",
"This example uses the dataset [Abirate/english_quotes](https://huggingface.co/datasets/Abirate/english_quotes)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "65467b361315"
},
"outputs": [],
"source": [
"dataset_name = \"Abirate/english_quotes\" # @param {type:\"string\"}\n",
"\n",
"# Worker pool spec.\n",
"# Finetunes LLaMA2 7B with 1 V100 (16G).\n",
"machine_type = \"n1-standard-8\"\n",
"accelerator_type = \"NVIDIA_TESLA_V100\"\n",
"accelerator_count = 1\n",
"\n",
"# Finetunes and LLaMA2 13B with 1 A100 (40G).\n",
"# machine_type = \"a2-highgpu-1g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 1\n",
"\n",
"# Finetunes and LLaMA2 70B with 4 A100 (40G).\n",
"# machine_type = \"a2-highgpu-4g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 4\n",
"\n",
"replica_count = 1\n",
"\n",
"\n",
"# Setup training job.\n",
"job_name = get_job_name_with_datetime(\"llama2-lora-train\")\n",
"train_job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=job_name,\n",
" container_uri=TRAIN_DOCKER_URI,\n",
")\n",
"output_dir = os.path.join(MODEL_BUCKET, job_name)\n",
"output_dir_gcsfuse = output_dir.replace(\"gs://\", \"/gcs/\")\n",
"\n",
"# Pass training arguments and launch job.\n",
"train_job.run(\n",
" args=[\n",
" \"--task=causal-language-modeling-lora\",\n",
" f\"--pretrained_model_id={base_model_id}\",\n",
" f\"--dataset_name={dataset_name}\",\n",
" f\"--output_dir={output_dir_gcsfuse}\",\n",
" \"--lora_rank=16\",\n",
" \"--lora_alpha=32\",\n",
" \"--lora_dropout=0.05\",\n",
" \"--warmup_steps=10\",\n",
" \"--max_steps=10\",\n",
" \"--learning_rate=2e-4\",\n",
" ],\n",
" replica_count=replica_count,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" boot_disk_size_gb=500,\n",
")\n",
"\n",
"print(\"Trained models were saved in: \", output_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jqmCtkGnhDmp"
},
"source": [
"### Deploy\n",
"This section uploads the model to Model Registry and deploys it on the Endpoint.\n",
"\n",
"The model deployment step will take ~15 minutes to complete."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bf55e38815dc"
},
"outputs": [],
"source": [
"# Sets V100 to deploy LLaMA2 7B.\n",
"machine_type = \"n1-standard-8\"\n",
"accelerator_type = \"NVIDIA_TESLA_V100\"\n",
"accelerator_count = 1\n",
"\n",
"# Sets A100 (40G) to deploy LLaMA2 13B.\n",
"# machine_type = \"a2-highgpu-1g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 1\n",
"\n",
"# Sets 4 A100 (40G) to deploy LLaMA2 70B models.\n",
"# machine_type = \"a2-highgpu-4g\"\n",
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"# accelerator_count = 4\n",
"\n",
"# The supported precision loading types are \"4bit\", \"8bit\", \"float16\" and \"float32\".\n",
"precision_loading_type = \"float16\"\n",
"\n",
"model_with_peft, endpoint_with_peft = deploy_model(\n",
" model_name=get_job_name_with_datetime(prefix=\"llama2-serve\"),\n",
" base_model_id=base_model_id,\n",
" finetuned_lora_model_path=output_dir,\n",
" service_account=SERVICE_ACCOUNT,\n",
" task=\"causal-language-modeling-lora\",\n",
" precision_loading_type=precision_loading_type,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=1,\n",
")\n",
"print(\"endpoint_name:\", endpoint_with_peft.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "80b3fd2ace09"
},
"source": [
"NOTE: After the deployment succeeds, the base model weights will be downloaded on the fly from $BASE_MODEL_BUCKET and LoRA model weights will be downloaded from the GCS bucket used in training above. Thus additional 5 minutes of waiting time is needed **after** the above model deployment step succeeds and before you run the next step below. Otherwise you might see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint.\n",
"\n",
"Once deployment succeeds, you can send requests to the endpoint with text prompts.\n",
"\n",
"Example:\n",
"\n",
"```\n",
"Human: What is a car?\n",
"Assistant: A car, or a motor car, is a road-connected human-transportation system used to move people or goods from one place to another. The term also encompasses a wide range of vehicles, including motorboats, trains, and aircrafts. Cars typically have four wheels, a cabin for passengers, and an engine or motor. They have been around since the early 19th century and are now one of the most popular forms of transportation, used for daily commuting, shopping, and other purposes.\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4ab04da3ec9a"
},
"outputs": [],
"source": [
"# Loads an existing endpoint as below.\n",
"# endpoint_name = endpoint_with_peft.name\n",
"# aip_endpoint_name = (\n",
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
"# )\n",
"# endpoint_with_peft = aiplatform.Endpoint(aip_endpoint_name)\n",
"instances = [\n",
" {\"prompt\": \"What is a car?\"},\n",
"]\n",
"response = endpoint_with_peft.predict(instances=instances)\n",
"\n",
"for prediction in response.predictions[0]:\n",
" print(prediction[\"generated_text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yfPDd91qlSlI"
},
"source": [
"Text moderation analyzes a document against a list of safety attributes, which include \"harmful categories\" and topics that may be considered sensitive."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lEUyncyklTEE"
},
"outputs": [],
"source": [
"for prediction in response.predictions[0]:\n",
" generated_text = prediction[\"generated_text\"]\n",
" # Send a request to the API.\n",
" response = moderate_text(generated_text)\n",
"\n",
" # Show the results.\n",
" show_text_moderation(generated_text, response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "af21a3cff1e0"
},
"source": [
"## Clean up resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "911406c1561e"
},
"outputs": [],
"source": [
"# Delete custom train jobs.\n",
"train_job.delete()\n",
"\n",
"# Undeploy model and delete endpoint.\n",
"endpoint_without_peft.delete(force=True)\n",
"endpoint_with_peft.delete(force=True)\n",
"\n",
"# Delete models.\n",
"model_without_peft.delete()\n",
"model_with_peft.delete()\n",
"\n",
"# Delete Cloud Storage objects that were created\n",
"delete_bucket = False\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $EXPERIMENT_BUCKET"
]
}
],
"metadata": {
"colab": {
"name": "model_garden_pytorch_llama2_peft.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}