xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

add a sperate notebook for openllama peft to be more specific (#2106)

Browse Source
This commit is contained in:
genquan9
2023-07-21 15:33:08 +00:00
committed by GitHub
parent 1b6e663af0
commit d9058c2e4e
2 changed files with 611 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/community/CODEOWNERS
+1
View File
@@ -74,3 +74,4 @@
/notebooks/community/model_garden/model_garden_keras_stable_diffusion.ipynb @genquan9
/notebooks/community/model_garden/model_garden_pytorch_sam.ipynb @huguensjean
/notebooks/community/model_garden/model_garden_pytorch_peft.ipynb @genquan9
/notebooks/community/model_garden/model_garden_pytorch_openllama_peft.ipynb @genquan9
notebooks/community/model_garden/model_garden_pytorch_openllama_peft.ipynb
+610
View File
@@ -0,0 +1,610 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7d9bbf86da5e"
},
"outputs": [],
"source": [
"# Copyright 2023 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "99c1c3fc2ca5"
},
"source": [
"# Vertex AI Model Garden - OpenLLaMA (PEFT)\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_openllama_peft.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_openllama_peft.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_openllama_peft.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
"Open in Vertex AI Workbench\n",
" </a> (A Python-3 CPU notebook is recommended)\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3de7470326a2"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates deploying prebuilt OpenLLaMA, and also finetuning and deploying OpenLLaMA with performance efficient finetuning libraries ([PEFT](https://github.com/huggingface/peft)) in Vertex AI.\n",
"\n",
"### Objective\n",
"\n",
"- Deploy prebuilt OpenLLaMA\n",
"- Finetune and deploy OpenLLaMA with PEFT, supporting\n",
"\n",
"| Models | LoRA |\n",
"| :- | :- |\n",
"| [openlm-research/open_llama_3b](https://huggingface.co/openlm-research/open_llama_3b) | Y |\n",
"| [openlm-research/open_llama_7b](https://huggingface.co/openlm-research/open_llama_7b) | Y |\n",
"| [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) | Y |\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "264c07757582"
},
"source": [
"## Before you begin\n",
"\n",
"**NOTE**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ioensNKM8ned"
},
"source": [
"### Colab only\n",
"Run the following commands for Colab and skip this section if you are using Workbench."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2707b02ef5df"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" ! pip3 install --upgrade google-cloud-aiplatform\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
" # Install gdown for downloading example training images.\n",
" ! pip3 install gdown\n",
"\n",
" # Restart the notebook kernel after installs.\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bb7adab99e41"
},
"source": [
"### Setup Google Cloud project\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
"\n",
"1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs.\n",
"\n",
"1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console) with `Vertex AI User` and `Storage Object Admin` roles for deploying fine tuned model to Vertex AI endpoint."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6c460088b873"
},
"source": [
"Fill following variables for experiments environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "855d6b96f291"
},
"outputs": [],
"source": [
"# Cloud project id.\n",
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
"\n",
"# The region you want to launch jobs in.\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"\n",
"# The Cloud Storage bucket for storing experiments output.\n",
"BUCKET_URI = \"\" # @param {type:\"string\"}\n",
"\n",
"! gcloud config set project $PROJECT_ID\n",
"\n",
"import os\n",
"\n",
"STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n",
"EXPERIMENT_BUCKET = os.path.join(BUCKET_URI, \"peft\")\n",
"DATA_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"data\")\n",
"MODEL_BUCKET = os.path.join(EXPERIMENT_BUCKET, \"model\")\n",
"\n",
"# The service account looks like:\n",
"# '@.iam.gserviceaccount.com'\n",
"# Please go to https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console\n",
"# and create service account with `Vertex AI User` and `Storage Object Admin` roles.\n",
"# The service account for deploying fine tuned model.\n",
"SERVICE_ACCOUNT = \"\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e828eb320337"
},
"source": [
"### Initialize Vertex AI API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "12cd25839741"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2cc825514deb"
},
"source": [
"### Define constants"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b42bd4fa2b2d"
},
"outputs": [],
"source": [
"# The pre-built training and serving docker images.\n",
"TRAIN_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-peft-train\"\n",
"PREDICTION_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-peft-serve\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0c250872074f"
},
"source": [
"### Define common functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "354da31189dc"
},
"outputs": [],
"source": [
"import os\n",
"from datetime import datetime\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"\n",
"def get_job_name_with_datetime(prefix: str):\n",
" \"\"\"Gets the job name with date time when triggering training or deployment\n",
" jobs in Vertex AI.\n",
" \"\"\"\n",
" return prefix + datetime.now().strftime(\"_%Y%m%d_%H%M%S\")\n",
"\n",
"\n",
"def deploy_model(\n",
" model_name,\n",
" base_model_id,\n",
" finetuned_lora_model_path,\n",
" service_account,\n",
" task,\n",
" machine_type=\"n1-standard-8\",\n",
" accelerator_type=\"NVIDIA_TESLA_V100\",\n",
"):\n",
" \"\"\"Deploys trained models into Vertex AI.\"\"\"\n",
" endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
" serving_env = {\n",
" \"BASE_MODEL_ID\": base_model_id,\n",
" \"TASK\": task,\n",
" }\n",
" if finetuned_lora_model_path:\n",
" serving_env[\"FINETUNED_LORA_MODEL_PATH\"] = finetuned_lora_model_path\n",
" model = aiplatform.Model.upload(\n",
" display_name=model_name,\n",
" serving_container_image_uri=PREDICTION_DOCKER_URI,\n",
" serving_container_ports=[7080],\n",
" serving_container_predict_route=\"/predictions/peft_serving\",\n",
" serving_container_health_route=\"/ping\",\n",
" serving_container_environment_variables=serving_env,\n",
" )\n",
" model.deploy(\n",
" endpoint=endpoint,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=1,\n",
" deploy_request_timeout=1800,\n",
" service_account=service_account,\n",
" )\n",
" return model, endpoint"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8neJc8CnDDpu"
},
"source": [
"## Deploy Prebuilt OpenLLaMA\n",
"\n",
"This section deploys prebuilt OpenLLaMA models on the Endpoint. The model deployment step will take ~15 minutes to complete.\n",
"\n",
"The peak GPU memory usages for [openlm-research/open_llama_3b](https://huggingface.co/openlm-research/open_llama_3b), [openlm-research/open_llama_7b](https://huggingface.co/openlm-research/open_llama_7b), and [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) are ~5.3G, ~8.7G and ~15.2G separately with the default settings. We use V100 in deployments for simplicity."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2MjaORIIFDVu"
},
"source": [
"Set the prebuilt model id."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E8OiHHNNE_wj"
},
"outputs": [],
"source": [
"prebuilt_model_id = \"openlm-research/open_llama_3b\" # @param [\"openlm-research/open_llama_3b\", \"openlm-research/open_llama_7b\", \"openlm-research/open_llama_13b\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dHFW7yvjaVFV"
},
"source": [
"We use the PEFT serving images to deploy prebuilt OpenLLaMA models, by setting finetuning LoRA model paths as empty."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Uak1pyEeExYM"
},
"outputs": [],
"source": [
"model_without_peft, endpoint_without_peft = deploy_model(\n",
" model_name=get_job_name_with_datetime(prefix=\"openllama-serve\"),\n",
" base_model_id=prebuilt_model_id,\n",
" finetuned_lora_model_path=\"\", # This will avoid override finetuning models.\n",
" service_account=SERVICE_ACCOUNT,\n",
" task=\"causal-language-modeling-lora\",\n",
")\n",
"print(\"endpoint_name:\", endpoint_without_peft.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sGKIjgmDFRW2"
},
"source": [
"NOTE: The prebuilt model weights will be downloaded on the fly from the orginal location after the deployment succeeds. Thus additional 5 minutes of waiting time is needed **after** the above model deployment step succeeds and before you run the next step below. Otherwise you might see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint.\n",
"\n",
"Once deployment succeeds, you can send requests to the endpoint with text prompts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rDHsCOqvFYBi"
},
"outputs": [],
"source": [
"# # Loads an existing endpoint as below.\n",
"# endpoint_name = endpoint_without_peft.name\n",
"# aip_endpoint_name = (\n",
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
"# )\n",
"# endpoint_without_peft = aiplatform.Endpoint(aip_endpoint_name)\n",
"instances = [\n",
" {\"prompt\": \"Hi, Google.\"},\n",
"]\n",
"response = endpoint_without_peft.predict(instances=instances)\n",
"\n",
"for prediction in response.predictions[0]:\n",
" print(prediction[\"generated_text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e70e3519ff8b"
},
"source": [
"## Finetune and deploy OpenLLaMA with PEFT\n",
"\n",
"This section demonstrates how to finetune and dpeloy OpenLLaMA with PEFT LoRA."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5qCrm_kJH5cz"
},
"source": [
"Set the base model id."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N3UBLiYrM3sU"
},
"outputs": [],
"source": [
"base_model_id = \"openlm-research/open_llama_3b\" # @param [\"openlm-research/open_llama_3b\", \"openlm-research/open_llama_7b\", \"openlm-research/open_llama_13b\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iWGwJHqI7LMs"
},
"source": [
"### Finetune"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KKEYoRfiHDVv"
},
"source": [
"Use the Vertex AI SDK to create and run the custom training jobs with Vertex AI Model Garden training images.\n",
"\n",
"This example uses the dataset [Abirate/english_quotes](https://huggingface.co/datasets/Abirate/english_quotes).\n",
"\n",
"In order to make the finetuning efficiently, we enabled quantization (8bits) when loading pretrained models for finetuning LoRA models. The peak GPU memory usages are ~7G, ~10G and ~16G for finetuning LoRA models for [openlm-research/open_llama_3b](https://huggingface.co/openlm-research/open_llama_3b), [openlm-research/open_llama_7b](https://huggingface.co/openlm-research/open_llama_7b), and [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) separately with default training parameters and the example dataset. In theory, open_llama_3b and open_llama_7b can be finetuned on 1 V100, and open_llama_13b can be finetuned on 1 A100 (40G). We choose to use 1 A100 (40G) by default to support all these models in this notebook for simplicity."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "65467b361315"
},
"outputs": [],
"source": [
"dataset_name = \"Abirate/english_quotes\" # @param {type:\"string\"}\n",
"\n",
"# Worker pool spec.\n",
"# machine_type = \"n1-standard-8\"\n",
"# accelerator_type = \"NVIDIA_TESLA_V100\"\n",
"machine_type = \"a2-highgpu-1g\"\n",
"accelerator_type = \"NVIDIA_TESLA_A100\"\n",
"replica_count = 1\n",
"accelerator_count = 1\n",
"\n",
"# Setup training job.\n",
"job_name = get_job_name_with_datetime(\"openllama-lora-train\")\n",
"train_job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=job_name,\n",
" container_uri=TRAIN_DOCKER_URI,\n",
")\n",
"output_dir = os.path.join(MODEL_BUCKET, job_name)\n",
"output_dir_gcsfuse = output_dir.replace(\"gs://\", \"/gcs/\")\n",
"\n",
"# Pass training arguments and launch job.\n",
"train_job.run(\n",
" args=[\n",
" \"--task=causal-language-modeling-lora\",\n",
" f\"--pretrained_model_id={base_model_id}\",\n",
" f\"--dataset_name={dataset_name}\",\n",
" f\"--output_dir={output_dir_gcsfuse}\",\n",
" \"--lora_rank=16\",\n",
" \"--lora_alpha=32\",\n",
" \"--lora_dropout=0.05\",\n",
" \"--warmup_steps=10\",\n",
" \"--max_steps=10\",\n",
" \"--learning_rate=2e-4\",\n",
" ],\n",
" replica_count=replica_count,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" boot_disk_size_gb=500,\n",
")\n",
"\n",
"print(\"Trained models were saved in: \", output_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jqmCtkGnhDmp"
},
"source": [
"### Deploy\n",
"This section uploads the model to Model Registry and deploys it on the Endpoint.\n",
"\n",
"The model deployment step will take ~15 minutes to complete.\n",
"\n",
"The peak GPU memory usages for [openlm-research/open_llama_3b](https://huggingface.co/openlm-research/open_llama_3b), [openlm-research/open_llama_7b](https://huggingface.co/openlm-research/open_llama_7b), and [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) with LoRA weights are ~5.3G, ~8.7G and ~15.2G separately with the default settings. We use V100 in deployments for simplicity."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bf55e38815dc"
},
"outputs": [],
"source": [
"model_with_peft, endpoint_with_peft = deploy_model(\n",
" model_name=get_job_name_with_datetime(prefix=\"openllama-peft-serve\"),\n",
" base_model_id=base_model_id,\n",
" finetuned_lora_model_path=output_dir,\n",
" service_account=SERVICE_ACCOUNT,\n",
" task=\"causal-language-modeling-lora\",\n",
")\n",
"print(\"endpoint_name:\", endpoint_with_peft.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "80b3fd2ace09"
},
"source": [
"NOTE: After the deployment succeeds, the base model weights will be downloaded one the fly from the original location and LoRA model weights will be downloaded from the GCS bucket used in training above. Thus additional 5 minutes of waiting time is needed **after** the above model deployment step succeeds and before you run the next step below. Otherwise you might see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint.\n",
"\n",
"Once deployment succeeds, you can send requests to the endpoint with text prompts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4ab04da3ec9a"
},
"outputs": [],
"source": [
"# # Loads an existing endpoint as below.\n",
"# endpoint_name = endpoint_with_peft.name\n",
"# aip_endpoint_name = (\n",
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
"# )\n",
"# endpoint_with_peft = aiplatform.Endpoint(aip_endpoint_name)\n",
"instances = [\n",
" {\"prompt\": \"Hi, Google.\"},\n",
"]\n",
"response = endpoint_with_peft.predict(instances=instances)\n",
"\n",
"for prediction in response.predictions[0]:\n",
" print(prediction[\"generated_text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "af21a3cff1e0"
},
"source": [
"## Clean up resources"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "911406c1561e"
},
"outputs": [],
"source": [
"# Delete custom train jobs.\n",
"train_job.delete()\n",
"\n",
"# Undeploy model and delete endpoint.\n",
"endpoint_without_peft.delete(force=True)\n",
"endpoint_with_peft.delete(force=True)\n",
"\n",
"# Delete models.\n",
"model_without_peft.delete()\n",
"model_with_peft.delete()"
]
}
],
"metadata": {
"colab": {
"name": "model_garden_pytorch_openllama_peft.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}