mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
remove: remove model_garden_pytorch_mistral notebook (#3225)
Co-authored-by: Rayan Dasoriya <dasoriya@google.com>
This commit is contained in:
Rayan Dasoriya
co-authored by
Rayan Dasoriya
Rayan Dasoriya
parent
24ff289855
commit
d57617c726
@@ -1,883 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7d9bbf86da5e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2023 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "99c1c3fc2ca5"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI Model Garden - Mistral and Mixtral 8x7B Models\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
"Open in Vertex AI Workbench\n",
|
||||
" </a> (A Python-3 CPU notebook is recommended)\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3de7470326a2"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates deploying prebuilt [Mistral](https://mistral.ai/) and Mixtral 8x7B models in Vertex AI.\n",
|
||||
"\n",
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"- Deploy prebuilt [Mistral models](https://huggingface.co/mistralai) with [vLLM](https://github.com/vllm-project/vllm) containers\n",
|
||||
" - [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1): pretrained generative text model with 7 billion parameters\n",
|
||||
" - [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1): Instruction fine-tuned version of the Mistral-7B-v0.1 generative text model\n",
|
||||
" - [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2): Improved instruction fine-tuned version of Mistral-7B-Instruct-v0.1 supporting 32k context length\n",
|
||||
"- Deploy prebuit [Mixtral 8x7B model](https://huggingface.co/mistralai) with [vLLM](https://github.com/vllm-project/vllm) containers\n",
|
||||
" - [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1): pretrained Mixture of Experts (MoE) model with 8 branches\n",
|
||||
" - [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1): Instruction fine-tuned version of the Mixture of Experts (MoE) model with 8 branches\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), [Cloud NL API pricing](https://cloud.google.com/natural-language/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "264c07757582"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ioensNKM8ned"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only\n",
|
||||
"Run the following commands for Colab and skip this section if you are using Workbench."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2707b02ef5df"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
" ! pip3 install --upgrade google-cloud-aiplatform\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
" # Install gdown for downloading example training images.\n",
|
||||
" ! pip3 install gdown\n",
|
||||
"\n",
|
||||
" # Restart the notebook kernel after installs.\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "46d25fe73955"
|
||||
},
|
||||
"source": [
|
||||
"### Install dependencies"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c75c2c1fa6e0"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install transformers==4.36.0\n",
|
||||
"! pip3 install accelerate==0.23.0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "bb7adab99e41"
|
||||
},
|
||||
"source": [
|
||||
"### Setup Google Cloud project\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API, Compute Engine API and Cloud Natural Language API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,language.googleapis.com).\n",
|
||||
"\n",
|
||||
"1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs.\n",
|
||||
"\n",
|
||||
"1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console) with `Vertex AI User` and `Storage Object Admin` roles for deploying fine tuned model to Vertex AI endpoint."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6c460088b873"
|
||||
},
|
||||
"source": [
|
||||
"### Define environment variables\n",
|
||||
"\n",
|
||||
"Set the following variables for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the specified region (`REGION`). Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\")."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "855d6b96f291"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Cloud project id.\n",
|
||||
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# The region you want to launch jobs in.\n",
|
||||
"# Select region based on the accelerators and regions supported by Vertex AI Prediction\n",
|
||||
"# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
|
||||
"REGION = \"\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# The Cloud Storage bucket for storing experiments output.\n",
|
||||
"# Start with gs:// prefix, e.g. gs://foo_bucket.\n",
|
||||
"BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"! gcloud config set project $PROJECT_ID\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n",
|
||||
"\n",
|
||||
"# The service account looks like:\n",
|
||||
"# '@.iam.gserviceaccount.com'\n",
|
||||
"# Please go to https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console\n",
|
||||
"# and create service account with `Vertex AI User` and `Storage Object Admin` roles.\n",
|
||||
"# The service account for deploying fine tuned model.\n",
|
||||
"SERVICE_ACCOUNT = \"\" # @param {type:\"string\"}",
|
||||
"# HuggingFace access token.\n",
|
||||
"# Create a token at https://huggingface.co/settings/tokens.\n",
|
||||
"# Accept the model terms of service for the model you wish to use.\n",
|
||||
"HF_TOKEN = \"\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e828eb320337"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize Vertex AI API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "12cd25839741"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import aiplatform\n",
|
||||
"\n",
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2cc825514deb"
|
||||
},
|
||||
"source": [
|
||||
"### Define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b42bd4fa2b2d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The pre-built serving docker images with vLLM\n",
|
||||
"VLLM_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240313_0916_RC00\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0c250872074f"
|
||||
},
|
||||
"source": [
|
||||
"### Define common functions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "354da31189dc"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from datetime import datetime\n",
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_job_name_with_datetime(prefix: str) -> str:\n",
|
||||
" \"\"\"Gets the job name with date time when triggering training or deployment\n",
|
||||
" jobs in Vertex AI.\n",
|
||||
" \"\"\"\n",
|
||||
" return prefix + datetime.now().strftime(\"_%Y%m%d_%H%M%S\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def deploy_model_vllm(\n",
|
||||
" model_name: str,\n",
|
||||
" model_id: str,\n",
|
||||
" service_account: str,\n",
|
||||
" machine_type: str = \"g2-standard-8\",\n",
|
||||
" accelerator_type: str = \"NVIDIA_L4\",\n",
|
||||
" accelerator_count: int = 1,\n",
|
||||
" max_model_len: int = 4096,\n",
|
||||
" gpu_memory_utilization: float = 0.9,\n",
|
||||
" use_openai_server: bool = False,\n",
|
||||
" use_chat_completions_if_openai_server: bool = False,\n",
|
||||
" huggingface_token: str = \"\",\n",
|
||||
") -> Tuple[aiplatform.Model, aiplatform.Endpoint]:\n",
|
||||
" \"\"\"Deploys Mistral models with vLLM on Vertex AI.\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" model_name: Display name of the model.\n",
|
||||
" model_id: Model ID or path to model weights.\n",
|
||||
" service_account: Service account for model uploading and deployment.\n",
|
||||
" machine_type: Deployment machine type.\n",
|
||||
" accelerator_type: Deployment accelerator type.\n",
|
||||
" accelerator_count: Number of accelerators to use.\n",
|
||||
" max_model_len: Maximum model length.\n",
|
||||
" gpu_memory_utilization: Fraction of GPU memory to be used for the model\n",
|
||||
" executor.\n",
|
||||
" use_openai_server: Whether to use the OpenAI-format vLLM model server.\n",
|
||||
" use_chat_completions_if_openai_server: If the OpenAI model server is\n",
|
||||
" used, whether to use the chat completion API as opposed to the text\n",
|
||||
" completion API. The vLLM text completion API mimics the OpenAI text\n",
|
||||
" completion API:\n",
|
||||
" https://platform.openai.com/docs/api-reference/completions/create.\n",
|
||||
" It has two required parameters: the model ID to direct requests to\n",
|
||||
" and the prompt. The response includes a \"choices\" field that\n",
|
||||
" contains the generated text and a \"usage\" field that contains token\n",
|
||||
" counts. The vLLM chat completion API mimics the OpenAI chat\n",
|
||||
" completion API:\n",
|
||||
" https://platform.openai.com/docs/api-reference/chat/create. It has\n",
|
||||
" two required parameters: the model ID to direct requests to and\n",
|
||||
" \"messages\" which is a sequence of system/user/assistant/tool\n",
|
||||
" messages that can represent a multi-turn chat conversation. The\n",
|
||||
" response includes a \"choices\" field that contains the generated\n",
|
||||
" message from a role and a \"usage\" field that contains token counts.\n",
|
||||
" huggingface_token: Huggingface token for accessing the model.\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" Model instance and endpoint instance.\n",
|
||||
" \"\"\"\n",
|
||||
" endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
|
||||
"\n",
|
||||
" dtype = \"bfloat16\"\n",
|
||||
" if accelerator_type in [\"NVIDIA_TESLA_T4\", \"NVIDIA_TESLA_V100\"]:\n",
|
||||
" dtype = \"float16\"\n",
|
||||
"\n",
|
||||
" vllm_args = [\n",
|
||||
" \"--host=0.0.0.0\",\n",
|
||||
" \"--port=7080\",\n",
|
||||
" f\"--model={model_id}\",\n",
|
||||
" f\"--tensor-parallel-size={accelerator_count}\",\n",
|
||||
" \"--swap-space=16\",\n",
|
||||
" f\"--dtype={dtype}\",\n",
|
||||
" f\"--gpu-memory-utilization={gpu_memory_utilization}\",\n",
|
||||
" f\"--max-model-len={max_model_len}\",\n",
|
||||
" \"--disable-log-stats\",\n",
|
||||
" ]\n",
|
||||
" serving_env = {\n",
|
||||
" \"MODEL_ID\": model_id,\n",
|
||||
" \"DEPLOY_SOURCE\": \"notebook\",\n",
|
||||
" \"HF_TOKEN\": huggingface_token,\n",
|
||||
" }\n",
|
||||
" if use_openai_server:\n",
|
||||
" if use_chat_completions_if_openai_server:\n",
|
||||
" serving_container_predict_route = \"/v1/chat/completions\"\n",
|
||||
" else:\n",
|
||||
" serving_container_predict_route = \"/v1/completions\"\n",
|
||||
" else:\n",
|
||||
" serving_container_predict_route = \"/generate\"\n",
|
||||
" model = aiplatform.Model.upload(\n",
|
||||
" display_name=model_name,\n",
|
||||
" serving_container_image_uri=VLLM_DOCKER_URI,\n",
|
||||
" serving_container_command=[\n",
|
||||
" \"python\",\n",
|
||||
" \"-m\",\n",
|
||||
" (\n",
|
||||
" \"vllm.entrypoints.api_server\"\n",
|
||||
" if not use_openai_server\n",
|
||||
" else \"vllm.entrypoints.openai.api_server\"\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" serving_container_args=vllm_args,\n",
|
||||
" serving_container_ports=[7080],\n",
|
||||
" serving_container_predict_route=serving_container_predict_route,\n",
|
||||
" serving_container_health_route=\"/health\",\n",
|
||||
" serving_container_environment_variables=serving_env,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model.deploy(\n",
|
||||
" endpoint=endpoint,\n",
|
||||
" machine_type=machine_type,\n",
|
||||
" accelerator_type=accelerator_type,\n",
|
||||
" accelerator_count=accelerator_count,\n",
|
||||
" deploy_request_timeout=1800,\n",
|
||||
" service_account=service_account,\n",
|
||||
" )\n",
|
||||
" return model, endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "e057b5edcf81"
|
||||
},
|
||||
"source": [
|
||||
"## Run inferences locally with prebuilt Mistral and Mixtral models\n",
|
||||
"\n",
|
||||
"You will need at least 24GB of memory to run inference with Mistral-7B. You can run locally or on Vertex AI Prediction endpoints with any of the following specs:\n",
|
||||
"- g2-standard-8 with 1 L4 GPU\n",
|
||||
"- n1-standard-16 with 2 V100 GPUs\n",
|
||||
"- n1-standard-16 with 2 T4 GPUs\n",
|
||||
"- a2-highgpu-1g with 1 A100 GPU\n",
|
||||
"\n",
|
||||
"You will need at least 96GB of memory to run inference with Mixtral 8x7B. You can run locally or on Vertex AI Prediction endpoints with any of the following specs:\n",
|
||||
"- g2-standard-96 with 8 L4 GPUs\n",
|
||||
"- n1-standard-32 with 8 V100 GPUs\n",
|
||||
"- n1-standard-32 with 8 T4 GPUs\n",
|
||||
"- a2-highgpu-4g with 4 A100 GPUs"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "31f6cc84efdd"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"import torch\n",
|
||||
"import transformers\n",
|
||||
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
|
||||
"\n",
|
||||
"device = \"cuda\" # the device to load the model onto\n",
|
||||
"model_name = \"mistralai/Mistral-7B-v0.1\" # @param [\"mistralai/Mistral-7B-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.2\", \"mistralai/Mixtral-8x7B-v0.1\", \"mistralai/Mixtral-8x7B-Instruct-v0.1\"]\n",
|
||||
"model = AutoModelForCausalLM.from_pretrained(\n",
|
||||
" model_name, device_map=\"auto\", return_dict=True, torch_dtype=torch.float16\n",
|
||||
")\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
|
||||
"\n",
|
||||
"pipeline = transformers.pipeline(\"text-generation\", model=model, tokenizer=tokenizer)\n",
|
||||
"\n",
|
||||
"prompt = \"My favourite condiment is\"\n",
|
||||
"\n",
|
||||
"sequences = pipeline(\n",
|
||||
" prompt,\n",
|
||||
" max_length=200,\n",
|
||||
" do_sample=True,\n",
|
||||
" top_k=10,\n",
|
||||
" num_return_sequences=1,\n",
|
||||
" eos_token_id=tokenizer.eos_token_id,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"for seq in sequences:\n",
|
||||
" print(f\"Result: {seq['generated_text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "YKZ4CBJ2kYaW"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy Prebuilt Mistral model with vLLM\n",
|
||||
"\n",
|
||||
"This section deploys the prebuilt Mistral model with [vLLM](https://github.com/vllm-project/vllm) on a Vertex endpoint. The model deployment step will take ~15 minutes to complete.\n",
|
||||
"\n",
|
||||
"vLLM is a highly optimized LLM serving framework which can significantly increase serving throughput. The higher QPS you have, the more benefits you get using vLLM."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "25b5b3a44cf8"
|
||||
},
|
||||
"source": [
|
||||
"Set the prebuilt model id."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "10547af949fc"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prebuilt_model_id = \"mistralai/Mistral-7B-v0.1\" # @param [\"mistralai/Mistral-7B-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.2\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "03d504bcd60b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Find Vertex AI prediction supported accelerators and regions in\n",
|
||||
"# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
|
||||
"\n",
|
||||
"# Sets 1 L4 to deploy Mistral 7B.\n",
|
||||
"machine_type = \"g2-standard-8\"\n",
|
||||
"accelerator_type = \"NVIDIA_L4\"\n",
|
||||
"accelerator_count = 1\n",
|
||||
"\n",
|
||||
"# Sets 2 V100s to deploy Mistral 7B.\n",
|
||||
"# machine_type = \"n1-standard-16\"\n",
|
||||
"# accelerator_type = \"NVIDIA_TESLA_V100\"\n",
|
||||
"# accelerator_count = 2\n",
|
||||
"\n",
|
||||
"# Sets 2 T4s to deploy Mistral 7B.\n",
|
||||
"# machine_type = \"n1-standard-16\"\n",
|
||||
"# accelerator_type = \"NVIDIA_TESLA_T4\"\n",
|
||||
"# accelerator_count = 2\n",
|
||||
"\n",
|
||||
"# Sets 1 A100 (40G) to deploy Mistral 7B.\n",
|
||||
"# machine_type = \"a2-highgpu-1g\"\n",
|
||||
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
|
||||
"# accelerator_count = 1\n",
|
||||
"\n",
|
||||
"# Larger setting of `max-model-len` can lead to higher requirements on\n",
|
||||
"# `gpu-memory-utilization` and GPU configuration. Larger setting of\n",
|
||||
"# `gpu-memory-utilization` increases the risk of running out of GPU memory with\n",
|
||||
"# long prompts.\n",
|
||||
"max_model_len = 4096\n",
|
||||
"gpu_memory_utilization = 0.9\n",
|
||||
"\n",
|
||||
"model, endpoint = deploy_model_vllm(\n",
|
||||
" model_name=get_job_name_with_datetime(prefix=\"mistral-serve-vllm\"),\n",
|
||||
" model_id=prebuilt_model_id,\n",
|
||||
" service_account=SERVICE_ACCOUNT,\n",
|
||||
" machine_type=machine_type,\n",
|
||||
" accelerator_type=accelerator_type,\n",
|
||||
" accelerator_count=accelerator_count,\n",
|
||||
" max_model_len=max_model_len,\n",
|
||||
" gpu_memory_utilization=gpu_memory_utilization,\n",
|
||||
" use_openai_server=False,\n",
|
||||
" use_chat_completions_if_openai_server=False,\n",
|
||||
" huggingface_token=HF_TOKEN,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "RRR11SWykYaX"
|
||||
},
|
||||
"source": [
|
||||
"NOTE: If you see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint, the model server is likely still initializing. Please retry later.\n",
|
||||
"\n",
|
||||
"NOTE: If you receive `InternalServerError: 500 System error` during the deployment, most likely the operation failed due to unavailability of resources. Either retry or use a different accelerator type.\n",
|
||||
"\n",
|
||||
"Once deployment succeeds, you can send requests to the endpoint with text prompts."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3a7948c56e3d"
|
||||
},
|
||||
"source": [
|
||||
"### Run sample prompt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "3f5a1e1de60d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Loads an existing endpoint instance using the endpoint name:\n",
|
||||
"# - Using `endpoint_name = endpoint.name` allows us to get the endpoint name of\n",
|
||||
"# the endpoint `endpoint` created in the cell above.\n",
|
||||
"# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n",
|
||||
"# an existing endpoint with the ID 1234567890123456789.\n",
|
||||
"# You may uncomment the code below to load an existing endpoint.\n",
|
||||
"\n",
|
||||
"# endpoint_name = endpoint_without_peft.name\n",
|
||||
"# # endpoint_name = \"\" # @param {type:\"string\"}\n",
|
||||
"# aip_endpoint_name = (\n",
|
||||
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
|
||||
"# )\n",
|
||||
"# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
|
||||
"\n",
|
||||
"instances = [\n",
|
||||
" {\n",
|
||||
" \"prompt\": \"My favourite condiment is\",\n",
|
||||
" \"n\": 1,\n",
|
||||
" \"max_tokens\": 200,\n",
|
||||
" \"temperature\": 1.0,\n",
|
||||
" \"top_p\": 1.0,\n",
|
||||
" \"top_k\": 10,\n",
|
||||
" },\n",
|
||||
"]\n",
|
||||
"response = endpoint.predict(instances=instances)\n",
|
||||
"\n",
|
||||
"for prediction in response.predictions:\n",
|
||||
" print(prediction)\n",
|
||||
"\n",
|
||||
"# Reference the following code for using the OpenAI vLLM server.\n",
|
||||
"# import json\n",
|
||||
"# response = endpoint.raw_predict(\n",
|
||||
"# body=json.dumps({\n",
|
||||
"# \"model\": prebuilt_model_id,\n",
|
||||
"# \"prompt\": \"My favourite condiment is\",\n",
|
||||
"# \"n\": 1,\n",
|
||||
"# \"max_tokens\": 200,\n",
|
||||
"# \"temperature\": 1.0,\n",
|
||||
"# \"top_p\": 1.0,\n",
|
||||
"# \"top_k\": 10,\n",
|
||||
"# }),\n",
|
||||
"# headers={\"Content-Type\": \"application/json\"},\n",
|
||||
"# )\n",
|
||||
"# print(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "wOh9irbqJ-MM"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy Prebuilt Mixtral 8x7B model with vLLM\n",
|
||||
"\n",
|
||||
"This section deploys the prebuilt Mixtral 8x7B model with [vLLM](https://github.com/vllm-project/vllm) on a Vertex endpoint. The model deployment step will take ~40 minutes to complete.\n",
|
||||
"\n",
|
||||
"vLLM is a highly optimized LLM serving framework which can significantly increase serving throughput. The higher QPS you have, the more benefits you get using vLLM."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "h2uCSnoaJ-MM"
|
||||
},
|
||||
"source": [
|
||||
"Set the prebuilt model id."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-X42gkGYJ-MM"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prebuilt_model_id = \"mistralai/Mixtral-8x7B-v0.1\" # @param [\"mistralai/Mixtral-8x7B-v0.1\", \"mistralai/Mixtral-8x7B-Instruct-v0.1\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "M-YiJXT3J-MM"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Find Vertex AI prediction supported accelerators and regions in\n",
|
||||
"# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
|
||||
"\n",
|
||||
"# Sets 8 L4s to deploy Mixtral 8x7B.\n",
|
||||
"machine_type = \"g2-standard-96\"\n",
|
||||
"accelerator_type = \"NVIDIA_L4\"\n",
|
||||
"accelerator_count = 8\n",
|
||||
"\n",
|
||||
"# Sets 4 A100s (40G) to deploy Mixtral 8x7B.\n",
|
||||
"# machine_type = \"a2-highgpu-4g\"\n",
|
||||
"# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
|
||||
"# accelerator_count = 4\n",
|
||||
"\n",
|
||||
"# Larger setting of `max-model-len` can lead to higher requirements on\n",
|
||||
"# `gpu-memory-utilization` and GPU configuration. Larger setting of\n",
|
||||
"# `gpu-memory-utilization` increases the risk of running out of GPU memory with\n",
|
||||
"# long prompts.\n",
|
||||
"max_model_len = 4096\n",
|
||||
"gpu_memory_utilization = 0.85\n",
|
||||
"\n",
|
||||
"model, endpoint = deploy_model_vllm(\n",
|
||||
" model_name=get_job_name_with_datetime(prefix=\"mixtral-serve-vllm\"),\n",
|
||||
" model_id=prebuilt_model_id,\n",
|
||||
" service_account=SERVICE_ACCOUNT,\n",
|
||||
" machine_type=machine_type,\n",
|
||||
" accelerator_type=accelerator_type,\n",
|
||||
" accelerator_count=accelerator_count,\n",
|
||||
" max_model_len=max_model_len,\n",
|
||||
" gpu_memory_utilization=gpu_memory_utilization,\n",
|
||||
" use_openai_server=False,\n",
|
||||
" use_chat_completions_if_openai_server=False,\n",
|
||||
" huggingface_token=HF_TOKEN,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "agDw0_7JJ-MM"
|
||||
},
|
||||
"source": [
|
||||
"NOTE: If you see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint, the model server is likely still initializing. Please retry later.\n",
|
||||
"\n",
|
||||
"NOTE: If you receive `InternalServerError: 500 System error` during the deployment, most likely the operation failed due to unavailability of resources. Either retry or use a different accelerator type.\n",
|
||||
"\n",
|
||||
"Once deployment succeeds, you can send requests to the endpoint with text prompts."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "zLyWJK5aJ-MM"
|
||||
},
|
||||
"source": [
|
||||
"### Run sample prompt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "NYN1Z49SJ-MM"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Loads an existing endpoint instance using the endpoint name:\n",
|
||||
"# - Using `endpoint_name = endpoint.name` allows us to get the endpoint name of\n",
|
||||
"# the endpoint `endpoint` created in the cell above.\n",
|
||||
"# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n",
|
||||
"# an existing endpoint with the ID 1234567890123456789.\n",
|
||||
"# You may uncomment the code below to load an existing endpoint.\n",
|
||||
"\n",
|
||||
"# endpoint_name = endpoint_without_peft.name\n",
|
||||
"# # endpoint_name = \"\" # @param {type:\"string\"}\n",
|
||||
"# aip_endpoint_name = (\n",
|
||||
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
|
||||
"# )\n",
|
||||
"# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
|
||||
"\n",
|
||||
"instances = [\n",
|
||||
" {\n",
|
||||
" \"prompt\": \"What is a car?\",\n",
|
||||
" \"max_tokens\": 50,\n",
|
||||
" \"temperature\": 1.0,\n",
|
||||
" \"top_p\": 1.0,\n",
|
||||
" \"top_k\": 10,\n",
|
||||
" },\n",
|
||||
"]\n",
|
||||
"response = endpoint.predict(instances=instances)\n",
|
||||
"\n",
|
||||
"for prediction in response.predictions:\n",
|
||||
" print(prediction)\n",
|
||||
"\n",
|
||||
"# Reference the following code for using the OpenAI vLLM server.\n",
|
||||
"# import json\n",
|
||||
"# response = endpoint.raw_predict(\n",
|
||||
"# body=json.dumps({\n",
|
||||
"# \"model\": prebuilt_model_id,\n",
|
||||
"# \"prompt\": \"My favourite condiment is\",\n",
|
||||
"# \"n\": 1,\n",
|
||||
"# \"max_tokens\": 200,\n",
|
||||
"# \"temperature\": 1.0,\n",
|
||||
"# \"top_p\": 1.0,\n",
|
||||
"# \"top_k\": 10,\n",
|
||||
"# }),\n",
|
||||
"# headers={\"Content-Type\": \"application/json\"},\n",
|
||||
"# )\n",
|
||||
"# print(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NRaBADRM4JEn"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0a9da70d4abc"
|
||||
},
|
||||
"source": [
|
||||
"### Undeploy models and Delete endpoints"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e53749ae6b2c"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set this flag to delete endpoint including undeploying models\n",
|
||||
"delete_endpoint = False"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "65078f3ec44e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def list_endpoints():\n",
|
||||
" return [\n",
|
||||
" (r.name, r.display_name)\n",
|
||||
" for r in aiplatform.Endpoint.list()\n",
|
||||
" if r.display_name.startswith(\"mistral-serve-vllm\")\n",
|
||||
" or r.display_name.startswith(\"mixtral-serve-vllm\")\n",
|
||||
" ]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "cf56ac4cc73b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete the endpoint using the Vertex AI fully qualified identifier for the endpoint\n",
|
||||
"try:\n",
|
||||
" if delete_endpoint:\n",
|
||||
" endpoints = list_endpoints()\n",
|
||||
" for endpoint_id, endpoint_name in endpoints:\n",
|
||||
" endpoint = aiplatform.Endpoint(endpoint_id)\n",
|
||||
" print(\n",
|
||||
" f\"Undeploying all deployed models and deleting endpoint {endpoint_id} [{endpoint_name}]\"\n",
|
||||
" )\n",
|
||||
" endpoint.delete(force=True)\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d25a89a34b5e"
|
||||
},
|
||||
"source": [
|
||||
"### Delete Cloud Storage bucket"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "PAr4UWWx4JEo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"job.delete()\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"ID_TESTING\"):\n",
|
||||
" ! gsutil rm -rf {BUCKET_URI}"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "model_garden_pytorch_mistral.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
Reference in New Issue
Block a user