mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
Add and update existing vertex skills (#4467)
* Add and update existing vertex skills - Add support for fine tuning for 1p gemini tuning - Add support for deploying fine tuned model support - Add support for running inference on MaaS models - Add open model support for regions and cost estimating for 3p tuning * fixing some of the commit errors * updated scripts to use existing gemini 1.5 pro model * swap gemini 1.5 pro to gemini 2.5 pro
This commit is contained in:
vincentkt-google
@@ -0,0 +1,301 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
---
|
||||
name: Vertex AI Model Garden Deploy
|
||||
description: Deploy open models or custom weights to Vertex AI endpoints.
|
||||
---
|
||||
|
||||
# Vertex AI Model Garden Deploy Skill
|
||||
|
||||
This skill provides instructions for deploying Open Models from Vertex AI Model
|
||||
Garden to endpoints, and subsequently undeploying them to clean up resources.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Before deploying, ensure you have the correct project and region set. The
|
||||
commands below use placeholder variables `PROJECT_ID` and `LOCATION_ID`.
|
||||
|
||||
Ensure you are authenticated:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
gcloud config set project $PROJECT_ID`
|
||||
```
|
||||
|
||||
## 2. Discovering Deployable Models
|
||||
|
||||
You can list models available in Model Garden and check if they can be
|
||||
self-deployed.
|
||||
|
||||
```bash
|
||||
gcloud ai model-garden models list
|
||||
```
|
||||
|
||||
To see what machine types and accelerators are supported for a specific model
|
||||
(e.g., `google/gemma3@gemma-3-27b-it`):
|
||||
|
||||
```bash
|
||||
gcloud ai model-garden models list-deployment-config \
|
||||
--model="google/gemma3@gemma-3-27b-it"
|
||||
```
|
||||
|
||||
> [!NOTE] Some models, especially Hugging Face models, might require a Hugging
|
||||
> Face Access Token for deployment.
|
||||
|
||||
> [!TIP] **Model Recommendation Instructions:** If a user asks to deploy a model
|
||||
> but **does not specify which one**, you should recommend a model based on
|
||||
> their use case (e.g., Llama 3.3 70B for general purpose or Gemma 3 for
|
||||
> lightweight tasks). * You **MUST** ensure you are recommending the **latest
|
||||
> version** or **popular version** of the suggested model family. * You **MUST**
|
||||
> verify the model is currently deployable using `gcloud ai model-garden models
|
||||
> list` before suggesting it to the user.
|
||||
|
||||
## 3. Deploying a Model
|
||||
|
||||
> [!WARNING] Deploying models, especially large ones, consumes significant
|
||||
> compute resources and incurs costs. 1. You **MUST** refer to
|
||||
> [Vertex AI prediction pricing](https://cloud.google.com/vertex-ai/pricing#prediction-and-explanation)
|
||||
> to calculate a rough cost estimation based on the requested `--machine-type`
|
||||
> and `--accelerator-type` (and count). 2. You **MUST** present this cost
|
||||
> estimation to the user and warn them that this is the **list price**, which
|
||||
> may differ from their actual bill due to potential discounts or reservations.
|
||||
> 3. You **MUST ALWAYS** request explicit confirmation from the user agreeing to
|
||||
> the estimated cost before executing any `deploy` command.
|
||||
|
||||
To deploy a model, use the `deploy` command. It is highly recommended to use the
|
||||
`--asynchronous` flag for long-running deployments, and then poll the status if
|
||||
necessary.
|
||||
|
||||
### Example: Deploying Gemma 3
|
||||
|
||||
Here is a typical bash script to deploy a model. You can run this block
|
||||
directly.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Example script to deploy a model from Model Garden
|
||||
|
||||
PROJECT_ID=$(gcloud config get-value project)
|
||||
LOCATION_ID="us-central1" # Recommended default region
|
||||
MODEL_ID="google/gemma3@gemma-3-27b-it" # Replace with your chosen model ID
|
||||
|
||||
echo "Deploying model $MODEL_ID to project $PROJECT_ID in $LOCATION_ID..."
|
||||
|
||||
# Model Garden can automatically select the required hardware based on the list-deployment-config if hardware params are omitted.
|
||||
# Below is a comprehensive command with all supported parameters:
|
||||
gcloud ai model-garden models deploy \
|
||||
--project=$PROJECT_ID \
|
||||
--region=$LOCATION_ID \
|
||||
--model=$MODEL_ID \
|
||||
--machine-type="g2-standard-48" \
|
||||
--accelerator-type="NVIDIA_L4" \
|
||||
--accelerator-count=4 \
|
||||
--endpoint-display-name="my-gemma-deployment" \
|
||||
--hugging-face-access-token="YOUR_HF_TOKEN" \
|
||||
--reservation-affinity="reservation-affinity-type=specific-reservation,key=compute.googleapis.com/reservation-name,values=my-reservation" \
|
||||
--asynchronous
|
||||
|
||||
echo "Deployment initiated asynchronously."
|
||||
echo "Check the Google Cloud Console (Vertex AI -> Online Prediction) for status."
|
||||
```
|
||||
|
||||
### Example: Deploying Custom Weights
|
||||
|
||||
To deploy a model using custom weights, you can use the exact same `deploy`
|
||||
command. Instead of providing the model garden model ID, provide the Google
|
||||
Cloud Storage (GCS) URI to your custom weights folder in the `--model` flag.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Example script to deploy a model with custom weights from a GCS bucket
|
||||
|
||||
PROJECT_ID=$(gcloud config get-value project)
|
||||
LOCATION_ID="us-central1"
|
||||
# Replace with the gs:// URI pointing to your custom weights
|
||||
MODEL_GCS_URI="gs://your-bucket-name/path/to/custom-weights"
|
||||
|
||||
echo "Deploying custom model from $MODEL_GCS_URI to project $PROJECT_ID in $LOCATION_ID..."
|
||||
|
||||
gcloud ai model-garden models deploy \
|
||||
--project=$PROJECT_ID \
|
||||
--region=$LOCATION_ID \
|
||||
--model=$MODEL_GCS_URI \
|
||||
--machine-type="g2-standard-12" \
|
||||
--accelerator-type="NVIDIA_L4" \
|
||||
--endpoint-display-name="my-custom-model" \
|
||||
--asynchronous
|
||||
|
||||
echo "Deployment initiated asynchronously."
|
||||
```
|
||||
|
||||
## 4. Checking Deployment Status
|
||||
|
||||
When you deploy a model asynchronously using the `--asynchronous` flag, the
|
||||
`deploy` command will return an operation ID. You can use this ID to check the
|
||||
ongoing status of the deployment.
|
||||
|
||||
```bash
|
||||
gcloud ai operations describe YOUR_OPERATION_ID \
|
||||
--region=$LOCATION_ID
|
||||
```
|
||||
|
||||
> [!NOTE] As an agent, you can also offer to check the status of a deployment
|
||||
> for the user if they provide an operation ID or if they just initiated the
|
||||
> deployment with you.
|
||||
|
||||
Alternatively, you can list your endpoints to see if it shows up and check the
|
||||
Cloud Console under the "Online prediction" tab.
|
||||
|
||||
```bash
|
||||
gcloud ai endpoints list \
|
||||
--region=$LOCATION_ID
|
||||
```
|
||||
|
||||
Note: Large models (like Llama 3.1 8B or Gemma 27B) may take 15-20 minutes to
|
||||
fully deploy and start serving.
|
||||
|
||||
### Verifying Deployment
|
||||
|
||||
If the model is successfully deployed, verify by making a prediction call to
|
||||
test. Because Model Garden models are often deployed to Dedicated Endpoints, you
|
||||
shouldn't use `gcloud ai endpoints predict`. Instead, you must fetch the
|
||||
endpoint's dedicated DNS name and send a `curl` request.
|
||||
|
||||
> [!TIP] Ask the user to try using their own prompt to see the results.
|
||||
> Otherwise use the default.
|
||||
|
||||
Use the following script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
PROJECT_ID=$(gcloud config get-value project)
|
||||
LOCATION_ID="us-central1"
|
||||
ENDPOINT_ID="YOUR_ENDPOINT_ID"
|
||||
PROMPT=${1:-"Explain quantum computing in simple terms."}
|
||||
|
||||
echo "Fetching dedicated Endpoint DNS..."
|
||||
ENDPOINT_URL=$(gcloud ai endpoints describe $ENDPOINT_ID --project=$PROJECT_ID --region=$LOCATION_ID --format="value(dedicatedEndpointDns)")
|
||||
|
||||
if [ -z "$ENDPOINT_URL" ]; then
|
||||
echo "Error: Could not retrieve a dedicated endpoint URL. Verify your ENDPOINT_ID."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Sending prediction request to $ENDPOINT_URL..."
|
||||
curl -X POST \
|
||||
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://${ENDPOINT_URL}/v1beta1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/endpoints/${ENDPOINT_ID}/chat/completions" \
|
||||
-d '{
|
||||
"model": "'"$ENDPOINT_ID"'",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "'"$PROMPT"'"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## 5. Undeploying and Cleaning Up
|
||||
|
||||
To stop incurring charges, you must undeploy the model from the endpoint. This
|
||||
is a multi-step process if you don't already have the exact endpoint and
|
||||
deployed model IDs.
|
||||
|
||||
### Example: Finding and Undeploying a Model
|
||||
|
||||
Here is a bash script demonstrating how to find the IDs and undeploy the model.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Example script to undeploy a model
|
||||
|
||||
PROJECT_ID=$(gcloud config get-value project)
|
||||
LOCATION_ID="us-central1"
|
||||
# The model ID used during deployment (without the provider prefix sometimes, or exactly as listed in describe)
|
||||
# It's usually easier to find the specific ID via `gcloud ai models list`
|
||||
# For this example, let's assume we know the exact Endpoint ID and Deployed Model ID.
|
||||
|
||||
# 1. Find the Endpoint ID
|
||||
echo "Listing endpoints in $LOCATION_ID:"
|
||||
gcloud ai endpoints list --project=$PROJECT_ID --region=$LOCATION_ID
|
||||
|
||||
# (Assuming you extracted ENDPOINT_ID from the above output)
|
||||
# ENDPOINT_ID="your_endpoint_id"
|
||||
|
||||
# 2. Find the Deployed Model ID
|
||||
echo "Listing models in $LOCATION_ID to find model description:"
|
||||
gcloud ai models list --project=$PROJECT_ID --region=$LOCATION_ID
|
||||
|
||||
# (Assuming you found the specific MODEL_ID)
|
||||
# MODEL_ID="your_model_id"
|
||||
# gcloud ai models describe $MODEL_ID --project=$PROJECT_ID --region=$LOCATION_ID
|
||||
# (Extract the deployedModelId from the output)
|
||||
# DEPLOYED_MODEL_ID="your_deployed_model_id"
|
||||
|
||||
# 3. Undeploy
|
||||
# Uncomment and replace the variables below to actually perform the undeployment
|
||||
# echo "Undeploying model $DEPLOYED_MODEL_ID from endpoint $ENDPOINT_ID..."
|
||||
# gcloud ai endpoints undeploy-model $ENDPOINT_ID \
|
||||
# --project=$PROJECT_ID \
|
||||
# --region=$LOCATION_ID \
|
||||
# --deployed-model-id=$DEPLOYED_MODEL_ID
|
||||
#
|
||||
# echo "Model undeployed."
|
||||
|
||||
# 4. Delete Endpoint
|
||||
# echo "Deleting endpoint $ENDPOINT_ID..."
|
||||
# gcloud ai endpoints delete $ENDPOINT_ID \
|
||||
# --project=$PROJECT_ID \
|
||||
# --region=$LOCATION_ID \
|
||||
# --quiet
|
||||
# echo "Endpoint deleted."
|
||||
|
||||
# 5. Delete Model
|
||||
# echo "Deleting model $MODEL_ID..."
|
||||
# gcloud ai models delete $MODEL_ID \
|
||||
# --project=$PROJECT_ID \
|
||||
# --region=$LOCATION_ID \
|
||||
# --quiet
|
||||
# echo "Model deleted."
|
||||
```
|
||||
|
||||
> [!WARNING] Failing to undeploy a model will result in continuous charges for
|
||||
> the allocated compute resources, even if you are not sending prediction
|
||||
> requests. Always clean up after testing.
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
### Deployment Failure: Quota or Resource Exhausted
|
||||
|
||||
If your deployment fails (or stays in an error state) due to `QUOTA_EXCEEDED` or
|
||||
`RESOURCE_EXHAUSTED` errors, the specific hardware requested (e.g., `NVIDIA_L4`
|
||||
or `g2-standard-24`) is either not available in your chosen region or exceeds
|
||||
your project's quota limits.
|
||||
|
||||
**Solution:** Look closely at the error message returned. It will often
|
||||
recommend an alternative region or machine type that currently has availability.
|
||||
**Ask the user for confirmation** to retry the deployment using the suggested
|
||||
`--region` or `--machine-type` parameters.
|
||||
|
||||
> [!WARNING] If the alternative suggestions involve changing the machine type or
|
||||
> accelerator, you **MUST** recalculate the estimated cost using
|
||||
> [Vertex AI prediction pricing](https://cloud.google.com/vertex-ai/pricing#prediction-and-explanation),
|
||||
> warn the user about list prices versus actual billing, and get their explicit
|
||||
> confirmation for the new cost before retrying the deployment.
|
||||
@@ -0,0 +1,246 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
---
|
||||
name: Vertex AI GenAI Inference
|
||||
description: Instructions for connecting to and performing inference with Google Cloud Vertex AI GenAI models, including Gemini and OpenMaaS (Llama, DeepSeek, Qwen, etc.).
|
||||
---
|
||||
|
||||
# Vertex AI GenAI Inference Skill
|
||||
|
||||
This skill provides instructions for authenticating and connecting to Google Cloud Vertex AI to use Generative AI models. It covers both First-Party (Gemini) and Third-Party (OpenMaaS) models.
|
||||
|
||||
> [!TIP]
|
||||
> **Sample Scripts**: This skill includes fully functional sample scripts in the `scripts/` directory (e.g., `scripts/openmaas_openai_sdk.py`). When running these scripts, **ALWAYS** create and use a local virtual environment:
|
||||
> ```bash
|
||||
> python3 -m venv .venv && source .venv/bin/activate
|
||||
> pip install -r scripts/requirements.txt
|
||||
> ```
|
||||
>
|
||||
> **Verify All Scripts**: You can run all scripts at once to verify your setup:
|
||||
> ```bash
|
||||
> ./scripts/verify_all.sh
|
||||
> ```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **CRITICAL: Model IDs & Availability**
|
||||
> * **Gemini Models**: See [Gemini Models](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/migrate) for valid Model IDs and Regions.
|
||||
> * **OpenMaaS Models**: See [Use Open Models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/use-open-models) for Llama, DeepSeek, Qwen, etc.
|
||||
> * **Incomplete Lists**: The Model IDs listed in this skill are **examples only** and may be incomplete or outdated.
|
||||
> * **Action**: Always verify the Model ID and Region using the links above before generating code.
|
||||
|
||||
## 1. Authentication (CRITICAL)
|
||||
|
||||
Before running any code, ensure you are authenticated with Application Default Credentials (ADC) and have the necessary API enabled.
|
||||
|
||||
1. **Login**:
|
||||
```bash
|
||||
gcloud auth application-default login
|
||||
```
|
||||
2. **Enable API** (if not already enabled):
|
||||
```bash
|
||||
gcloud services enable aiplatform.googleapis.com
|
||||
```
|
||||
|
||||
## 2. Gemini Models
|
||||
|
||||
For Gemini models (e.g., `gemini-1.5-pro`, `gemini-2.0-flash`), the **GenAI SDK** (`google-genai`) is the **PREFERRED** method. The legacy `vertexai` SDK is still supported but GenAI SDK is recommended for new projects.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Preview Models (including Gemini 2.0)** are often **ONLY** available in the `global` region. Stable models are available in `us-central1` and other regions.
|
||||
|
||||
### Choosing the Right SDK
|
||||
|
||||
* **Gemini Models**: **GenAI SDK** (`google-genai`) is **PREFERRED**. Use OpenAI SDK for compatibility, or Legacy SDK (`vertexai`) if needed.
|
||||
* **OpenMaaS Models**: **OpenAI SDK** is **HIGHLY RECOMMENDED**. Use GenAI SDK or Legacy SDK if you have specific infrastructure requirements.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install google-genai
|
||||
```
|
||||
|
||||
### Python Example (GenAI SDK - Preferred)
|
||||
|
||||
See [`scripts/gemini_genai_sdk.py`](scripts/gemini_genai_sdk.py) for the complete code.
|
||||
|
||||
### Alternative: OpenAI SDK (Chat Completions)
|
||||
|
||||
Use the standard OpenAI SDK with the Vertex AI endpoint. This is great for cross-compatibility.
|
||||
|
||||
See [`scripts/gemini_openai_sdk.py`](scripts/gemini_openai_sdk.py) for the complete code.
|
||||
|
||||
### Legacy: Vertex AI SDK
|
||||
|
||||
The legacy `vertexai` SDK is still widely used but `google-genai` is preferred for new Gemini projects.
|
||||
|
||||
See [`scripts/gemini_vertexai_sdk.py`](scripts/gemini_vertexai_sdk.py) for the complete code.
|
||||
|
||||
**Documentation**: [Google GenAI SDK](https://github.com/googleapis/python-genai)
|
||||
|
||||
**Documentation**: [Vertex AI Gemini Models](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models)
|
||||
|
||||
## 3. OpenMaaS Models (Llama, DeepSeek, Qwen, etc.)
|
||||
|
||||
For OpenMaaS (Model-as-a-Service) models, the **HIGHLY RECOMMENDED** approach is to use the standard **OpenAI SDK** with a specific Vertex AI endpoint.
|
||||
|
||||
> [!WARNING]
|
||||
> While `GenerativeModel` *can* support some OpenMaaS models, it is **discouraged**. Use the OpenAI SDK for best compatibility (especially for Chat Completions).
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install openai google-auth
|
||||
```
|
||||
|
||||
### Authentication for OpenAI SDK
|
||||
|
||||
You **MUST** use a Google Cloud OAuth access token as the API key for the OpenAI SDK.
|
||||
|
||||
```python
|
||||
import google.auth
|
||||
from google.auth.transport.requests import Request
|
||||
|
||||
def get_gcp_access_token():
|
||||
creds, _ = google.auth.default()
|
||||
creds.refresh(Request())
|
||||
return creds.token
|
||||
|
||||
> [!NOTE]
|
||||
> Google Cloud access tokens typically expire after 1 hour. The `get_gcp_access_token()` function above retrieves a *fresh* token at the time it is called.
|
||||
> For long-running applications, you implement a refresh mechanism. See [Refresh the access token](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/openai-sdk-auth#refresh-token) for details.
|
||||
```
|
||||
|
||||
### Configuration (Base URL)
|
||||
|
||||
- **Global Endpoint** (Recommended for most models requiring global availability):
|
||||
`https://aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/global/endpoints/openapi`
|
||||
- **Regional Endpoint**:
|
||||
`https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/openapi`
|
||||
|
||||
### Python Example (OpenMaaS - Chat Completions)
|
||||
|
||||
See [`scripts/openmaas_openai_sdk.py`](scripts/openmaas_openai_sdk.py) for the complete code.
|
||||
|
||||
> [!TIP]
|
||||
> **Alternative: Environment Variables**
|
||||
> You can set environment variables in your shell instead of updating the code.
|
||||
> ```bash
|
||||
> export OPENAI_BASE_URL="https://aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/global/endpoints/openapi"
|
||||
> export OPENAI_API_KEY="$(gcloud auth print-access-token)"
|
||||
> ```
|
||||
> Then initialize the client without arguments: `client = OpenAI()`
|
||||
|
||||
### Python Example (OpenMaaS - Completions API)
|
||||
|
||||
The following models support the legacy Completions API: `zai-org/glm-5-maas`, `moonshotai/kimi-k2-thinking-maas`, `minimaxai/minimax-m2-maas`, `deepseek-ai/deepseek-v3.1-maas`, and `deepseek-ai/deepseek-v3.2-maas`.
|
||||
|
||||
```python
|
||||
response = client.completions.create(
|
||||
model="deepseek-ai/deepseek-v3.2-maas",
|
||||
prompt="Once upon a time",
|
||||
max_tokens=100
|
||||
)
|
||||
print(response.choices[0].text)
|
||||
```
|
||||
|
||||
### Python Example (OpenMaaS - Embeddings)
|
||||
|
||||
```python
|
||||
# Verify specific Embedding Model ID on Model Garden (e.g., intfloat/multilingual-e5-small)
|
||||
response = client.embeddings.create(
|
||||
model="intfloat/multilingual-e5-large-maas",
|
||||
input="The quick brown fox jumps over the lazy dog",
|
||||
)
|
||||
print(response.data[0].embedding)
|
||||
```
|
||||
|
||||
### Alternative: GenAI SDK
|
||||
|
||||
The `google-genai` SDK can also access OpenMaaS models via the `vertexai` backend.
|
||||
|
||||
See [`scripts/openmaas_genai_sdk.py`](scripts/openmaas_genai_sdk.py) for the complete code.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Model ID Format**: For GenAI SDK with OpenMaaS, you **MUST** use the full path: `publishers/PUBLISHER/models/MODEL` (e.g., `publishers/zai-org/models/glm-5-maas`).
|
||||
|
||||
### Legacy: Vertex AI SDK (OpenMaaS)
|
||||
|
||||
For OpenMaaS, you can also use `GenerativeModel` (if supported).
|
||||
|
||||
See [`scripts/openmaas_vertexai_sdk.py`](scripts/openmaas_vertexai_sdk.py) for the complete code.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Model ID Format**: For Vertex AI SDK with OpenMaaS, you **MUST** use the full path: `publishers/PUBLISHER/models/MODEL`.
|
||||
|
||||
### Model Reference & Availability
|
||||
|
||||
**Documentation**: [Use Open Models on Vertex AI](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/maas/use-open-models)
|
||||
|
||||
> [!TIP]
|
||||
> **Self-Deployment for Control**: If you need **dedicated hardware** (GPUs/TPUs), **guaranteed capacity**, or **specific regional placement** not offered by MaaS, you can **Self-Deploy** these models to Vertex AI Endpoints. Search for the model in Model Garden and click "Deploy" to select your machine type.
|
||||
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Finding Inference Examples**: The list above is a starting point. For the **definitive** inference snippets (especially for Chat Completions payload structure):
|
||||
> 1. Consult the [Use Open Models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/use-open-models) list.
|
||||
> 2. Click the link for your specific model (e.g., "DeepSeek-V3") to visit its **Model Garden** page.
|
||||
> 3. Look for the **"Sample Code"** or **"Use this model"** button on the Model Garden page to get the exact `curl` or Python code for that specific model version.
|
||||
|
||||
> [!NOTE]
|
||||
> This list is **INCOMPLETE**. See [Use Open Models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/use-open-models) for the full list of supported models.
|
||||
|
||||
| Model Family | Model ID Examples | Location | Notes |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Llama 4** | `meta/llama-4-maverick-17b-128e-instruct-maas` | `us-east5` | |
|
||||
| **Llama 4** | `meta/llama-4-scout-17b-16e-instruct-maas` | `us-east5` | |
|
||||
| **Llama 3.3** | `meta/llama-3.3-70b-instruct-maas` | `us-central1` | |
|
||||
| **DeepSeek** | `deepseek-ai/deepseek-v3.2-maas` | `global` | Global ONLY |
|
||||
| **DeepSeek** | `deepseek-ai/deepseek-v3.1-maas` | `us-west2` | US-West2 ONLY |
|
||||
| **DeepSeek** | `deepseek-ai/deepseek-r1-0528-maas` | `us-central1` | |
|
||||
| **Qwen 3** | `qwen/qwen3-coder-480b-a35b-instruct-maas` | `global` | |
|
||||
| **Qwen 3** | `qwen/qwen3-next-80b-a3b-instruct-maas` | `global` | |
|
||||
| **Kimi** | `moonshotai/kimi-k2-thinking-maas` | `global` | |
|
||||
| **MiniMax** | `minimaxai/minimax-m2-maas` | `global` | |
|
||||
| **GLM** | `zai-org/glm-4.7-maas`, `zai-org/glm-5-maas` | `global` | |
|
||||
|
||||
## 4. Troubleshooting & Common Error Codes
|
||||
|
||||
### 429: Resource Exhausted
|
||||
|
||||
* **Cause**: OpenMaaS and Gemini models use **Dynamic Shared Quota (DSQ)**. Resources are pooled and allocated dynamically based on availability. A 429 error indicates the shared pool is temporarily exhausted, not necessarily that *your* specific project quota is hit (though it can be).
|
||||
* **Solution**: Implement strict **exponential backoff and retry** strategies.
|
||||
* **High Throughput**: For production workloads requiring high throughput or guaranteed capacity, consider **Provisioned Throughput (PT)**.
|
||||
* **Important**: Quota increases through normal cloud processes (Cloud Console) are **NOT** applicable for DSQ constraints.
|
||||
* **Documentation**: [Quotas and limits (DSQ)](https://cloud.google.com/vertex-ai/generative-ai/docs/quotas)
|
||||
|
||||
### 400: User Validation Error
|
||||
|
||||
* **Cause**: Invalid request format, unsupported parameter, or incorrect Model ID.
|
||||
* **Action**: Double-check your request payload and parameters. Verify the Model ID and Region are correct.
|
||||
|
||||
### 404: Not Found / Model Not Available
|
||||
|
||||
* **Cause**: The model is not enabled, or not available in the specified project or region.
|
||||
* **Action**:
|
||||
1. **Check Location Availability**:
|
||||
* **OpenMaaS**: Verify the model is available in your region. See [Model Availability by Location](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#genai-open-models).
|
||||
* **Gemini**:
|
||||
* **Source of Truth**: Always check [Gemini Model Locations](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#google-models) for the authoritative list.
|
||||
* **Preview Models**: All Preview models (e.g., Gemini 2.0, experimental versions) are often **ONLY** available in the `us-central1` or `global` regions.
|
||||
* **Stable Models**: (e.g., Gemini 1.5 Pro) Available in `us-central1`, `europe-west4`, and many other regions.
|
||||
* **Important**: If you get a 404/400 error, try switching your client location to `us-central1` or `global`.
|
||||
2. **Enable Llama Models**: For **Llama 3.3** and **Llama 4**, you **MUST** enable the model in Model Garden before use. Go to the [Model Garden](https://console.cloud.google.com/vertex-ai/model-garden), search for the model card (e.g., "Llama 3.3 API Service"), and click **Enable**. Only then can you make inference requests.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from google import genai
|
||||
import google.auth
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
# Initialize GenAI Client with Vertex AI backend
|
||||
# Use location="global" for Preview models (Gemini 2.0)
|
||||
client = genai.Client(vertexai=True, project=project_id, location="us-central1")
|
||||
|
||||
response = client.models.generate_content(
|
||||
model="gemini-2.5-pro", contents="Why is the sky blue?"
|
||||
)
|
||||
print(response.text)
|
||||
@@ -0,0 +1,38 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import google.auth
|
||||
from google.auth.transport.requests import Request
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
def get_gcp_access_token():
|
||||
creds, _ = google.auth.default()
|
||||
creds.refresh(Request())
|
||||
return creds.token
|
||||
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
client = OpenAI(
|
||||
base_url=f"https://aiplatform.googleapis.com/v1/projects/{project_id}/locations/us-central1/endpoints/openapi",
|
||||
api_key=get_gcp_access_token(),
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemini-2.5-pro",
|
||||
messages=[{"role": "user", "content": "Why is the sky blue?"}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
@@ -0,0 +1,26 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import google.auth
|
||||
import vertexai
|
||||
from vertexai.generative_models import GenerativeModel
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
vertexai.init(project=project_id, location="us-central1")
|
||||
|
||||
model = GenerativeModel("gemini-2.5-pro")
|
||||
response = model.generate_content("Why is the sky blue?")
|
||||
print(response.text)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from google import genai
|
||||
import google.auth
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=project_id,
|
||||
location="global", # OpenMaaS models are often global
|
||||
)
|
||||
|
||||
# Note: For GenAI SDK/Vertex with OpenMaaS, you MUST use the full path: `publishers/PUBLISHER/models/MODEL`
|
||||
response = client.models.generate_content(
|
||||
model="publishers/zai-org/models/glm-5-maas",
|
||||
contents="Explain quantum computing in simple terms.",
|
||||
)
|
||||
print(response.text)
|
||||
@@ -0,0 +1,45 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import google.auth
|
||||
from google.auth.transport.requests import Request
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
def get_gcp_access_token():
|
||||
creds, _ = google.auth.default()
|
||||
creds.refresh(Request())
|
||||
return creds.token
|
||||
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
client = OpenAI(
|
||||
base_url=f"https://aiplatform.googleapis.com/v1/projects/{project_id}/locations/global/endpoints/openapi",
|
||||
api_key=get_gcp_access_token(),
|
||||
)
|
||||
|
||||
# NOTE: For OpenMaaS models, you MUST use the format: `publisher/model`
|
||||
# Example: `deepseek-ai/deepseek-v3.2-maas`, `zai-org/glm-5-maas` etc.
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/glm-5-maas",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Explain quantum computing in simple terms.",
|
||||
}],
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import google.auth
|
||||
import vertexai
|
||||
from vertexai.generative_models import GenerativeModel
|
||||
|
||||
# Get default project ID from environment
|
||||
_, project_id = google.auth.default()
|
||||
|
||||
vertexai.init(project=project_id, location="global")
|
||||
|
||||
# Important: Use the full resource path: `publishers/PUBLISHER/models/MODEL`
|
||||
model = GenerativeModel("publishers/zai-org/models/glm-5-maas")
|
||||
response = model.generate_content("Explain quantum computing.")
|
||||
print(response.text)
|
||||
@@ -0,0 +1,4 @@
|
||||
google-genai
|
||||
google-cloud-aiplatform
|
||||
openai
|
||||
google-auth
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Create a temporary directory for the virtual environment
|
||||
VENV_DIR=$(mktemp -d -t venv_verify.XXXXXX)
|
||||
python3 -m venv "$VENV_DIR"
|
||||
source "$VENV_DIR/bin/activate"
|
||||
|
||||
# Trap to ensure cleanup happens on exit
|
||||
cleanup() {
|
||||
echo "Cleaning up virtual environment..."
|
||||
deactivate 2>/dev/null || true
|
||||
rm -rf "$VENV_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "Installing requirements..."
|
||||
pip install -q -r scripts/requirements.txt
|
||||
|
||||
echo "Running verification tests..."
|
||||
FAILED=0
|
||||
|
||||
# Iterate directly over the files in the scripts directory
|
||||
for script in scripts/*.py; do
|
||||
echo "Running $script..."
|
||||
if python3 "$script" > /dev/null 2>&1; then
|
||||
echo " PASS: $script"
|
||||
else
|
||||
echo " FAIL: $script"
|
||||
python3 "$script" # Run again to show output
|
||||
FAILED=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $FAILED -eq 0 ]; then
|
||||
echo "All scripts passed verification!"
|
||||
exit 0
|
||||
else
|
||||
echo "Some scripts failed verification."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,72 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
# Vertex AI Model Tuning Skill
|
||||
|
||||
This skill allows agents to fine-tune Large Language Models (LLMs) using
|
||||
Vertex AI's managed tuning service. It encapsulates the workflow for data
|
||||
preparation, job submission, monitoring, and model deployment.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Clone the repo
|
||||
|
||||
Add instruction here with the correct repo handle
|
||||
|
||||
> git clone repo-link
|
||||
|
||||
### 2. Installing Agent Skills
|
||||
|
||||
Install the skill to your favorite AI assisted coding tool.
|
||||
|
||||
- [Antigravity](https://antigravity.google/docs/skills#:~:text=and%20follows%20them.-,Where%20skills%20live,-Antigravity%20supports%20two)
|
||||
- [Gemini CLI](https://geminicli.com/docs/cli/skills/#managing-skills)
|
||||
- [Claude Code](https://code.claude.com/docs/en/skills#where-skills-live)
|
||||
|
||||
#### Verification
|
||||
|
||||
To confirm the installation was successful ask:
|
||||
|
||||
> "What skills do you have?"
|
||||
|
||||
The agent should respond with a list including your newly added skill
|
||||
`vertex-tuning` alongside its default capabilities.
|
||||
|
||||
## Getting started
|
||||
|
||||
Try the following prompt to get started:
|
||||
|
||||
> "I want to fine-tune a Llama 3.1 8B model for text classification on Vertex.
|
||||
Can you help me with this?"
|
||||
|
||||
## Features
|
||||
|
||||
* **Data Preparation**: Converts and validates datasets (JSONL format) for Vertex AI.
|
||||
* **Model Tuning**: Submits tuning jobs for supported models (Llama, Gemma,
|
||||
Qwen, etc.) with customizable hyperparameters (PEFT/Full).
|
||||
* **Model Deployment**: Deploys tuned models to Vertex AI Endpoints for serving.
|
||||
* **Guidance**: Provides recommendations for models and hyperparameters based on the task and dataset.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
* `scripts/`: Python scripts for each stage of the workflow.
|
||||
* `prepare_dataset.py`: Converts, splits, and validates datasets.
|
||||
* `tune_model.py`: Submits the tuning job to Vertex AI.
|
||||
* `deploy_model.py`: Deploys the tuned model to an endpoint.
|
||||
* `references/`: Documentation and catalogs.
|
||||
* `models.md`: Supported models, hardware requirements, and hyperparameter baselines.
|
||||
* `data_prep.md`: Data formatting guidelines.
|
||||
* `tuning_guide.md`: Detailed tuning advice.
|
||||
+37
-200
@@ -1,212 +1,49 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
---
|
||||
name: vertex-tuning
|
||||
description: >
|
||||
Vertex AI Model Tuning. Use when you need to fine-tune models
|
||||
using Vertex AI's infrastructure.
|
||||
Vertex AI Model Tuning Router. Use this skill when the user wants to fine-tune
|
||||
models using Vertex AI. This skill routes to either vertex-tuning-open-model
|
||||
or vertex-tuning-gemini.
|
||||
---
|
||||
|
||||
# Vertex AI Model Tuning
|
||||
# Vertex AI Model Tuning (Router)
|
||||
|
||||
## Overview
|
||||
|
||||
This skill provides procedural knowledge for fine-tuning Large Language Models
|
||||
(LLMs) using Vertex AI's tuning service. It covers the entire lifecycle from
|
||||
environment setup and data preparation to job configuration, monitoring, and
|
||||
deployment.
|
||||
This skill acts as a router for Vertex AI tuning tasks. The tuning procedures
|
||||
for Open Models and Gemini Models differ significantly. Your first step is to
|
||||
determine which category the user intends to tune and then read the
|
||||
corresponding sub-skill.
|
||||
|
||||
## Workflow Decision Tree
|
||||
|
||||
1. **Environment Check**: Has the environment (Auth, APIs, IAM, Venv) been
|
||||
initialized?
|
||||
1. **Model Category**: Has the user explicitly stated whether they want to tune
|
||||
an **Open Model** or a **Gemini Model**?
|
||||
|
||||
- **No** → Go to [Phase 0: Environment & IAM Setup](#phase-0).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
2. **Dataset Status**: Is the dataset ready in JSONL format and uploaded to
|
||||
GCS?
|
||||
|
||||
- **No** → Go to [Phase 1: Dataset Preparation & Upload](#phase-1).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
3. **Configuration**: Have the target model and hyperparameters been decided?
|
||||
|
||||
- **No** → Go to [Phase 2: Recommendations & Configuration](#phase-2).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
4. **Job Status**: Has the tuning job been submitted?
|
||||
|
||||
- **No** → Go to
|
||||
[Phase 3: Tuning Job Execution](#phase-3-tuning-job-execution).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
5. **Job Completion**: Is the tuning job complete?
|
||||
|
||||
- **No** → Go to [Phase 4: Monitoring](#phase-4-monitoring).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
6. **Deployment**: Has the tuned model been deployed to an endpoint?
|
||||
|
||||
- **No** → Go to [Phase 5: Model Deployment](#phase-5-model-deployment).
|
||||
- **Yes** → Task Complete.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 0: Environment & IAM Setup {#phase-0}
|
||||
|
||||
Ensure the foundational environment is ready before proceeding.
|
||||
|
||||
### 0.1 Authentication & Project Context
|
||||
|
||||
- Check if `gcloud` CLI is installed. If it is not installed, prompt the user
|
||||
for permission to install it before proceeding.
|
||||
- Verify `gcloud auth list`. If not authenticated, run `gcloud auth login`.
|
||||
- Ensure `project` and `location` are known. Use `gcloud config get project`
|
||||
to retrieve the current project (and `gcloud config get compute/region` for
|
||||
region).
|
||||
- **CRITICAL: Ask for Confirmation.** You must prompt the user to confirm the
|
||||
retrieved project and region before proceeding, in case they want to switch
|
||||
to a different one.
|
||||
|
||||
### 0.2 Enable APIs
|
||||
|
||||
Ensure `aiplatform.googleapis.com` and `storage.googleapis.com` are enabled.
|
||||
`bash gcloud services enable aiplatform.googleapis.com storage.googleapis.com
|
||||
--project=YOUR_PROJECT`
|
||||
|
||||
### 0.3 IAM Permissions
|
||||
|
||||
Verify the following identities have the required roles.
|
||||
|
||||
- **Vertex AI Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`
|
||||
- **Managed OSS Fine Tuning Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com`
|
||||
- **User Identity**: The account running the commands.
|
||||
|
||||
### 0.4 Virtual Environment
|
||||
|
||||
Create and use a virtual environment named `tuning_agent_venv` in the home
|
||||
directory. Install dependencies from `references/requirements.txt`. `bash
|
||||
python3 -m venv ~/tuning_agent_venv source ~/tuning_agent_venv/bin/activate pip
|
||||
install -r references/requirements.txt`
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 1: Dataset Preparation & Upload {#phase-1}
|
||||
|
||||
Vertex AI requires valid JSONL format in GCS.
|
||||
|
||||
### 1.0 Dataset Discovery & Confirmation
|
||||
|
||||
- **Ask the User First:** Ask the user if they already have a dataset they
|
||||
want to use.
|
||||
- **Auto-Discovery:** If the user does not have a dataset, search the
|
||||
authenticated project's GCS buckets to find if any existing file has a
|
||||
reasonable dataset that can do the job the user prompted initially.
|
||||
- **CRITICAL: Ask for Confirmation.** Do not proceed with dataset preparation
|
||||
or upload until you present the found or provided dataset to the user and
|
||||
they confirm the dataset to use.
|
||||
|
||||
### 1.1 Formatting & Validation
|
||||
|
||||
- **Conversion**: If data is in CSV or JSON, use `vertex-tuning/scripts/prepare_dataset.py`
|
||||
to convert.
|
||||
- **Validation Split Confirmation**: If the user only provides a training
|
||||
dataset, **you must prompt the user** to seek permission to split the
|
||||
training dataset 80/20 to form a validation dataset (using
|
||||
`--validation_split 0.2`). If they agree, proceed with the split. If they
|
||||
decline, just use the training dataset without a validation dataset.
|
||||
- **Validation**: If data is already in JSONL, validate it before uploading:
|
||||
`bash python3 vertex-tuning/scripts/prepare_dataset.py
|
||||
\ --input my_data.jsonl \ --format messages \ --validate_only`
|
||||
- Refer to [Data Preparation Guide](references/data_prep.md) for required
|
||||
schemas.
|
||||
|
||||
### 1.2 Upload
|
||||
|
||||
Upload formatted `.jsonl` files to GCS using a unique directory (e.g., with a
|
||||
datetime timestamp) to avoid overwriting outputs from different runs.
|
||||
|
||||
```bash
|
||||
gcloud storage cp dataset.jsonl gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 2: Recommendations & Configuration {#phase-2}
|
||||
|
||||
Help the user choose the best model and parameters. **Always seek user
|
||||
confirmation before submitting the job.**
|
||||
|
||||
### 2.1 Model Selection
|
||||
|
||||
- If the user does not specify a model in their prompt, recommend a model
|
||||
based on the user's prompt by referencing the
|
||||
[Models Catalog](references/models.md).
|
||||
- **Prompt for Confirmation:** Present the recommended model to the user and
|
||||
ask for their confirmation.
|
||||
|
||||
### 2.2 Hyperparameters
|
||||
|
||||
- If the user does not specify hyperparameters, recommend `tuning_mode`,
|
||||
`epochs`, `learning_rate`, and `adapter_size` based on the
|
||||
[Tuning Guide](references/tuning_guide.md) and model-specific baselines in
|
||||
the [Models Catalog](references/models.md).
|
||||
- **Prompt for Confirmation:** Present the recommended hyperparameter
|
||||
configuration to the user and ask for their approval before proceeding to
|
||||
job submission.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 3: Tuning Job Execution
|
||||
|
||||
Submit the job using `scripts/tune_model.py`.
|
||||
|
||||
```bash
|
||||
python3 scripts/tune_model.py \
|
||||
--project YOUR_PROJECT \
|
||||
--location YOUR_LOCATION \
|
||||
--bucket YOUR_STAGING_BUCKET \
|
||||
--base_model BASE_MODEL_ID \
|
||||
--train_dataset gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl \
|
||||
--output_uri gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output \
|
||||
--epochs EPOCHS \
|
||||
--learning_rate LR \
|
||||
--tuning_mode MODE
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 4: Monitoring
|
||||
|
||||
Monitor the job via the Cloud Console link provided in the script output or by
|
||||
polling the job status.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 5: Model Deployment
|
||||
|
||||
Once the job is `SUCCEEDED`, deploy the model using `scripts/deploy_model.py`.
|
||||
|
||||
```bash
|
||||
python3 vertex_tuning/scripts/deploy_model.py \
|
||||
--project YOUR_PROJECT \
|
||||
--location YOUR_LOCATION \
|
||||
--artifacts_uri gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output/postprocess/node-0/checkpoints/final \
|
||||
--machine_type MACHINE_TYPE \
|
||||
--accelerator_type ACCELERATOR_TYPE \
|
||||
--accelerator_count COUNT
|
||||
```
|
||||
|
||||
Refer to [Models Catalog](references/models.md) for hardware recommendations.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Resources
|
||||
|
||||
- [Data Preparation Guide](references/data_prep.md)
|
||||
- [Models Catalog](references/models.md)
|
||||
- [Tuning Guide](references/tuning_guide.md)
|
||||
- `scripts/prepare_dataset.py`: Data conversion & validation.
|
||||
- `scripts/tune_model.py`: Job submission.
|
||||
- `scripts/deploy_model.py`: Model deployment.
|
||||
- **No** → **STOP**. Ask the user if they want to tune an Open Model or a
|
||||
Gemini Model. Do not proceed or recommend any specific models until this
|
||||
is confirmed.
|
||||
- **Yes** (Open Model) → The user wants to tune an Open Model. Stop
|
||||
reading this file and IMMEDIATELY read the skill instructions located at
|
||||
`open-model/SKILL.md`. Follow the instructions inside that skill to
|
||||
complete the task.
|
||||
- **Yes** (Gemini Model) → The user wants to tune a Gemini Model. Stop
|
||||
reading this file and IMMEDIATELY read the skill instructions located at
|
||||
`gemini/SKILL.md`. Follow the instructions inside that skill to complete
|
||||
the task.
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
---
|
||||
name: vertex-tuning-gemini
|
||||
description: >
|
||||
Vertex AI Gemini Model Tuning. Use when you need to fine-tune Gemini models
|
||||
using Vertex AI's infrastructure.
|
||||
---
|
||||
|
||||
# Vertex AI Gemini Model Tuning
|
||||
|
||||
## Overview
|
||||
|
||||
This skill provides procedural knowledge for fine-tuning Gemini Large Language
|
||||
Models using Vertex AI's tuning service. It covers the entire lifecycle from
|
||||
environment setup and data preparation to job configuration, monitoring, and
|
||||
deployment.
|
||||
|
||||
## Workflow Decision Tree
|
||||
|
||||
1. **Environment Check**: Has the environment (Auth, APIs, IAM, Venv) been
|
||||
initialized?
|
||||
|
||||
- **No** → Go to [Phase 0: Environment & IAM Setup](#phase-0).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
2. **Dataset Status**: Is the dataset ready in JSONL format and uploaded to
|
||||
GCS?
|
||||
|
||||
- **No** → Go to [Phase 1: Dataset Preparation & Upload](#phase-1).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
3. **Configuration**: Have the target Gemini model and hyperparameters been
|
||||
decided?
|
||||
|
||||
- **No** → Go to
|
||||
[Phase 2: Model Configuration & Recommendation](#phase-2).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
4. **Job Status**: Has the tuning job been submitted?
|
||||
|
||||
- **No** → Go to
|
||||
[Phase 3: Tuning Job Execution](#phase-3-tuning-job-execution).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
5. **Job Completion**: Is the tuning job complete?
|
||||
|
||||
- **No** → Go to [Phase 4: Monitoring](#phase-4-monitoring).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
6. **Deployment**: Has the tuned model been deployed (if required)?
|
||||
|
||||
- **No** → Go to [Phase 5: Model Deployment](#phase-5-model-deployment).
|
||||
- **Yes** → Task Complete.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 0: Environment & IAM Setup {#phase-0}
|
||||
|
||||
Ensure the foundational environment is ready before proceeding.
|
||||
|
||||
### 0.1 Authentication & Project Context
|
||||
|
||||
- Check if `gcloud` CLI is installed. If it is not installed, prompt the user
|
||||
for permission to install it before proceeding.
|
||||
- Verify `gcloud auth list`. If not authenticated, run `gcloud auth login`.
|
||||
- Ensure `project` and `location` are known. Use `gcloud config get project`
|
||||
to retrieve the current project (and `gcloud config get compute/region` for
|
||||
region).
|
||||
- **CRITICAL: Ask for Confirmation.** You must prompt the user to confirm the
|
||||
retrieved project and region before proceeding, in case they want to switch
|
||||
to a different one.
|
||||
|
||||
### 0.2 Enable APIs
|
||||
|
||||
Ensure `aiplatform.googleapis.com` and `storage.googleapis.com` are enabled.
|
||||
```bash
|
||||
gcloud services enable aiplatform.googleapis.com storage.googleapis.com --project=YOUR_PROJECT
|
||||
```
|
||||
|
||||
### 0.3 IAM Permissions
|
||||
|
||||
Verify the following identities have the required roles.
|
||||
|
||||
- **Vertex AI Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`
|
||||
- **Managed OSS Fine Tuning Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com`
|
||||
- **User Identity**: The account running the commands.
|
||||
|
||||
### 0.4 Virtual Environment
|
||||
|
||||
Create and use a virtual environment named `tuning_agent_venv` in the home
|
||||
directory. Install dependencies from `references/requirements.txt`.
|
||||
```bash
|
||||
python3 -m venv ~/tuning_agent_venv
|
||||
source ~/tuning_agent_venv/bin/activate
|
||||
pip install -r references/requirements.txt
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 1: Dataset Preparation & Upload {#phase-1}
|
||||
|
||||
Vertex AI requires valid JSONL format in GCS.
|
||||
|
||||
### 1.0 Dataset Discovery & Confirmation
|
||||
|
||||
- **Ask the User First:** Ask the user if they already have a dataset they
|
||||
want to use.
|
||||
- **Auto-Discovery:** If the user does not have a dataset, search the
|
||||
authenticated project's GCS buckets to find if any existing file has a
|
||||
reasonable dataset that can do the job the user prompted initially.
|
||||
- **CRITICAL: Ask for Confirmation.** Do not proceed with dataset preparation
|
||||
or upload until you present the found or provided dataset to the user and
|
||||
they confirm the dataset to use.
|
||||
|
||||
### 1.1 Formatting & Validation
|
||||
|
||||
- **Conversion**: If data is in CSV or JSON, use `scripts/prepare_dataset.py`
|
||||
to convert.
|
||||
- **Validation**: If data is already in JSONL, validate it before uploading:
|
||||
`bash python3 scripts/prepare_dataset.py \ --input my_data.jsonl \ --format
|
||||
messages_gemini \ --validate_only`
|
||||
- Refer to [Data Preparation Guide](references/data_prep.md) for required
|
||||
schemas.
|
||||
|
||||
### 1.2 Upload
|
||||
|
||||
Upload formatted `.jsonl` files to GCS using a unique directory (e.g., with a
|
||||
datetime timestamp) to avoid overwriting outputs from different runs.
|
||||
|
||||
```bash
|
||||
gcloud storage cp dataset.jsonl gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 2: Model Configuration & Recommendation {#phase-2}
|
||||
|
||||
Help the user choose the best Gemini model and parameters.
|
||||
**Always seek user confirmation before submitting the job.**
|
||||
|
||||
- If the user does not specify a specific model in their prompt, calculate
|
||||
recommendations based on the **Models Catalog**.
|
||||
- **Prompt for Confirmation:** Present the recommended model to the user and
|
||||
ask for their confirmation before configuring hyperparameters.
|
||||
|
||||
### 2.1 Configuration
|
||||
|
||||
- *Hyperparameter recommendation guidelines and options for Gemini models will
|
||||
be populated here.*
|
||||
- **Prompt for Confirmation:** Present the proposed configuration to the user
|
||||
and ask for their approval before proceeding.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 3: Tuning Job Execution {#phase-3-tuning-job-execution}
|
||||
|
||||
Submit the Gemini model tuning job using `scripts/tune_gemini_model.py`.
|
||||
|
||||
```bash
|
||||
python3 scripts/tune_gemini_model.py
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 4: Monitoring {#phase-4-monitoring}
|
||||
|
||||
Monitor the job via the Cloud Console link provided in the script output or by
|
||||
polling the job status.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 5: Model Deployment {#phase-5-model-deployment}
|
||||
|
||||
Once the Gemini model tuning job is `SUCCEEDED`, deploy the model using
|
||||
`scripts/deploy_gemini_model.py`.
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_gemini_model.py
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Resources
|
||||
|
||||
- [Data Preparation Guide](references/data_prep.md)
|
||||
- [Models Catalog](references/models.md)
|
||||
- [Tuning Guide](references/tuning_guide.md)
|
||||
- `scripts/prepare_dataset.py`: Data conversion & validation.
|
||||
- `scripts/tune_gemini_model.py`: Gemini model tuning job submission.
|
||||
- `scripts/deploy_gemini_model.py`: Gemini model deployment.
|
||||
@@ -0,0 +1,261 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
---
|
||||
name: vertex-tuning-open-model
|
||||
description: >
|
||||
Vertex AI Open Model Tuning. Use when you need to fine-tune open models
|
||||
using Vertex AI's infrastructure.
|
||||
---
|
||||
|
||||
# Vertex AI Open Model Tuning
|
||||
|
||||
## Overview
|
||||
|
||||
This skill provides procedural knowledge for fine-tuning Open Large Language
|
||||
Models (LLMs) using Vertex AI's tuning service. It covers the entire lifecycle
|
||||
from environment setup and data preparation to job configuration, monitoring,
|
||||
and deployment.
|
||||
|
||||
## Workflow Decision Tree
|
||||
|
||||
1. **Environment Check**: Has the environment (Auth, APIs, IAM, Venv) been
|
||||
initialized?
|
||||
|
||||
- **No** → Go to [Phase 0: Environment & IAM Setup](#phase-0).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
2. **Dataset Status**: Is the dataset ready in JSONL format and uploaded to
|
||||
GCS?
|
||||
|
||||
- **No** → Go to [Phase 1: Dataset Preparation & Upload](#phase-1).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
3. **Configuration**: Have the target open model and hyperparameters been
|
||||
decided?
|
||||
|
||||
- **No** → Go to
|
||||
[Phase 2: Model Configuration & Recommendation](#phase-2).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
4. **Job Status**: Has the tuning job been submitted?
|
||||
|
||||
- **No** → Go to
|
||||
[Phase 3: Tuning Job Execution](#phase-3-tuning-job-execution).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
5. **Job Completion**: Is the tuning job complete?
|
||||
|
||||
- **No** → Go to [Phase 4: Monitoring](#phase-4-monitoring).
|
||||
- **Yes** → Proceed.
|
||||
|
||||
6. **Deployment**: Has the tuned model been deployed (if required)?
|
||||
|
||||
- **No** → Go to [Phase 5: Model Deployment](#phase-5-model-deployment).
|
||||
- **Yes** → Task Complete.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 0: Environment & IAM Setup {#phase-0}
|
||||
|
||||
Ensure the foundational environment is ready before proceeding.
|
||||
|
||||
### 0.1 Authentication & Project Context
|
||||
|
||||
- Check if `gcloud` CLI is installed. If it is not installed, prompt the user
|
||||
for permission to install it before proceeding.
|
||||
- Verify `gcloud auth list`. If not authenticated, run `gcloud auth login`.
|
||||
- Ensure `project` and `location` are known. Use `gcloud config get project`
|
||||
to retrieve the current project (and `gcloud config get compute/region` for
|
||||
region).
|
||||
- **CRITICAL: Ask for Confirmation.** You must prompt the user to confirm the
|
||||
retrieved project and region before proceeding, in case they want to switch
|
||||
to a different one.
|
||||
|
||||
### 0.2 Possible Locations
|
||||
|
||||
The following locations are available for tuning:
|
||||
|
||||
- us-central1
|
||||
- europe-west4
|
||||
- us-west1
|
||||
- us-east5
|
||||
- asia-southeast1
|
||||
|
||||
No other values are supported for this section, ensure that the location is
|
||||
listed above.
|
||||
|
||||
### 0.3 Enable APIs
|
||||
|
||||
Ensure `aiplatform.googleapis.com` and `storage.googleapis.com` are enabled.
|
||||
```bash
|
||||
gcloud services enable aiplatform.googleapis.com storage.googleapis.com --project=YOUR_PROJECT
|
||||
```
|
||||
|
||||
### 0.4 IAM Permissions
|
||||
|
||||
Verify the following identities have the required roles.
|
||||
|
||||
- **Vertex AI Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`
|
||||
- **Managed OSS Fine Tuning Service Agent**:
|
||||
`service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com`
|
||||
- **User Identity**: The account running the commands.
|
||||
|
||||
### 0.5 Virtual Environment
|
||||
|
||||
Create and use a virtual environment named `tuning_agent_venv` in the home
|
||||
directory. Install dependencies from `references/requirements.txt`.
|
||||
```bash
|
||||
python3 -m venv ~/tuning_agent_venv
|
||||
source ~/tuning_agent_venv/bin/activate
|
||||
pip install -r references/requirements.txt
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 1: Dataset Preparation & Upload {#phase-1}
|
||||
|
||||
Vertex AI requires valid JSONL format in GCS.
|
||||
|
||||
### 1.0 Dataset Discovery & Confirmation
|
||||
|
||||
- **Ask the User First:** Ask the user if they already have a dataset they
|
||||
want to use.
|
||||
- **Auto-Discovery:** If the user does not have a dataset, search the
|
||||
authenticated project's GCS buckets to find if any existing file has a
|
||||
reasonable dataset that can do the job the user prompted initially.
|
||||
- **CRITICAL: Ask for Confirmation.** Do not proceed with dataset preparation
|
||||
or upload until you present the found or provided dataset to the user and
|
||||
they confirm the dataset to use.
|
||||
|
||||
### 1.1 Formatting & Validation
|
||||
|
||||
- **Conversion**: If data is in CSV or JSON, use `scripts/prepare_dataset.py`
|
||||
to convert.
|
||||
- **Validation Split Confirmation**: If the user only provides a training
|
||||
dataset, **you must prompt the user** to seek permission to split the
|
||||
training dataset 80/20 to form a validation dataset (using
|
||||
`--validation_split 0.2`). If they agree, proceed with the split. If they
|
||||
decline, just use the training dataset without a validation dataset.
|
||||
- **Validation**: If data is already in JSONL, validate it before uploading:
|
||||
`bash python3 scripts/prepare_dataset.py \ --input my_data.jsonl \ --format
|
||||
messages \ --validate_only`
|
||||
- Refer to [Data Preparation Guide](references/data_prep.md) for required
|
||||
schemas.
|
||||
|
||||
### 1.2 Upload
|
||||
|
||||
Upload formatted `.jsonl` files to GCS using a unique directory (e.g., with a
|
||||
datetime timestamp) to avoid overwriting outputs from different runs.
|
||||
|
||||
```bash
|
||||
gcloud storage cp dataset.jsonl gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 2: Model Configuration & Recommendation {#phase-2}
|
||||
|
||||
Help the user choose the best open model and parameters.
|
||||
**Always seek user confirmation before submitting the job.**
|
||||
|
||||
- If the user does not specify a specific model in their prompt, calculate
|
||||
recommendations based on the **Models Catalog**.
|
||||
- **Prompt for Confirmation:** Present the recommended model to the user and
|
||||
ask for their confirmation before configuring hyperparameters.
|
||||
|
||||
### 2.1 Configuration
|
||||
|
||||
- Recommend `tuning_mode`, `epochs`, `learning_rate`, and `adapter_size` based
|
||||
on the [Tuning Guide](references/tuning_guide.md) and model-specific
|
||||
baselines in the [Models Catalog](references/models.md).
|
||||
|
||||
### 2.2 Calculating Cost
|
||||
|
||||
- We can calculate a rough estimate of cost of tuning based on the dataset and
|
||||
the selected model in the [Models Catalog](references/models.md):
|
||||
```bash
|
||||
python3 \
|
||||
open-model/scripts/calculate_cost.py \
|
||||
--input my_data.jsonl \
|
||||
--model MODEL_NAME \
|
||||
--tuning_mode TUNING_MODE \
|
||||
--epochs EPOCHS
|
||||
```
|
||||
- **Prompt for Confirmation:** Present the recommended hyperparameter
|
||||
configuration and estimated cost to the user and ask for their approval
|
||||
before proceeding to job submission. Make sure to note that the estimated
|
||||
cost is just an estimate and can vary from actual billing costs.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 3: Tuning Job Execution {#phase-3-tuning-job-execution}
|
||||
|
||||
Submit the open model tuning job using `scripts/tune_open_model.py`. Identify
|
||||
the model id using available models documentation at
|
||||
[documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/open-model-tuning#supported-models).
|
||||
|
||||
```bash
|
||||
python3 scripts/tune_open_model.py \
|
||||
--project YOUR_PROJECT \
|
||||
--location YOUR_LOCATION \
|
||||
--bucket YOUR_STAGING_BUCKET \
|
||||
--base_model BASE_MODEL_ID \
|
||||
--train_dataset gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl \
|
||||
--output_uri gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output \
|
||||
--epochs EPOCHS \
|
||||
--learning_rate LR \
|
||||
--tuning_mode MODE
|
||||
```
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 4: Monitoring {#phase-4-monitoring}
|
||||
|
||||
Monitor the job via the Cloud Console link provided in the script output or by
|
||||
polling the job status.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Phase 5: Model Deployment {#phase-5-model-deployment}
|
||||
|
||||
Once the open model tuning job is `SUCCEEDED`, deploy the model using
|
||||
`scripts/deploy_open_model.py`.
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_open_model.py \
|
||||
--project YOUR_PROJECT \
|
||||
--location YOUR_LOCATION \
|
||||
--artifacts_uri gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output/postprocess/node-0/checkpoints/final \
|
||||
--machine_type MACHINE_TYPE \
|
||||
--accelerator_type ACCELERATOR_TYPE \
|
||||
--accelerator_count COUNT
|
||||
```
|
||||
|
||||
Refer to [Models Catalog](references/models.md) for hardware recommendations for
|
||||
specific open models.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Resources
|
||||
|
||||
- [Data Preparation Guide](references/data_prep.md)
|
||||
- [Models Catalog](references/models.md)
|
||||
- [Tuning Guide](references/tuning_guide.md)
|
||||
- `scripts/prepare_dataset.py`: Data conversion & validation.
|
||||
- `scripts/tune_open_model.py`: Open model tuning job submission.
|
||||
- `scripts/deploy_open_model.py`: Open model deployment.
|
||||
@@ -0,0 +1,66 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
# Data Preparation for Vertex AI Model Tuning
|
||||
|
||||
Vertex AI Model Tuning requires training data in **JSON Lines (JSONL)** format
|
||||
stored in Google Cloud Storage (GCS).
|
||||
|
||||
## Supported JSONL Formats for Open Models
|
||||
|
||||
### 1. Conversational (Messages) Format
|
||||
Recommended for chat-based models (Llama 3.1/3.2/3.3 Chat, Gemma 3 IT, etc.).
|
||||
|
||||
```json
|
||||
{
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is the capital of France?"},
|
||||
{"role": "assistant", "content": "The capital of France is Paris."}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Instruction (Prompt/Completion) Format
|
||||
Suitable for base models or simple completion tasks.
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": "Summarize the following text: [TEXT]",
|
||||
"completion": "[SUMMARY]"
|
||||
}
|
||||
```
|
||||
|
||||
## Dataset Requirements
|
||||
|
||||
- **File Type**: Must be `.jsonl`.
|
||||
- **Encoding**: UTF-8.
|
||||
- **Location**: Must be in a GCS bucket (e.g., `gs://my-bucket/train.jsonl`).
|
||||
- **Validation Split**: A separate validation file is optional but recommended. It must be no more than 25% of the training dataset size.
|
||||
|
||||
## Bucket Considerations
|
||||
|
||||
If a bucket does not exist, create one in the same region as your tuning job:
|
||||
|
||||
```bash
|
||||
gcloud storage buckets create gs://YOUR_BUCKET_NAME --location=YOUR_LOCATION
|
||||
```
|
||||
|
||||
## Formatting Best Practices
|
||||
|
||||
1. **Quality over Quantity**: 100 high-quality examples often outperform 1,000 noisy ones.
|
||||
2. **Consistency**: Use consistent formatting for system prompts and instruction styles.
|
||||
3. **No Empty Values**: Ensure every example has a valid prompt/user message and completion/assistant response. Use the [preparation script](../scripts/prepare_dataset.py) to validate this.
|
||||
@@ -0,0 +1,76 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
# Vertex AI Supported Models and Recommendations
|
||||
|
||||
This reference catalog provides technical specifications, tuning
|
||||
recommendations, and deployment hardware requirements for supported models in
|
||||
Vertex AI.
|
||||
|
||||
## Supported Models Catalog
|
||||
|
||||
> [!WARNING] **CRITICAL AGENT INSTRUCTION**
|
||||
> Do NOT use this catalog to recommend a specific model to the user until they
|
||||
> have explicitly confirmed their **Model Category** as Open Model.
|
||||
|
||||
Available open models can be found in Google Cloud [documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/open-model-tuning#supported-models).
|
||||
This is the list of open models that are available for tuning; do not suggest
|
||||
any other open models besides the one listed here.
|
||||
Each model has some [limitations](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/open-model-tuning#limitations) for tuning.
|
||||
|
||||
## Model Selection Guidelines
|
||||
|
||||
**Identify Task**: Check a few samples from the dataset to identify the task.
|
||||
|
||||
Choose a model family based on your task type:
|
||||
|
||||
- **Qwen**: Best for code generation or complex math-based tasks.
|
||||
- **Gemma**: Optimized for chat-based interactions, creative writing and multilingual tasks.
|
||||
- **Llama (Instruct)**: Strong general-purpose chat/instruction models.
|
||||
- **Llama (Base/Scout)**: Best for continuation tasks or building custom instruction-tuned models.
|
||||
|
||||
**Complexity Heuristics**:
|
||||
|
||||
- **Simple (QA, Extraction)**: 1B - 3B models.
|
||||
- **Intermediate (Summarization, Reasoning)**: 8B - 17B models.
|
||||
- **Complex (Multi-turn, Tool use, Deep reasoning)**: 27B - 70B models.
|
||||
|
||||
## Baseline Hyperparameter Recommendations
|
||||
|
||||
These values are starting points and should be adjusted based on your dataset
|
||||
size.
|
||||
|
||||
| Model | Tuning Mode | Learning Rate | Epochs | Adapter Size (PEFT) |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| Gemma 3 1B IT | Full | 2.0E-5 | 3 | N/A |
|
||||
| Gemma 3 4B IT | Full | 1.0E-5 | 3 | N/A |
|
||||
| Gemma 3 12B IT | Full | 1.0E-5 | 3 | N/A |
|
||||
| Gemma 3 27B IT | PEFT | 2.0E-4 | 3 | 32 |
|
||||
| Gemma 3 27B IT | Full | 2.0E-4 | 3 | N/A |
|
||||
| Llama 3.1 8B | PEFT | 2.0E-4 | 3 | 16 |
|
||||
| Llama 3.1 8B | Full | 2.0E-4 | 3 | N/A |
|
||||
| Llama 3.1 8B Instruct | PEFT | 2.0E-4 | 3 | 16 |
|
||||
| Llama 3.1 8B Instruct | Full | 2.0E-4 | 3 | N/A |
|
||||
| Llama 3.2 1B Instruct | Full | 1.5E-6 | 3 | N/A |
|
||||
| Llama 3.2 3B Instruct | Full | 1.0E-7 | 3 | N/A |
|
||||
| Llama 3.3 70B Instruct | PEFT | 5.0E-5 | 3 | 16 |
|
||||
| Llama 3.3 70B Instruct | Full | 5.0E-5 | 3 | N/A |
|
||||
| Llama 4 Scout 17B 16E | PEFT | 2.0E-5 | 3 | 16 |
|
||||
| Qwen 3 4B | Full | 7.5e-5 | 3 | N/A |
|
||||
| Qwen 3 8B | Full | 5e-5 | 3 | N/A |
|
||||
| Qwen 3 14B | Full | 4e-5 | 3 | N/A |
|
||||
| Qwen 3 32B | PEFT | 2.0E-4 | 3 | 16 |
|
||||
| Qwen 3 32B | Full | 2.5e-5 | 3 | N/A |
|
||||
@@ -0,0 +1,5 @@
|
||||
google-cloud-aiplatform==1.138.0
|
||||
numpy==2.4.2
|
||||
pandas==3.0.1
|
||||
datasets==2.18.0
|
||||
smart_open[gcs]==7.5.1
|
||||
@@ -0,0 +1,55 @@
|
||||
<!--
|
||||
Copyright 2026 Google LLC
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
# Vertex AI Model Tuning Heuristics and Concepts
|
||||
|
||||
This guide details the core concepts of fine-tuning and provides heuristics for
|
||||
adjusting hyperparameters based on your specific dataset.
|
||||
|
||||
## Core Tuning Concepts
|
||||
|
||||
### Open Models Tuning Modes
|
||||
|
||||
- **FULL**: Updates all parameters of the model. Requires more GPU memory and a larger dataset to avoid catastrophic forgetting.
|
||||
- **PEFT_ADAPTER**: Parameter-Efficient Fine-Tuning. Only a small set of "adapter" weights are trained. Faster, uses less memory, and is less prone to overfitting on small datasets.
|
||||
|
||||
### Hyperparameters
|
||||
|
||||
- **Epochs**: Number of times the model sees the entire dataset.
|
||||
- **Learning Rate**: Step size for optimization. Too high can cause instability; too low can lead to very slow convergence.
|
||||
- **Adapter Size (Rank)**: For PEFT, this determines the capacity of the adapters. Higher rank allows more complex learning but increases the risk of overfitting.
|
||||
|
||||
## Dataset Heuristics
|
||||
|
||||
The size and quality of your dataset should dictate your parameter choices. Refer to [Models Catalog](models.md) for baseline values, then adjust as follows:
|
||||
|
||||
### 1. Dataset Size Implications
|
||||
|
||||
| Dataset Size | Tuning Mode Recommendation | Learning Rate Adjustment | Epochs Recommendation |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **< 100 examples** | PEFT_ADAPTER (Rank 8) | Lower than baseline | 1-2 |
|
||||
| **100 - 1000 examples** | PEFT_ADAPTER (Rank 16/32) | Baseline | 3 |
|
||||
| **> 1000 examples** | FULL or PEFT (Rank 32) | Higher than baseline | 3-5 |
|
||||
|
||||
### 2. General Best Practices
|
||||
|
||||
- **Overfitting**: If validation loss starts increasing while training loss decreases, you are overfitting. Reduce epochs or decrease the learning rate.
|
||||
- **Underfitting**: If both training and validation loss remain high, increase the learning rate or use more epochs.
|
||||
- **Validation**: Always use a validation set to monitor performance. If not provided, a 10-20% split is highly recommended.
|
||||
- **Checkpoints**: The final model is always saved to `<output_uri>/postprocess/node-0/checkpoints/final`.
|
||||
|
||||
## Hardware and Limitations
|
||||
For specific hardware recommendations and sequence length limits per model, please refer to the [Models Catalog](models.md).
|
||||
@@ -0,0 +1,169 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Calculate tuning cost for a given dataset and model."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import smart_open
|
||||
|
||||
# Data from
|
||||
# https://docs.google.com/spreadsheets/d/1pOXzfQBSCaKJYcemvRKv4b30qmUBx3yG28yScH-pnVI/edit?resourcekey=0-0kJGshytd3yrxB41YM4OFg&gid=0#gid=0
|
||||
MODEL_DATA = {
|
||||
'Gemma 3 1B IT': {
|
||||
'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 0.47},
|
||||
},
|
||||
'Gemma 3 4B IT': {
|
||||
'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 1.14},
|
||||
},
|
||||
'Gemma 3 12B IT': {
|
||||
'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 1.82},
|
||||
},
|
||||
'Gemma 3 27B IT': {
|
||||
'PEFT': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 6.83},
|
||||
'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 6.83},
|
||||
},
|
||||
'Llama 3.1 8B': {
|
||||
'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 0.67},
|
||||
'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.67},
|
||||
},
|
||||
'Llama 3.1 8B Instruct': {
|
||||
'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 0.67},
|
||||
'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.67},
|
||||
},
|
||||
'Llama 3.2 1B Instruct': {
|
||||
'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.28},
|
||||
},
|
||||
'Llama 3.2 3B Instruct': {
|
||||
'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.61},
|
||||
},
|
||||
'Llama 3.3 70B Instruct': {
|
||||
'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 6.72},
|
||||
'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 6.72},
|
||||
},
|
||||
'Llama 4 Scout 17B 16E': {
|
||||
'PEFT': {'tokens_per_character': 0.295, 'cost_per_1m_tokens': 5.77},
|
||||
},
|
||||
'Qwen 3 4B': {
|
||||
'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 1.35},
|
||||
},
|
||||
'Qwen 3 8B': {
|
||||
'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 4.18},
|
||||
},
|
||||
'Qwen 3 14B': {
|
||||
'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 8.46},
|
||||
},
|
||||
'Qwen 3 32B': {
|
||||
'PEFT': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 6.57},
|
||||
'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 6.57},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def count_characters(input_file: str) -> int:
|
||||
"""Counts the characters in a jsonl dataset.
|
||||
|
||||
It is expected that each line in the jsonl file is a json object
|
||||
with a "messages" key, which is a list of dictionaries. Each
|
||||
dictionary in the "messages" list should have a "content" key.
|
||||
This function counts the characters in the "content" field of each
|
||||
dictionary in the "messages" list.
|
||||
|
||||
Args:
|
||||
input_file: Path to the input jsonl file.
|
||||
|
||||
Returns:
|
||||
Total character count.
|
||||
"""
|
||||
total_character_count = 0
|
||||
if not input_file.startswith('gs://') and '://' in input_file:
|
||||
raise ValueError(
|
||||
f'Unsupported file path: {input_file}. '
|
||||
'Only local paths and gs:// paths are supported.'
|
||||
)
|
||||
with smart_open.smart_open(input_file, 'r') as f:
|
||||
for line in f:
|
||||
data = json.loads(line)
|
||||
for message in data['messages']:
|
||||
content = message['content']
|
||||
total_character_count += len(content)
|
||||
return total_character_count
|
||||
|
||||
|
||||
def calculate_cost(
|
||||
count: int,
|
||||
model: str,
|
||||
tuning_mode: str,
|
||||
epochs: int,
|
||||
) -> float:
|
||||
"""Calculates the tuning cost.
|
||||
|
||||
Args:
|
||||
count: Total character count of the dataset.
|
||||
model: Model to use for tuning.
|
||||
tuning_mode: Tuning mode.
|
||||
epochs: Number of epochs.
|
||||
|
||||
Returns:
|
||||
Estimated tuning cost.
|
||||
"""
|
||||
model_data = MODEL_DATA[model][tuning_mode]
|
||||
tokens_per_character = model_data['tokens_per_character']
|
||||
cost_per_1m_tokens = model_data['cost_per_1m_tokens']
|
||||
num_tokens = count * tokens_per_character * epochs
|
||||
return (num_tokens / 1000000) * cost_per_1m_tokens
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Calculate tuning cost for a given dataset and model.'
|
||||
)
|
||||
parser.add_argument('--input_file', help='Input jsonl file.', required=True)
|
||||
parser.add_argument(
|
||||
'--model',
|
||||
help='Model to use for tuning.',
|
||||
required=True,
|
||||
choices=MODEL_DATA.keys(),
|
||||
)
|
||||
parser.add_argument(
|
||||
'--tuning_mode',
|
||||
help='Tuning mode.',
|
||||
required=True,
|
||||
choices=['PEFT', 'Full'],
|
||||
)
|
||||
parser.add_argument(
|
||||
'--epochs',
|
||||
help='Number of epochs.',
|
||||
required=True,
|
||||
type=int,
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if (
|
||||
args.model not in MODEL_DATA
|
||||
or args.tuning_mode not in MODEL_DATA[args.model]
|
||||
):
|
||||
print(
|
||||
f'Error: Tuning mode {args.tuning_mode} not supported for model'
|
||||
f' {args.model}'
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
character_count = count_characters(args.input_file)
|
||||
cost = calculate_cost(
|
||||
character_count, args.model, args.tuning_mode, args.epochs
|
||||
)
|
||||
print(f'Total character count: {character_count}')
|
||||
print(f'Estimated tuning cost: ${cost:.2f}')
|
||||
@@ -0,0 +1,64 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Module for deploying tuned models to Vertex AI endpoints."""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
import vertexai # type: ignore
|
||||
from vertexai.preview import model_garden # type: ignore
|
||||
|
||||
|
||||
def deploy_open_model(
|
||||
project: str,
|
||||
location: str,
|
||||
artifacts_uri: str,
|
||||
machine_type: str,
|
||||
accelerator_type: str,
|
||||
accelerator_count: int,
|
||||
) -> vertexai.aiplatform.Endpoint:
|
||||
"""Deploys a tuned model to a Vertex AI endpoint."""
|
||||
vertexai.init(project=project, location=location)
|
||||
|
||||
logging.info("Deploying model from %s...", artifacts_uri)
|
||||
tuned_model = model_garden.CustomModel(gcs_uri=artifacts_uri)
|
||||
|
||||
endpoint = tuned_model.deploy(
|
||||
machine_type=machine_type,
|
||||
accelerator_type=accelerator_type,
|
||||
accelerator_count=accelerator_count,
|
||||
)
|
||||
|
||||
logging.info("Model deployed to endpoint: %s", endpoint.resource_name)
|
||||
return endpoint
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Deploy Tuned Model to Vertex Endpoint"
|
||||
)
|
||||
parser.add_argument("--project", required=True)
|
||||
parser.add_argument("--location", required=True)
|
||||
parser.add_argument(
|
||||
"--artifacts_uri",
|
||||
required=True,
|
||||
help="GCS path to postprocess/node-0/checkpoints/final",
|
||||
)
|
||||
parser.add_argument("--machine_type", required=True)
|
||||
parser.add_argument("--accelerator_type", required=True)
|
||||
parser.add_argument("--accelerator_count", type=int, required=True)
|
||||
|
||||
args = parser.parse_args()
|
||||
deploy_open_model(**vars(args))
|
||||
+277
@@ -0,0 +1,277 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Module for preparing and validating datasets for Vertex AI model tuning."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import datasets
|
||||
|
||||
|
||||
def _validate_example(example: dict[str, Any], format_type: str) -> bool:
|
||||
"""Validates a single example against the expected format."""
|
||||
if format_type == "messages":
|
||||
if "messages" not in example or not isinstance(example["messages"], list):
|
||||
return False
|
||||
for msg in example["messages"]:
|
||||
if not all(k in msg for k in ("role", "content")):
|
||||
return False
|
||||
if not msg["content"] or str(msg["content"]).strip().lower() == "nan":
|
||||
return False
|
||||
else:
|
||||
if not all(k in example for k in ("prompt", "completion")):
|
||||
return False
|
||||
for k in ("prompt", "completion"):
|
||||
if not example[k] or str(example[k]).strip().lower() == "nan":
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _format_row(
|
||||
row, format_type: str, prompt_col: str, completion_col: str
|
||||
) -> dict[str, Any]:
|
||||
"""Formats a single row into the expected JSON structure."""
|
||||
prompt_text = str(row[prompt_col])
|
||||
completion_text = str(row[completion_col])
|
||||
|
||||
if format_type == "messages":
|
||||
return {
|
||||
"messages": [
|
||||
{"role": "user", "content": prompt_text},
|
||||
{"role": "assistant", "content": completion_text},
|
||||
]
|
||||
}
|
||||
return {
|
||||
"prompt": prompt_text,
|
||||
"completion": completion_text,
|
||||
}
|
||||
|
||||
|
||||
def validate_jsonl(file_path: str, format_type: str) -> bool:
|
||||
"""Validates an existing JSONL file."""
|
||||
if not os.path.exists(file_path):
|
||||
logging.error("File not found: %s", file_path)
|
||||
return False
|
||||
|
||||
valid_count = 0
|
||||
invalid_count = 0
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
for i, line in enumerate(f):
|
||||
try:
|
||||
example = json.loads(line)
|
||||
if _validate_example(example, format_type):
|
||||
valid_count += 1
|
||||
else:
|
||||
invalid_count += 1
|
||||
logging.warning("Invalid format/empty content at line %d", i + 1)
|
||||
except json.JSONDecodeError:
|
||||
invalid_count += 1
|
||||
logging.warning("Invalid JSON at line %d", i + 1)
|
||||
|
||||
logging.info("Validation complete for %s", file_path)
|
||||
logging.info("Valid: %d, Invalid: %d", valid_count, invalid_count)
|
||||
return invalid_count == 0
|
||||
|
||||
|
||||
def convert_to_jsonl(
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
format_type: str,
|
||||
prompt_col: str,
|
||||
completion_col: str,
|
||||
validation_split: float | None = 0.2,
|
||||
):
|
||||
"""Converts a JSON or CSV file to JSONL format for Vertex AI tuning."""
|
||||
if not os.path.exists(input_file):
|
||||
logging.error("Input file not found: %s", input_file)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
if input_file.endswith(".csv"):
|
||||
dataset = datasets.load_dataset(
|
||||
"csv", data_files=input_file, split="train"
|
||||
)
|
||||
elif input_file.endswith(".json"):
|
||||
dataset = datasets.load_dataset(
|
||||
"json", data_files=input_file, split="train"
|
||||
)
|
||||
elif input_file.endswith(".parquet"):
|
||||
dataset = datasets.load_dataset(
|
||||
"parquet", data_files=input_file, split="train"
|
||||
)
|
||||
else:
|
||||
logging.error("Unsupported file format. Use .csv, .json, or .parquet")
|
||||
sys.exit(1)
|
||||
except Exception as e: # pylint: disable=broad-exception-caught
|
||||
logging.exception("Failed to read input file: %s", e)
|
||||
sys.exit(1)
|
||||
|
||||
for col in [prompt_col, completion_col]:
|
||||
if col not in dataset.column_names:
|
||||
logging.error(
|
||||
"Column '%s' not found. Available columns: %s",
|
||||
col,
|
||||
dataset.column_names,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
# Remove rows with empty values in critical columns
|
||||
def is_valid(example):
|
||||
prompt_text = str(example[prompt_col]).strip().lower()
|
||||
completion_text = str(example[completion_col]).strip().lower()
|
||||
return (
|
||||
len(prompt_text) > 0
|
||||
and len(completion_text) > 0
|
||||
and prompt_text != "nan"
|
||||
and completion_text != "nan"
|
||||
and prompt_text != "none"
|
||||
and completion_text != "none"
|
||||
)
|
||||
|
||||
initial_len = len(dataset)
|
||||
dataset = dataset.filter(is_valid)
|
||||
if len(dataset) < initial_len:
|
||||
logging.warning(
|
||||
"Dropped %d rows with empty or NaN values", initial_len - len(dataset)
|
||||
)
|
||||
|
||||
def format_example(example):
|
||||
prompt_text = str(example[prompt_col])
|
||||
completion_text = str(example[completion_col])
|
||||
if format_type == "messages":
|
||||
return {
|
||||
"messages": [
|
||||
{"role": "user", "content": prompt_text},
|
||||
{"role": "assistant", "content": completion_text},
|
||||
]
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"prompt": prompt_text,
|
||||
"completion": completion_text,
|
||||
}
|
||||
|
||||
formatted_dataset = dataset.map(
|
||||
format_example, remove_columns=dataset.column_names
|
||||
)
|
||||
|
||||
if validation_split and validation_split > 0:
|
||||
if not (0 < validation_split < 1):
|
||||
logging.error("validation_split must be between 0 and 1")
|
||||
sys.exit(1)
|
||||
|
||||
# Use the datasets library to perform the split as requested
|
||||
split_dict = formatted_dataset.train_test_split(
|
||||
seed=42, test_size=validation_split
|
||||
)
|
||||
train_ds = split_dict["train"]
|
||||
val_ds = split_dict["test"]
|
||||
|
||||
val_output_file = output_file.replace(".jsonl", "_validation.jsonl")
|
||||
if val_output_file == output_file:
|
||||
val_output_file = output_file + ".validation.jsonl"
|
||||
|
||||
train_ds.to_json(output_file, force_ascii=False, lines=True)
|
||||
val_ds.to_json(val_output_file, force_ascii=False, lines=True)
|
||||
|
||||
logging.info(
|
||||
"Successfully saved %d training examples to %s",
|
||||
len(train_ds),
|
||||
output_file,
|
||||
)
|
||||
logging.info(
|
||||
"Successfully saved %d validation examples to %s",
|
||||
len(val_ds),
|
||||
val_output_file,
|
||||
)
|
||||
else:
|
||||
formatted_dataset.to_json(output_file, force_ascii=False, lines=True)
|
||||
logging.info(
|
||||
"Successfully saved %d examples to %s",
|
||||
len(formatted_dataset),
|
||||
output_file,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Prepare or Validate dataset for Vertex AI Model Tuning"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
help="Input CSV, JSON, Parquet, or JSONL file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default="tuning_dataset.jsonl",
|
||||
help="Output JSONL file (only for conversion)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--format",
|
||||
choices=["messages", "prompt"],
|
||||
default="messages",
|
||||
help="Target format (messages or prompt/completion)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt_col",
|
||||
help="Column name for prompt/user message (CSV/JSON/Parquet only)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--completion_col",
|
||||
help=(
|
||||
"Column name for completion/assistant response (CSV/JSON/Parquet"
|
||||
" only)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--validation_split",
|
||||
type=float,
|
||||
default=0.2,
|
||||
help="Fraction of data to use for validation (e.g. 0.2)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--validate_only",
|
||||
action="store_true",
|
||||
help="Only validate the input JSONL file without converting",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.validate_only:
|
||||
if not args.input:
|
||||
logging.error("--input is required for validation")
|
||||
sys.exit(1)
|
||||
success = validate_jsonl(args.input, args.format)
|
||||
sys.exit(0 if success else 1)
|
||||
else:
|
||||
if not all([args.input, args.prompt_col, args.completion_col]):
|
||||
logging.error(
|
||||
"--input, --prompt_col, and --completion_col are required for"
|
||||
" conversion"
|
||||
)
|
||||
sys.exit(1)
|
||||
convert_to_jsonl(
|
||||
args.input,
|
||||
args.output,
|
||||
args.format,
|
||||
args.prompt_col,
|
||||
args.completion_col,
|
||||
args.validation_split,
|
||||
)
|
||||
@@ -0,0 +1,92 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Module for launching Vertex AI model tuning jobs."""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
|
||||
import vertexai # type: ignore
|
||||
from vertexai.tuning import sft # type: ignore
|
||||
from vertexai.tuning import SourceModel # type: ignore
|
||||
|
||||
|
||||
def tune_open_model(
|
||||
project: str,
|
||||
location: str,
|
||||
bucket: str,
|
||||
base_model: str,
|
||||
train_dataset: str,
|
||||
validation_dataset: str | None,
|
||||
output_uri: str,
|
||||
epochs: int,
|
||||
learning_rate: float,
|
||||
tuning_mode: str,
|
||||
adapter_size: int | None = None,
|
||||
) -> sft.SupervisedTuningJob:
|
||||
"""Launches a Vertex AI model tuning job."""
|
||||
vertexai.init(project=project, location=location, staging_bucket=bucket)
|
||||
|
||||
source_model = SourceModel(base_model=base_model)
|
||||
|
||||
tuning_job = sft.train(
|
||||
source_model=source_model,
|
||||
train_dataset=train_dataset,
|
||||
validation_dataset=validation_dataset if validation_dataset else None,
|
||||
epochs=epochs,
|
||||
learning_rate=learning_rate,
|
||||
tuning_mode=tuning_mode,
|
||||
adapter_size=adapter_size,
|
||||
output_uri=output_uri,
|
||||
labels={"mg-source": "vertex-tuning-skill"},
|
||||
)
|
||||
|
||||
logging.info("Tuning job launched: %s", tuning_job.resource_name)
|
||||
logging.info(
|
||||
"View job in console:"
|
||||
" https://console.cloud.google.com/vertex-ai/locations/%s/tuning-jobs/%s?project=%s",
|
||||
location,
|
||||
tuning_job.name,
|
||||
project,
|
||||
)
|
||||
return tuning_job
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Launch Vertex AI Model Tuning Job"
|
||||
)
|
||||
parser.add_argument("--project", required=True)
|
||||
parser.add_argument("--location", required=True)
|
||||
parser.add_argument("--bucket", required=True)
|
||||
parser.add_argument("--base_model", required=True)
|
||||
parser.add_argument("--train_dataset", required=True)
|
||||
parser.add_argument(
|
||||
"--validation_dataset", help="Optional validation dataset URI"
|
||||
)
|
||||
parser.add_argument("--output_uri", required=True)
|
||||
parser.add_argument("--epochs", type=int, required=True)
|
||||
parser.add_argument("--learning_rate", type=float, required=True)
|
||||
parser.add_argument(
|
||||
"--tuning_mode", choices=["FULL", "PEFT_ADAPTER"], required=True
|
||||
)
|
||||
parser.add_argument(
|
||||
"--adapter_size",
|
||||
type=int,
|
||||
choices=[1, 4, 8, 16, 32],
|
||||
help="Adapter size for PEFT",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
tune_open_model(**vars(args))
|
||||
@@ -75,16 +75,4 @@ if __name__ == "__main__":
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
tune_model(
|
||||
project=args.project,
|
||||
location=args.location,
|
||||
bucket=args.bucket,
|
||||
base_model=args.base_model,
|
||||
train_dataset=args.train_dataset,
|
||||
validation_dataset=args.validation_dataset,
|
||||
output_uri=args.output_uri,
|
||||
epochs=args.epochs,
|
||||
learning_rate=args.learning_rate,
|
||||
tuning_mode=args.tuning_mode,
|
||||
adapter_size=args.adapter_size,
|
||||
)
|
||||
tune_model(**vars(args))
|
||||
|
||||
Reference in New Issue
Block a user