Update the server port in model_garden_keras_stable_diffusion.ipynb

Add new model Claude 3.5 Sonnet and update regions for other models (#3112 )
* add new model and update region for others * fix minor error --------- Co-authored-by: Huy Ngo <huyngo@google.com>
2026-09-26 22:51:56 +00:00 · 2024-06-21 10:49:11 -07:00 · 2024-06-20 18:22:34 +00:00 · 2024-06-20 12:39:50 +00:00 · 2024-06-18 21:47:37 +00:00 · 2024-06-18 12:36:46 +00:00
8 changed files with 196 additions and 167 deletions
@@ -1,7 +1,5 @@
 # ![Google Cloud](https://avatars.githubusercontent.com/u/2810941?s=60&v=4) Google Cloud Vertex AI Samples

-**Welcome to the Google Cloud Vertex AI Sample Repository!**
-
 This repository contains notebooks, code samples, sample apps, and other resources that demonstrate how to use, develop and manage machine learning and generative AI workflows using Google Cloud Vertex AI.

 ## Overview
@@ -10,26 +8,37 @@ This repository contains notebooks, code samples, sample apps, and other resourc

 For more Vertex AI Generative AI notebook samples, please visit the Vertex AI [Generative AI](https://github.com/GoogleCloudPlatform/generative-ai) GitHub repository.

-## Usage
+## Explore and learn

-You can explore, learn, and contribute to this repository to unleash the full potential of machine learning on Vertex AI!
+You can explore, learn, and contribute to this repository to unleash the full potential of machine learning on Vertex AI! You can follow the links in the header section of each of the notebooks to -

-![Github](https://cloud.google.com/ml-engine/images/github-logo-32px.png) You can view the notebooks on Github.\
-![Colab](https://cloud.google.com/ml-engine/images/colab-logo-32px.png) You can open and run the notebooks in [Colab](https://colab.google/).\
-![Colab Enterprise](https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png) You can open and run the notebooks in [Colab Enterprise](https://cloud.google.com/colab/docs/introduction).\
-![Workbench](https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32) You can open and run the notebooks in [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction).
+![Colab](https://cloud.google.com/ml-engine/images/colab-logo-32px.png) Open and run the notebook in [Colab](https://colab.google/)\
+![Colab Enterprise](https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png) Open and run the notebook in [Colab Enterprise](https://cloud.google.com/colab/docs/introduction)\
+![Workbench](https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32) Open and run the notebook in [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction)\
+![Github](https://cloud.google.com/ml-engine/images/github-logo-32px.png) View the notebook on Github
+
+
+## Get started
+
+To get started using Vertex AI, you must have a Google Cloud project. 
+
+- If you don't have a Google Cloud project, you can learn and build on GCP for free using [Free Trail](https://cloud.google.com/free).
+- Once you have a Google Cloud project, you can learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).


 ## Repository structure

 ```bash
-├── community-content - Sample code and tutorials contributed by the community
 ├── notebooks
-│   ├── community - Notebooks contributed by the community
 │   ├── official - Notebooks demonstrating use of each Vertex AI service
 │   │   ├── automl
 │   │   ├── custom
 │   │   ├── ...
+│   ├── community - Notebooks contributed by the community
+│   │   ├── model_garden
+│   │   ├── ...
+├── community-content - Sample code and tutorials contributed by the community
+
 ```

 ## Contributing
@@ -47,3 +56,6 @@ This is not an officially supported Google product. The code in this repository
 ## Feedback

 Please feel free to fill out our [survey](https://bit.ly/vertex-ai-samples-survey) to give us feedback on the repo and its content.
+
+## References
+- [Vertex AI Jupyter Notebook tutorials](https://cloud.google.com/vertex-ai/docs/tutorials/jupyter-notebooks)
@@ -0,0 +1,15 @@
+# Vertex AI custom prediction routines samples
+
+## Overview
+Vertex Custom Prediction Routines(CPR) simplify the process of building custom containers 
+and make local model testing easy. Here are the sameple codes for different libraries.
+
+
+### Objectives
+The objective is to provide various samples for Vertex Custom Prediction Routine(CPR).
+
+
+### Supporting libraries
+* torch
+* sklearn
+* xgboost
@@ -0,0 +1,73 @@
+import ast
+import json
+import os
+import pickle
+import torch
+
+from google.cloud.aiplatform.utils import prediction_utils
+from google.cloud.aiplatform.prediction.predictor import Predictor
+from transformers import AutoModelForQuestionAnswering
+from typing import Dict, List
+
+class TorchTransformersPredictor(Predictor):
+
+    def __init__(self):
+        return
+
+    def load(self, artifacts_uri: str) -> None:
+        prediction_utils.download_model_artifacts(artifacts_uri)
+        
+        if os.path.isfile("setup_config.json"):
+            with open("setup_config.json") as setup_config_file:
+                self.setup_config = json.load(setup_config_file)
+
+        if os.path.exists("model.pt"):
+            self.model = AutoModelForQuestionAnswering.from_pretrained("model.pt")
+            self.model.eval()
+        else:
+            raise ValueError("One of the following model files must be provided: model.pt.")
+
+    def preprocess(self, prediction_input: dict) -> torch.Tensor:
+        max_length = self.setup_config["max_length"]
+        instances = prediction_input["instances"]
+        question_context = ast.literal_eval(instances)
+        question = question_context["question"]
+        context = question_context["context"]
+        inputs = self.tokenizer.encode_plus(
+            question,
+            context,
+            max_length=int(max_length),
+            pad_to_max_length=True,
+            add_special_tokens=True,
+            return_tensors="pt",
+        )
+        input_ids = inputs["input_ids"]
+        attention_mask = inputs["attention_mask"]
+        return torch.Tensor(input_ids, attention_mask)
+    
+    @torch.inference_mode()
+    def predict(self, instances: torch.Tensor) -> List[str]:
+        input_ids, attention_mask = instances
+        outputs = self._model(input_ids, attention_mask)
+        answer_start_scores = outputs.start_logits
+        answer_end_scores = outputs.end_logits
+        
+        num_rows, num_cols = answer_start_scores.shape
+        inferences = []
+        for i in range(num_rows):
+            answer_start_scores_one_seq = answer_start_scores[i].unsqueeze(0)
+            answer_start = torch.argmax(answer_start_scores_one_seq)
+            answer_end_scores_one_seq = answer_end_scores[i].unsqueeze(0)
+            answer_end = torch.argmax(answer_end_scores_one_seq) + 1
+            prediction = self.tokenizer.convert_tokens_to_string(
+                self.tokenizer.convert_ids_to_tokens(
+                    input_ids[i].tolist()[answer_start:answer_end]
+                )
+            )
+            inferences.append(prediction)
+        return inferences
+
+    def postprocess(self, prediction_results: List[str]) -> Dict:
+        return {"predictions": prediction_results}
+     
+
@@ -16,7 +16,7 @@ Pillow==10.3.0
 rawpy==0.18.1
 scipy==1.11.3
 scikit-image==0.22.0
-scikit-learn==1.3.2
+scikit-learn==1.5.0
 tensorboard==2.15.0
 tensorboardX==2.6.2.2
 tqdm==4.66.3
@@ -356,7 +356,7 @@
        "    model = aiplatform.Model.upload(\n",
        "        display_name=deploy_model_name,\n",
        "        serving_container_image_uri=PREDICTION_CONTAINER_URI,\n",
-        "        serving_container_ports=[8501],\n",
+        "        serving_container_ports=[8080],\n",
        "        serving_container_predict_route=\"/predict\",\n",
        "        serving_container_health_route=\"/ping\",\n",
        "        serving_container_environment_variables=serving_env,\n",
@@ -92,16 +92,23 @@
        "# @title Setup Google Cloud project\n",
        "# Import the necessary packages\n",
        "\n",
-        "import json\n",
+        "! git clone https://github.com/GoogleCloudPlatform/vertex-ai-samples.git\n",
+        "\n",
+        "import importlib\n",
        "import os\n",
+        "import re\n",
        "from datetime import datetime\n",
        "from typing import Tuple\n",
        "\n",
        "from google.cloud import aiplatform\n",
        "\n",
+        "common_util = importlib.import_module(\n",
+        "    \"vertex-ai-samples.community-content.vertex_model_garden.model_oss.notebook_util.common_util\"\n",
+        ")\n",
+        "\n",
        "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
-        "# @markdown 2. [Optional] [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the specified region (`REGION`). Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n",
+        "# @markdown 2. [Optional] [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n",
        "\n",
        "# Get the default cloud project id.\n",
        "PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n",
@@ -115,7 +122,7 @@
        "\n",
        "# Cloud Storage bucket for storing the experiment artifacts.\n",
        "# A unique GCS bucket will be created for the purpose of this notebook. If you\n",
-        "# prefer using your own GCS bucket, please change the value yourself below.\n",
+        "# prefer using your own GCS bucket, change the value yourself below.\n",
        "now = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
        "BUCKET_URI = \"gs://\"  # @param {type:\"string\"}\n",
        "\n",
@@ -169,6 +176,12 @@
        "assert (\n",
        "    VERTEX_AI_MODEL_GARDEN_LLAMA3\n",
        "), \"Please click the agreement of LLaMA3 in Vertex AI Model Garden, and get the GCS path of LLaMA3 model artifacts.\"\n",
+        "parsed_gcs_url = re.search(\"gs://.*?(?=[ ]|$)\", VERTEX_AI_MODEL_GARDEN_LLAMA3)\n",
+        "if parsed_gcs_url:\n",
+        "    VERTEX_AI_MODEL_GARDEN_LLAMA3 = parsed_gcs_url.group()\n",
+        "assert VERTEX_AI_MODEL_GARDEN_LLAMA3.startswith(\n",
+        "    \"gs://\"\n",
+        "), \"VERTEX_AI_MODEL_GARDEN_LLAMA3 is expected to be a GCS URI and must start with `gs://`.\"\n",
        "print(\n",
        "    \"Copying LLaMA3 model artifacts from\",\n",
        "    VERTEX_AI_MODEL_GARDEN_LLAMA3,\n",
@@ -182,13 +195,6 @@
        "VLLM_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240508_0916_RC02\"\n",
        "\n",
        "\n",
-        "def get_job_name_with_datetime(prefix: str) -> str:\n",
-        "    \"\"\"Gets the job name with date time when triggering deployment jobs in\n",
-        "    Vertex AI.\n",
-        "    \"\"\"\n",
-        "    return prefix + datetime.now().strftime(\"_%Y%m%d_%H%M%S\")\n",
-        "\n",
-        "\n",
        "def deploy_model_vllm(\n",
        "    model_name: str,\n",
        "    model_id: str,\n",
@@ -237,101 +243,7 @@
        "    )\n",
        "    print(\"endpoint_name:\", endpoint.name)\n",
        "\n",
-        "    return model, endpoint\n",
-        "\n",
-        "\n",
-        "SERVICE_ENDPOINT = \"aiplatform.googleapis.com\"\n",
-        "\n",
-        "\n",
-        "def get_quota(project_id: str, region: str, resource_id: str) -> int:\n",
-        "    \"\"\"Returns the quota for a resource in a region. Returns -1 if can not figure out the quota.\"\"\"\n",
-        "    quota_list_output = !gcloud alpha services quota list --service=$SERVICE_ENDPOINT  --consumer=projects/$project_id --filter=\"$SERVICE_ENDPOINT/$resource_id\" --format=json\n",
-        "    # Use '.s' on the command output because it is an SList type.\n",
-        "    quota_data = json.loads(quota_list_output.s)\n",
-        "    if len(quota_data) == 0 or \"consumerQuotaLimits\" not in quota_data[0]:\n",
-        "        return -1\n",
-        "    if (\n",
-        "        len(quota_data[0][\"consumerQuotaLimits\"]) == 0\n",
-        "        or \"quotaBuckets\" not in quota_data[0][\"consumerQuotaLimits\"][0]\n",
-        "    ):\n",
-        "        return -1\n",
-        "    all_regions_data = quota_data[0][\"consumerQuotaLimits\"][0][\"quotaBuckets\"]\n",
-        "    for region_data in all_regions_data:\n",
-        "        if (\n",
-        "            region_data.get(\"dimensions\")\n",
-        "            and region_data[\"dimensions\"][\"region\"] == region\n",
-        "        ):\n",
-        "            if \"effectiveLimit\" in region_data:\n",
-        "                return int(region_data[\"effectiveLimit\"])\n",
-        "            else:\n",
-        "                return 0\n",
-        "    return -1\n",
-        "\n",
-        "\n",
-        "def get_resource_id(accelerator_type: str, is_for_training: bool) -> str:\n",
-        "    \"\"\"Returns the resource id for a given accelerator type and the use case.\n",
-        "    Args:\n",
-        "      accelerator_type: The accelerator type.\n",
-        "      is_for_training: Whether the resource is used for training. Set false\n",
-        "      for serving use case.\n",
-        "    Returns:\n",
-        "      The resource id.\n",
-        "    \"\"\"\n",
-        "    training_accelerator_map = {\n",
-        "        \"NVIDIA_TESLA_V100\": \"custom_model_training_nvidia_v100_gpus\",\n",
-        "        \"NVIDIA_L4\": \"custom_model_training_nvidia_l4_gpus\",\n",
-        "        \"NVIDIA_TESLA_A100\": \"custom_model_training_nvidia_a100_gpus\",\n",
-        "    }\n",
-        "    serving_accelerator_map = {\n",
-        "        \"NVIDIA_TESLA_V100\": \"custom_model_serving_nvidia_v100_gpus\",\n",
-        "        \"NVIDIA_L4\": \"custom_model_serving_nvidia_l4_gpus\",\n",
-        "        \"NVIDIA_TESLA_A100\": \"custom_model_serving_nvidia_a100_gpus\",\n",
-        "    }\n",
-        "    if is_for_training:\n",
-        "        if accelerator_type in training_accelerator_map:\n",
-        "            return training_accelerator_map[accelerator_type]\n",
-        "        else:\n",
-        "            raise ValueError(\n",
-        "                f\"Could not find accelerator type: {accelerator_type} for training.\"\n",
-        "            )\n",
-        "    else:\n",
-        "        if accelerator_type in serving_accelerator_map:\n",
-        "            return serving_accelerator_map[accelerator_type]\n",
-        "        else:\n",
-        "            raise ValueError(\n",
-        "                f\"Could not find accelerator type: {accelerator_type} for serving.\"\n",
-        "            )\n",
-        "\n",
-        "\n",
-        "def check_quota(\n",
-        "    project_id: str,\n",
-        "    region: str,\n",
-        "    accelerator_type: str,\n",
-        "    accelerator_count: int,\n",
-        "    is_for_training: bool,\n",
-        "):\n",
-        "    \"\"\"Checks if the project and the region has the required quota.\"\"\"\n",
-        "    resource_id = get_resource_id(accelerator_type, is_for_training)\n",
-        "    quota = get_quota(project_id, region, resource_id)\n",
-        "    quota_request_instruction = (\n",
-        "        \"Either use \"\n",
-        "        \"a different region or request additional quota. Follow \"\n",
-        "        \"instructions here \"\n",
-        "        \"https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota\"\n",
-        "        \" to check quota in a region or request additional quota for \"\n",
-        "        \"your project.\"\n",
-        "    )\n",
-        "    if quota == -1:\n",
-        "        raise ValueError(\n",
-        "            f\"\"\"Quota not found for: {resource_id} in {region}.\n",
-        "            {quota_request_instruction}\"\"\"\n",
-        "        )\n",
-        "    if quota < accelerator_count:\n",
-        "        raise ValueError(\n",
-        "            f\"\"\"Quota not enough for {resource_id} in {region}:\n",
-        "            {quota} < {accelerator_count}.\n",
-        "            {quota_request_instruction}\"\"\"\n",
-        "        )"
+        "    return model, endpoint\n"
      ]
    },
    {
@@ -356,9 +268,10 @@
        "\n",
        "# @markdown This section uploads prebuilt LLaMA3 models to Model Registry and deploys it to a Vertex AI Endpoint. It takes 15 minutes to 1 hour to finish depending on the size of the model.\n",
        "\n",
-        "\n",
        "# @markdown NVIDIA_L4 GPUs are used for demonstration. The serving efficiency of L4 GPUs is inferior to that of A100 GPUs, but L4 GPUs are nevertheless good serving solutions if you do not have A100 quota.\n",
        "\n",
+        "# @markdown Llama 3 uses a context length of 8,192 tokens, double the context length of Llama 2. Please see this [Meta blog post](https://ai.meta.com/blog/meta-llama-3/) for more details.\n",
+        "\n",
        "# @markdown Set the model to deploy.\n",
        "\n",
        "base_model_name = \"llama3-8b-chat-hf\"  # @param [\"llama3-8b-hf\", \"llama3-8b-chat-hf\", \"llama3-70b-hf\", \"llama3-70b-chat-hf\"] {isTemplate:true}\n",
@@ -401,7 +314,7 @@
        "    raise ValueError(\n",
        "        f\"Recommended GPU setting not found for: {accelerator_type} and {base_model_name}.\"\n",
        "    )\n",
-        "check_quota(\n",
+        "common_util.check_quota(\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "    accelerator_type=accelerator_type,\n",
@@ -412,8 +325,12 @@
        "gpu_memory_utilization = 0.85\n",
        "max_model_len = 8192  # Maximum context length.\n",
        "\n",
+        "# Ensure max_model_len does not exceed the limit\n",
+        "if max_model_len > 8192:\n",
+        "    raise ValueError(\"max_model_len cannot exceed 8192\")\n",
+        "\n",
        "model, endpoint = deploy_model_vllm(\n",
-        "    model_name=get_job_name_with_datetime(prefix=\"llama3-serve\"),\n",
+        "    model_name=common_util.get_job_name_with_datetime(prefix=\"llama3-serve\"),\n",
        "    model_id=model_id,\n",
        "    service_account=SERVICE_ACCOUNT,\n",
        "    machine_type=machine_type,\n",
@@ -436,7 +353,7 @@
      "source": [
        "# @title Predict\n",
        "\n",
-        "# @markdown Once deployment succeeds, you can send requests to the endpoint with text prompts. Sampling parameters supported by vLLM can be found [here](https://github.com/vllm-project/vllm/blob/2e8e49fce3775e7704d413b2f02da6d7c99525c9/vllm/sampling_params.py#L23-L64).\n",
+        "# @markdown Once deployment succeeds, you can send requests to the endpoint with text prompts. Sampling parameters supported by vLLM can be found [here](https://docs.vllm.ai/en/latest/dev/sampling_params.html).\n",
        "\n",
        "# @markdown Example:\n",
        "\n",
@@ -187,7 +187,9 @@
        "# @markdown Alternatively, you can also load the original Llama 3 models for finetuning and serving from Vertex AI after accepting the agreement.\n",
        "\n",
        "# @markdown **Only select and fill one of the following sections.**\n",
+        "# fmt: off\n",
        "LOAD_MODEL_FROM = \"Hugging Face\"  # @param [\"Hugging Face\", \"Google Cloud\"] {isTemplate:true}\n",
+        "# fmt: on\n",
        "\n",
        "# @markdown ---\n",
        "\n",
@@ -446,7 +448,7 @@
        "        )\n",
        "elif \"70b\" in MODEL_ID.lower():\n",
        "    if accelerator_type == \"NVIDIA_TESLA_A100\":\n",
-        "        accelerator_count = 8\n",
+        "        accelerator_count = 4\n",
        "        machine_type = \"a2-highgpu-4g\"\n",
        "    else:\n",
        "        raise ValueError(\n",
@@ -465,7 +467,7 @@
        "    is_for_training=True,\n",
        ")\n",
        "\n",
-        "job_name = common_util.get_job_name_with_datetime(\"llama3-lora-train\").replace('_', '-')\n",
+        "job_name = common_util.get_job_name_with_datetime(\"llama3-lora-train\").replace(\"_\", \"-\")\n",
        "\n",
        "base_output_dir = os.path.join(STAGING_BUCKET, job_name)\n",
        "# Create a GCS folder to store the LORA adapter.\n",
@@ -485,37 +487,37 @@
        "]\n",
        "\n",
        "train_job_args = [\n",
-        "        \"--config_file=vertex_vision_model_garden_peft/deepspeed_config_file_zero2_4gpu.yaml\",\n",
-        "        \"--task=instruct-lora\",\n",
-        "        \"--completion_only=True\",\n",
-        "        f\"--pretrained_model_id={base_model_id}\",\n",
-        "        f\"--dataset_name={train_dataset_name}\",\n",
-        "        f\"--train_split_name={train_split_name}\",\n",
-        "        f\"--instruct_column_in_dataset={instruct_column_in_dataset}\",\n",
-        "        f\"--output_dir={lora_output_dir}\",\n",
-        "        f\"--merge_base_and_lora_output_dir={merged_model_output_dir}\",\n",
-        "        f\"--per_device_train_batch_size={per_device_train_batch_size}\",\n",
-        "        f\"--gradient_accumulation_steps={gradient_accumulation_steps}\",\n",
-        "        f\"--lora_rank={lora_rank}\",\n",
-        "        f\"--lora_alpha={lora_alpha}\",\n",
-        "        f\"--lora_dropout={lora_dropout}\",\n",
-        "        f\"--max_steps={max_steps}\",\n",
-        "        f\"--max_seq_length={max_seq_length}\",\n",
-        "        f\"--learning_rate={learning_rate}\",\n",
-        "        f\"--lr_scheduler_type={lr_scheduler_type}\",\n",
-        "        f\"--precision_mode={finetuning_precision_mode}\",\n",
-        "        f\"--enable_gradient_checkpointing={enable_gradient_checkpointing}\",\n",
-        "        f\"--num_epochs={num_epochs}\",\n",
-        "        f\"--attn_implementation={attn_implementation}\",\n",
-        "        f\"--optimizer={optimizer}\",\n",
-        "        f\"--warmup_ratio={warmup_ratio}\",\n",
-        "        f\"--report_to={report_to}\",\n",
-        "        f\"--logging_output_dir={base_output_dir}\",\n",
-        "        f\"--save_steps={save_steps}\",\n",
-        "        f\"--logging_steps={logging_steps}\",\n",
-        "        f\"--template={template}\",\n",
-        "        f\"--huggingface_access_token={HF_TOKEN}\",\n",
-        "    ] + eval_args\n",
+        "    \"--config_file=vertex_vision_model_garden_peft/deepspeed_config_file_zero2_4gpu.yaml\",\n",
+        "    \"--task=instruct-lora\",\n",
+        "    \"--completion_only=True\",\n",
+        "    f\"--pretrained_model_id={base_model_id}\",\n",
+        "    f\"--dataset_name={train_dataset_name}\",\n",
+        "    f\"--train_split_name={train_split_name}\",\n",
+        "    f\"--instruct_column_in_dataset={instruct_column_in_dataset}\",\n",
+        "    f\"--output_dir={lora_output_dir}\",\n",
+        "    f\"--merge_base_and_lora_output_dir={merged_model_output_dir}\",\n",
+        "    f\"--per_device_train_batch_size={per_device_train_batch_size}\",\n",
+        "    f\"--gradient_accumulation_steps={gradient_accumulation_steps}\",\n",
+        "    f\"--lora_rank={lora_rank}\",\n",
+        "    f\"--lora_alpha={lora_alpha}\",\n",
+        "    f\"--lora_dropout={lora_dropout}\",\n",
+        "    f\"--max_steps={max_steps}\",\n",
+        "    f\"--max_seq_length={max_seq_length}\",\n",
+        "    f\"--learning_rate={learning_rate}\",\n",
+        "    f\"--lr_scheduler_type={lr_scheduler_type}\",\n",
+        "    f\"--precision_mode={finetuning_precision_mode}\",\n",
+        "    f\"--enable_gradient_checkpointing={enable_gradient_checkpointing}\",\n",
+        "    f\"--num_epochs={num_epochs}\",\n",
+        "    f\"--attn_implementation={attn_implementation}\",\n",
+        "    f\"--optimizer={optimizer}\",\n",
+        "    f\"--warmup_ratio={warmup_ratio}\",\n",
+        "    f\"--report_to={report_to}\",\n",
+        "    f\"--logging_output_dir={base_output_dir}\",\n",
+        "    f\"--save_steps={save_steps}\",\n",
+        "    f\"--logging_steps={logging_steps}\",\n",
+        "    f\"--template={template}\",\n",
+        "    f\"--huggingface_access_token={HF_TOKEN}\",\n",
+        "] + eval_args\n",
        "\n",
        "# Create TensorBoard\n",
        "tensorboard = aiplatform.Tensorboard.create(job_name)\n",
@@ -72,11 +72,17 @@
        "\n",
        "### Available Anthropic Claude models\n",
        "\n",
-        "#### Claude 3 Sonnet\n",
-        "Anthropic Claude 3 Sonnet provides a balance between intelligence and speed for enterprise workloads. It's a high-endurance model for scaled AI that's available at a competitive price.\n",
+        "#### Claude 3.5 Sonnet\n",
+        "Anthropic's most powerful AI model. Claude 3.5 Sonnet outperforms competitor models and Claude 3 Opus at higher speeds and lower cost.\n",
+        "\n",
+        "#### Claude 3 Opus\n",
+        "Claude 3 Opus is Anthropic's second-most intelligent AI model, with top-level performance on highly complex tasks.\n",
        "\n",
        "#### Claude 3 Haiku\n",
-        "Anthropic Claude 3 Haiku is the fastest, most compact model available from Anthropic. It is designed to answer simple queries and requests quickly. You can use it to build AI experiences that mimic human interactions.\n",
+        "Anthropic Claude 3 Haiku is Anthropic's fastest, most compact vision and text model for near-instant responses to simple queries, meant for seamless AI experiences mimicking human interactions.\n",
+        "\n",
+        "#### Claude 3 Sonnet\n",
+        "Anthropic Claude 3 Sonnet is engineered to be dependable for scaled AI deployments across a variety of use cases.\n",
        "\n",
        "All Claude 3 models can process images and return text outputs, and feature a 200K context window.\n",
        "\n",
@@ -210,12 +216,14 @@
      },
      "outputs": [],
      "source": [
-        "MODEL = \"claude-3-sonnet@20240229\"  # @param [\"claude-3-sonnet@20240229\", \"claude-3-haiku@20240307\", \"claude-3-opus@20240229\"]\n",
-        "if MODEL == \"claude-3-sonnet@20240229\":\n",
-        "    available_regions = [\"us-central1\", \"asia-southeast1\"]\n",
+        "MODEL = \"claude-3-5-sonnet@20240620\"  # @param [\"claude-3-5-sonnet@20240620\", \"claude-3-opus@20240229\", \"claude-3-haiku@20240307\", \"claude-3-sonnet@20240229\" ]\n",
+        "if MODEL == \"claude-3-5-sonnet@20240620\":\n",
+        "    available_regions = [\"us-east5\", \"europe-west1\"]\n",
+        "elif MODEL == \"claude-3-opus@20240229\":\n",
+        "    available_regions = [\"us-east5\"]\n",
        "elif MODEL == \"claude-3-haiku@20240307\":\n",
-        "    available_regions = [\"us-central1\", \"europe-west4\"]\n",
-        "else:\n",
+        "    available_regions = [\"us-east5\", \"europe-west1\"]\n",
+        "elif MODEL == \"claude-3-sonnet@20240229\":\n",
        "    available_regions = [\"us-east5\"]"
      ]
    },
@@ -630,12 +638,14 @@
      },
      "outputs": [],
      "source": [
-        "MODEL = \"claude-3-sonnet@20240229\"  # @param [\"claude-3-sonnet@20240229\", \"claude-3-haiku@20240307\", \"claude-3-opus@20240229\"]\n",
-        "if MODEL == \"claude-3-sonnet@20240229\":\n",
-        "    available_regions = [\"us-central1\", \"asia-southeast1\"]\n",
+        "MODEL = \"claude-3-5-sonnet@20240620\"  # @param [\"claude-3-5-sonnet@20240620\", \"claude-3-opus@20240229\", \"claude-3-haiku@20240307\", \"claude-3-sonnet@20240229\" ]\n",
+        "if MODEL == \"claude-3-5-sonnet@20240620\":\n",
+        "    available_regions = [\"us-east5\", \"europe-west1\"]\n",
+        "elif MODEL == \"claude-3-opus@20240229\":\n",
+        "    available_regions = [\"us-east5\"]\n",
        "elif MODEL == \"claude-3-haiku@20240307\":\n",
-        "    available_regions = [\"us-central1\", \"europe-west4\"]\n",
-        "else:\n",
+        "    available_regions = [\"us-east5\", \"europe-west1\"]\n",
+        "elif MODEL == \"claude-3-sonnet@20240229\":\n",
        "    available_regions = [\"us-east5\"]"
      ]
    },
Author	SHA1	Message	Date
KCFindstrandGitHub	b94d774e1d	Update the server port in model_garden_keras_stable_diffusion.ipynb	2024-06-21 10:49:11 -07:00
Huy Ngo GitHub Huy Ngo	8eb2aa93d5	Add new model Claude 3.5 Sonnet and update regions for other models (#3112 ) * add new model and update region for others * fix minor error --------- Co-authored-by: Huy Ngo <huyngo@google.com>	2024-06-20 18:22:34 +00:00
sefgsefgandGitHub	3947a8bc24	Upload torch transformers predictor sample (#3087 ) This sample uses the aiplatform SDK and torch library to implement transformers predictor.	2024-06-20 12:39:50 +00:00
Eric DongandGitHub	022e1c8ee7	Update README.md 4 (#3104 ) Add Get started section	2024-06-18 21:47:37 +00:00
Rayan Dasoriya GitHub Rayan Dasoriya	cec3f9dd55	refactor: refactor llama3 deployment nb (#3100 ) Co-authored-by: Rayan Dasoriya <dasoriya@google.com>	2024-06-18 12:36:46 +00:00
Minwoo Park GitHub minwoopark	70ebb04f06	Update llama3 finetuning notebook to use 4 A100s instead of 8 (#3105 ) Co-authored-by: minwoopark <minwoopark@google.com>	2024-06-18 12:31:42 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	fd21267d70	chore(deps): bump scikit-learn (#3106 ) Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.3.2 to 1.5.0. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](https://github.com/scikit-learn/scikit-learn/compare/1.3.2...1.5.0) --- updated-dependencies: - dependency-name: scikit-learn dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-06-18 12:31:00 +00:00