xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add SAM 3 notebook to Vertex AI Model Garden.

Browse Source 
PiperOrigin-RevId: 866598579
This commit is contained in:
Vertex MG Team
2026-02-06 13:45:47 -08:00
committed by Copybara-Service
parent bf2c1226fd
commit 28f7fc4445
1 changed files with 841 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb
+841
View File
@@ -0,0 +1,841 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "KEiqUBnnbGwo"
},
"outputs": [],
"source": [
"# Copyright 2026 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "idcJ3uajbGwo"
},
"source": [
"# Vertex AI Model Garden - SAM 3 (Segment Anything Model 3)\n",
"<table><tbody><tr>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb\">\n",
" <img alt=\"Workbench logo\" src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" width=\"32px\"><br> Run in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fmodel_garden%2Fmodel_garden_pytorch_sam3.ipynb\">\n",
" <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb\">\n",
" <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</tr></tbody></table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rzbW4muObGwo"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates deploying **SAM 3** on Vertex AI for:\n",
"- **Image Segmentation** (text-prompted)\n",
"- **Point-Click Segmentation** (coordinate-based)\n",
"- **Video Segmentation** (object tracking)\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n",
"\n",
"### File a bug\n",
"\n",
"File a bug on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new) if you encounter any issue with the notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_TUvfvA-bGwo"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "yFIKRg-pbGwo"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet 'google-cloud-aiplatform>=1.106.0' 'google-cloud-storage' 'pycocotools' 'numpy<2.0' 'opencv-python-headless' 'matplotlib' 'Pillow==11.3.0'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "fTVQfwSJbGwo"
},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "0YdXF0EGk1Aj"
},
"outputs": [],
"source": [
"import base64\n",
"import io\n",
"import re\n",
"import subprocess\n",
"import tempfile\n",
"import uuid\n",
"\n",
"import cv2\n",
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import requests\n",
"from google.cloud import aiplatform, storage\n",
"from PIL import Image, ImageDraw\n",
"from pycocotools import mask as mask_utils"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "vnTVbHMibGwo"
},
"outputs": [],
"source": [
"# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"# @markdown 2. **[Optional]** [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n",
"\n",
"BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n",
"\n",
"# @markdown 3. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n",
"\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"\n",
"\n",
"# Get the default cloud project id.\n",
"PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n",
"\n",
"# Get the default region for launching jobs.\n",
"if not REGION:\n",
" if not os.environ.get(\"GOOGLE_CLOUD_REGION\"):\n",
" raise ValueError(\n",
" \"REGION must be set. See\"\n",
" \" https://cloud.google.com/vertex-ai/docs/general/locations for\"\n",
" \" available cloud locations.\"\n",
" )\n",
" REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n",
"\n",
"# Enable the Vertex AI API and Compute Engine API, if not already.\n",
"print(\"Enabling Vertex AI API and Compute Engine API.\")\n",
"! gcloud services enable aiplatform.googleapis.com compute.googleapis.com\n",
"\n",
"# Initialize Vertex AI API.\n",
"print(\"Initializing Vertex AI API.\")\n",
"aiplatform.init(project=PROJECT_ID, location=REGION)\n",
"! gcloud config set project $PROJECT_ID\n",
"\n",
"models, endpoints = {}, {}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "i7Z2h9aJbGwo"
},
"outputs": [],
"source": [
"# GCS bucket for large images/videos (required for files > 1.1MB)\n",
"GCS_BUCKET = BUCKET_URI if BUCKET_URI and BUCKET_URI != \"gs://\" else None\n",
"\n",
"# Hugging Face token (required for gated model facebook/sam3)\n",
"HF_TOKEN = \"\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "pm0Ybff-bGwo"
},
"outputs": [],
"source": [
"# SAM3 Configuration\n",
"SAM3_DOCKER_URI = \"us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/pytorch-inference.cu125.0-4.ubuntu2204.py310\"\n",
"DEFAULT_MASK_BLUR_SIGMA = 3.5\n",
"\n",
"# GPU Configuration: (accelerator_type, gpu_count) -> machine_type\n",
"GPU_MACHINE_TYPE_MAP = {\n",
" (\"NVIDIA_L4\", 1): \"g2-standard-12\",\n",
" (\"NVIDIA_L4\", 2): \"g2-standard-24\",\n",
" (\"NVIDIA_L4\", 4): \"g2-standard-48\",\n",
" (\"NVIDIA_L4\", 8): \"g2-standard-96\",\n",
" (\"NVIDIA_H100_80GB\", 1): \"a3-highgpu-1g\",\n",
" (\"NVIDIA_H100_80GB\", 2): \"a3-highgpu-2g\",\n",
" (\"NVIDIA_H100_80GB\", 4): \"a3-highgpu-4g\",\n",
" (\"NVIDIA_H100_80GB\", 8): \"a3-highgpu-8g\",\n",
"}\n",
"\n",
"\n",
"def deploy_sam3_model(\n",
" model_id: str,\n",
" accelerator_type: str = \"NVIDIA_L4\",\n",
" accelerator_count: int = 1,\n",
" use_dedicated_endpoint: bool = True,\n",
" hf_token: str = None,\n",
") -> tuple:\n",
" \"\"\"Deploy SAM3 model to Vertex AI endpoint.\"\"\"\n",
" machine_type = GPU_MACHINE_TYPE_MAP[(accelerator_type, accelerator_count)]\n",
"\n",
" endpoint = aiplatform.Endpoint.create(\n",
" display_name=\"sam3-endpoint-notebook\",\n",
" dedicated_endpoint_enabled=use_dedicated_endpoint,\n",
" )\n",
"\n",
" env_vars = {\n",
" \"MODEL_ID\": model_id,\n",
" \"TASK\": \"mask-generation\",\n",
" \"DEPLOY_SOURCE\": \"notebook\",\n",
" }\n",
" if hf_token:\n",
" env_vars[\"HF_TOKEN\"] = hf_token\n",
"\n",
" model = aiplatform.Model.upload(\n",
" display_name=\"sam3-notebook\",\n",
" serving_container_image_uri=SAM3_DOCKER_URI,\n",
" serving_container_ports=[8080],\n",
" serving_container_predict_route=\"/predict\",\n",
" serving_container_health_route=\"/health\",\n",
" serving_container_environment_variables=env_vars,\n",
" serving_container_shared_memory_size_mb=(16 * 1024),\n",
" serving_container_deployment_timeout=7200,\n",
" )\n",
"\n",
" model.deploy(\n",
" endpoint=endpoint,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" min_replica_count=1,\n",
" max_replica_count=1,\n",
" deploy_request_timeout=1800,\n",
" )\n",
" return model, endpoint\n",
"\n",
"\n",
"def _get_auth_headers():\n",
" token = (\n",
" subprocess.check_output([\"gcloud\", \"auth\", \"print-access-token\"])\n",
" .decode()\n",
" .strip()\n",
" )\n",
" return {\"Authorization\": f\"Bearer {token}\", \"Content-Type\": \"application/json\"}\n",
"\n",
"\n",
"def _get_dedicated_dns(endpoint):\n",
" try:\n",
" if (\n",
" hasattr(endpoint, \"gca_resource\")\n",
" and endpoint.gca_resource.dedicated_endpoint_dns\n",
" ):\n",
" return endpoint.gca_resource.dedicated_endpoint_dns\n",
" except:\n",
" pass\n",
" return None\n",
"\n",
"\n",
"def call_sam3_endpoint(payload, endpoint, timeout=180):\n",
" \"\"\"Call SAM3 Vertex AI endpoint with automatic dedicated DNS resolution.\"\"\"\n",
" endpoint_id = endpoint.name.split(\"/\")[-1]\n",
" dns = _get_dedicated_dns(endpoint)\n",
" host = dns if dns else f\"{REGION}-aiplatform.googleapis.com\"\n",
" url = f\"https://{host}/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_id}:predict\"\n",
"\n",
" response = requests.post(\n",
" url, json=payload, headers=_get_auth_headers(), timeout=timeout\n",
" )\n",
" if response.status_code == 400 and \"dedicated domain name\" in response.text:\n",
" dns = re.search(r\"dedicated domain name '([^']+)'\", response.text).group(1)\n",
" url = f\"https://{dns}/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_id}:predict\"\n",
" response = requests.post(\n",
" url, json=payload, headers=_get_auth_headers(), timeout=timeout\n",
" )\n",
"\n",
" if response.status_code != 200:\n",
" raise RuntimeError(\n",
" f\"Vertex AI error {response.status_code}: {response.text[:500]}\"\n",
" )\n",
" return response.json()\n",
"\n",
"\n",
"def decode_rle_mask(rle):\n",
" counts = (\n",
" rle[\"counts\"].encode(\"utf-8\")\n",
" if isinstance(rle[\"counts\"], str)\n",
" else rle[\"counts\"]\n",
" )\n",
" return mask_utils.decode({\"size\": rle[\"size\"], \"counts\": counts}).astype(np.uint8)\n",
"\n",
"\n",
"def image_to_base64(img):\n",
" buf = io.BytesIO()\n",
" img.save(buf, format=\"PNG\")\n",
" return base64.b64encode(buf.getvalue()).decode()\n",
"\n",
"\n",
"def load_image(path_or_url):\n",
" if path_or_url.startswith((\"http://\", \"https://\")):\n",
" return Image.open(\n",
" io.BytesIO(requests.get(path_or_url, timeout=30).content)\n",
" ).convert(\"RGB\")\n",
" return Image.open(path_or_url).convert(\"RGB\")\n",
"\n",
"\n",
"def upload_to_gcs(data, gcs_uri, prefix, content_type, is_file=False):\n",
" \"\"\"Upload data or file to GCS. Returns gs:// URI.\"\"\"\n",
" bucket_name, path = (\n",
" gcs_uri[5:].split(\"/\", 1) if \"/\" in gcs_uri[5:] else (gcs_uri[5:], \"\")\n",
" )\n",
" ext = \".mp4\" if \"video\" in content_type else \".png\"\n",
" blob_path = f\"{path}/{prefix}-{uuid.uuid4().hex[:8]}{ext}\".lstrip(\"/\")\n",
"\n",
" blob = storage.Client().bucket(bucket_name).blob(blob_path)\n",
" if is_file:\n",
" blob.upload_from_filename(data, content_type=content_type)\n",
" else:\n",
" blob.upload_from_string(data, content_type=content_type)\n",
" return f\"gs://{bucket_name}/{blob_path}\"\n",
"\n",
"\n",
"def delete_gcs_file(gcs_uri):\n",
" try:\n",
" path = gcs_uri[5:]\n",
" bucket_name, blob_path = path.split(\"/\", 1)\n",
" storage.Client().bucket(bucket_name).blob(blob_path).delete()\n",
" except:\n",
" pass\n",
"\n",
"\n",
"def apply_mask_overlay(image, masks, opacity=0.5):\n",
" \"\"\"Overlay colored masks on image.\"\"\"\n",
" if isinstance(image, np.ndarray):\n",
" image = Image.fromarray(image)\n",
" image = image.convert(\"RGBA\")\n",
" if not masks:\n",
" return image.convert(\"RGB\")\n",
"\n",
" cmap = matplotlib.colormaps[\"rainbow\"].resampled(len(masks))\n",
" composite = Image.new(\"RGBA\", image.size, (0, 0, 0, 0))\n",
"\n",
" for i, mask in enumerate(masks):\n",
" color = tuple(int(c * 255) for c in cmap(i)[:3])\n",
" mask_img = Image.fromarray((mask * 255).astype(np.uint8))\n",
" if mask_img.size != image.size:\n",
" mask_img = mask_img.resize(image.size, Image.NEAREST)\n",
" fill = Image.new(\"RGBA\", image.size, color + (0,))\n",
" fill.putalpha(mask_img.point(lambda v: int(v * opacity) if v > 0 else 0))\n",
" composite = Image.alpha_composite(composite, fill)\n",
"\n",
" return Image.alpha_composite(image, composite).convert(\"RGB\")\n",
"\n",
"\n",
"def draw_points(image, points):\n",
" \"\"\"Draw red circles at click points.\"\"\"\n",
" if isinstance(image, np.ndarray):\n",
" image = Image.fromarray(image)\n",
" img = image.copy()\n",
" draw = ImageDraw.Draw(img)\n",
" for x, y in points:\n",
" draw.ellipse((x - 8, y - 8, x + 8, y + 8), fill=\"red\", outline=\"white\", width=4)\n",
" return img\n",
"\n",
"\n",
"def segment_image(\n",
" image_path_or_url,\n",
" text_prompt,\n",
" endpoint,\n",
" gcs_bucket=None,\n",
" blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n",
"):\n",
" \"\"\"Text-prompted image segmentation. Returns (original, overlay, masks).\"\"\"\n",
" img = load_image(image_path_or_url)\n",
" img_b64 = image_to_base64(img)\n",
" gcs_uri = None\n",
"\n",
" # Upload large images to GCS\n",
" if len(img_b64) * 0.75 > 1.1 * 1024 * 1024:\n",
" if not gcs_bucket:\n",
" raise RuntimeError(\"Image too large. Set GCS_BUCKET for large images.\")\n",
" buf = io.BytesIO()\n",
" img.save(buf, format=\"PNG\")\n",
" gcs_uri = upload_to_gcs(buf.getvalue(), gcs_bucket, \"sam3-img\", \"image/png\")\n",
" payload = {\n",
" \"instances\": [\n",
" {\"image\": gcs_uri, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n",
" ],\n",
" \"parameters\": {\"mask_format\": \"rle\"},\n",
" }\n",
" else:\n",
" payload = {\n",
" \"instances\": [\n",
" {\"image\": img_b64, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n",
" ],\n",
" \"parameters\": {\"mask_format\": \"rle\"},\n",
" }\n",
"\n",
" try:\n",
" result = call_sam3_endpoint(payload, endpoint)\n",
" finally:\n",
" if gcs_uri:\n",
" delete_gcs_file(gcs_uri)\n",
"\n",
" masks = [\n",
" decode_rle_mask(rle)\n",
" for rle in result.get(\"predictions\", [{}])[0].get(\"masks_rle\", [])\n",
" ]\n",
" return img, apply_mask_overlay(img, masks), masks\n",
"\n",
"\n",
"def segment_by_points(\n",
" image_path_or_url,\n",
" points,\n",
" endpoint,\n",
" gcs_bucket=None,\n",
" blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n",
"):\n",
" \"\"\"Point-click segmentation. Returns (original, overlay with points, masks).\"\"\"\n",
" img = load_image(image_path_or_url)\n",
" img_b64 = image_to_base64(img)\n",
" gcs_uri = None\n",
"\n",
" if len(img_b64) * 0.75 > 1.1 * 1024 * 1024:\n",
" if not gcs_bucket:\n",
" raise RuntimeError(\"Image too large. Set GCS_BUCKET for large images.\")\n",
" buf = io.BytesIO()\n",
" img.save(buf, format=\"PNG\")\n",
" gcs_uri = upload_to_gcs(buf.getvalue(), gcs_bucket, \"sam3-click\", \"image/png\")\n",
" payload = {\n",
" \"instances\": [\n",
" {\n",
" \"image\": gcs_uri,\n",
" \"input_points\": points,\n",
" \"mask_blur_sigma\": blur_sigma,\n",
" }\n",
" ],\n",
" \"parameters\": {\"mask_format\": \"rle\"},\n",
" }\n",
" else:\n",
" payload = {\n",
" \"instances\": [\n",
" {\n",
" \"image\": img_b64,\n",
" \"input_points\": points,\n",
" \"mask_blur_sigma\": blur_sigma,\n",
" }\n",
" ],\n",
" \"parameters\": {\"mask_format\": \"rle\"},\n",
" }\n",
"\n",
" try:\n",
" result = call_sam3_endpoint(payload, endpoint)\n",
" finally:\n",
" if gcs_uri:\n",
" delete_gcs_file(gcs_uri)\n",
"\n",
" masks = [\n",
" decode_rle_mask(rle)\n",
" for rle in result.get(\"predictions\", [{}])[0].get(\"masks_rle\", [])\n",
" ]\n",
" overlay = draw_points(apply_mask_overlay(img, masks), points)\n",
" return img, overlay, masks\n",
"\n",
"\n",
"def segment_video(\n",
" video_path,\n",
" text_prompt,\n",
" endpoint,\n",
" gcs_bucket,\n",
" frame_limit=60,\n",
" timeout=1600,\n",
" blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n",
"):\n",
" \"\"\"Video segmentation. Returns (output_path, sample_frames, status).\"\"\"\n",
" cap = cv2.VideoCapture(video_path)\n",
" fps, w, h = (\n",
" cap.get(cv2.CAP_PROP_FPS),\n",
" int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),\n",
" int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)),\n",
" )\n",
"\n",
" frames = []\n",
" while cap.isOpened() and len(frames) < frame_limit:\n",
" ret, frame = cap.read()\n",
" if not ret:\n",
" break\n",
" frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))\n",
" cap.release()\n",
"\n",
" # Write temp video and upload to GCS\n",
" tmp = tempfile.mktemp(suffix=\".mp4\")\n",
" writer = cv2.VideoWriter(tmp, cv2.VideoWriter_fourcc(*\"mp4v\"), fps, (w, h))\n",
" for f in frames:\n",
" writer.write(cv2.cvtColor(f, cv2.COLOR_RGB2BGR))\n",
" writer.release()\n",
"\n",
" gcs_uri = upload_to_gcs(tmp, gcs_bucket, \"sam3-vid\", \"video/mp4\", is_file=True)\n",
" os.unlink(tmp)\n",
"\n",
" try:\n",
" payload = {\n",
" \"instances\": [\n",
" {\"video\": gcs_uri, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n",
" ],\n",
" \"parameters\": {\"mask_format\": \"rle\"},\n",
" }\n",
" result = call_sam3_endpoint(payload, endpoint, timeout=max(timeout, 1600))\n",
" finally:\n",
" delete_gcs_file(gcs_uri)\n",
"\n",
" masks_video = result.get(\"predictions\", [{}])[0].get(\"masks_rle_video\", [])\n",
"\n",
" # Create output video\n",
" out_path = tempfile.mktemp(suffix=\".mp4\")\n",
" writer = cv2.VideoWriter(out_path, cv2.VideoWriter_fourcc(*\"mp4v\"), fps, (w, h))\n",
" overlay_frames = []\n",
"\n",
" for i, frame_masks in enumerate(masks_video):\n",
" if i >= len(frames):\n",
" break\n",
" decoded = [decode_rle_mask(rle) for rle in frame_masks] if frame_masks else []\n",
" overlay = apply_mask_overlay(Image.fromarray(frames[i]), decoded)\n",
" overlay_frames.append(overlay)\n",
" writer.write(cv2.cvtColor(np.array(overlay), cv2.COLOR_RGB2BGR))\n",
" writer.release()\n",
"\n",
" return out_path, overlay_frames, f\"Processed {len(masks_video)} frames\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pZyx8HJmbGwo"
},
"source": [
"## Deploy SAM3 Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "vQEX7jYGbGwo"
},
"outputs": [],
"source": [
"MODEL_ID = \"facebook/sam3\" # @param [\"facebook/sam3\"] {isTemplate:true}\n",
"ACCELERATOR_TYPE = \"NVIDIA_L4\" # @param [\"NVIDIA_L4\", \"NVIDIA_H100_80GB\"] {isTemplate:true}\n",
"ACCELERATOR_COUNT = 1 # @param [1, 2, 4, 8]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "3XeDOlk2bGwo"
},
"outputs": [],
"source": [
"# @markdown Set `use_dedicated_endpoint` to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint).\n",
"use_dedicated_endpoint = True # @param {type:\"boolean\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "X0NV1guUbGwo"
},
"outputs": [],
"source": [
"models[\"sam3\"], endpoints[\"sam3\"] = deploy_sam3_model(\n",
" MODEL_ID, ACCELERATOR_TYPE, ACCELERATOR_COUNT, use_dedicated_endpoint, HF_TOKEN\n",
")\n",
"endpoint = endpoints[\"sam3\"]\n",
"print(f\"Endpoint deployed: {endpoint.resource_name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wfxEoBhRbGwo"
},
"source": [
"### Connect to Existing Endpoint (Optional)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "vkVvyMv-bGwo"
},
"outputs": [],
"source": [
"# ENDPOINT_ID = \"YOUR_ENDPOINT_ID\" # Uncomment to use existing endpoint\n",
"# endpoint = aiplatform.Endpoint(f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\")\n",
"# endpoints[\"sam3\"] = endpoint"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qwsDVT6xbGwo"
},
"source": [
"## Image Segmentation (Text-Prompted)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "anaMNUePbGwo"
},
"outputs": [],
"source": [
"image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\" # @param {type:\"string\"}\n",
"text_prompt = \"cat\" # @param {type:\"string\"}\n",
"\n",
"original, segmented, masks = segment_image(image_url, text_prompt, endpoint, GCS_BUCKET)\n",
"print(f\"Found {len(masks)} mask(s)\")\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
"axes[0].imshow(original)\n",
"axes[0].set_title(\"Original\")\n",
"axes[0].axis(\"off\")\n",
"axes[1].imshow(segmented)\n",
"axes[1].set_title(f\"'{text_prompt}' ({len(masks)} masks)\")\n",
"axes[1].axis(\"off\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iVsaZ8rFbGwo"
},
"source": [
"## Point-Click Segmentation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "cGleQYk8bGwo"
},
"outputs": [],
"source": [
"click_image = \"http://images.cocodataset.org/val2017/000000039769.jpg\" # @param {type:\"string\"}\n",
"click_points = [[220, 300], [400, 350]] # @param {type:\"raw\"}\n",
"\n",
"original, segmented, masks = segment_by_points(\n",
" click_image, click_points, endpoint, GCS_BUCKET\n",
")\n",
"print(f\"Found {len(masks)} mask(s) for {len(click_points)} point(s)\")\n",
"\n",
"fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
"axes[0].imshow(draw_points(original, click_points))\n",
"axes[0].set_title(\"Click Points\")\n",
"axes[0].axis(\"off\")\n",
"axes[1].imshow(segmented)\n",
"axes[1].set_title(f\"Segmentation ({len(masks)} masks)\")\n",
"axes[1].axis(\"off\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yT351_IvbGwp"
},
"source": [
"## Video Segmentation\n",
"\n",
"> **Note:** Video segmentation requires a GCS bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "nCBEWrE-bGwp"
},
"outputs": [],
"source": [
"video_url = \"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4\" # @param {type:\"string\"}\n",
"video_prompt = \"person\" # @param {type:\"string\"}\n",
"frame_limit = 60 # @param {type:\"integer\"}\n",
"\n",
"if video_url and GCS_BUCKET:\n",
" # Download video from URL to temporary file\n",
" if video_url.startswith((\"http://\", \"https://\")):\n",
" print(f\"Downloading video from {video_url}...\")\n",
" video_response = requests.get(video_url, timeout=400, stream=True)\n",
" video_response.raise_for_status()\n",
" video_path = tempfile.mktemp(suffix=\".mp4\")\n",
" with open(video_path, \"wb\") as f:\n",
" for chunk in video_response.iter_content(chunk_size=8192):\n",
" f.write(chunk)\n",
" print(f\"Downloaded to temporary file: {video_path}\")\n",
" cleanup_temp_video = True\n",
" else:\n",
" video_path = video_url\n",
" cleanup_temp_video = False\n",
"\n",
" try:\n",
" out_path, frames, status = segment_video(\n",
" video_path, video_prompt, endpoint, GCS_BUCKET, frame_limit\n",
" )\n",
" print(f\"{status}. Output: {out_path}\")\n",
"\n",
" # Display sample frames\n",
" if frames:\n",
" indices = np.linspace(0, len(frames) - 1, min(6, len(frames)), dtype=int)\n",
" fig, axes = plt.subplots(2, 3, figsize=(15, 8))\n",
" for i, ax in enumerate(axes.flat):\n",
" if i < len(indices):\n",
" ax.imshow(frames[indices[i]])\n",
" ax.set_title(f\"Frame {indices[i]}\")\n",
" ax.axis(\"off\")\n",
" plt.suptitle(f\"Video: '{video_prompt}'\")\n",
" plt.tight_layout()\n",
" plt.show()\n",
" finally:\n",
" if cleanup_temp_video and os.path.exists(video_path):\n",
" os.unlink(video_path)\n",
"else:\n",
" print(\"Set video_url and ensure GCS_BUCKET (BUCKET_URI) is configured.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y1zyxQZFbGwp"
},
"source": [
"## Clean Up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "uL-KrhjCbGwp"
},
"outputs": [],
"source": [
"# @markdown Delete the experiment models and endpoints to recycle the resources\n",
"# @markdown and avoid unnecessary continuous charges that may incur.\n",
"\n",
"# Undeploy model and delete endpoint.\n",
"for endpoint in endpoints.values():\n",
" endpoint.delete(force=True)\n",
"\n",
"# Delete models.\n",
"for model in models.values():\n",
" model.delete()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "qQTsiBMqbGwp"
},
"outputs": [],
"source": [
"# @markdown Delete temporary GCS buckets.\n",
"\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
}
],
"metadata": {
"colab": {
"name": "model_garden_pytorch_sam3.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}