diff --git a/notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb b/notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb new file mode 100644 index 000000000..14d692f0a --- /dev/null +++ b/notebooks/community/model_garden/model_garden_pytorch_sam3.ipynb @@ -0,0 +1,841 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "KEiqUBnnbGwo" + }, + "outputs": [], + "source": [ + "# Copyright 2026 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "idcJ3uajbGwo" + }, + "source": [ + "# Vertex AI Model Garden - SAM 3 (Segment Anything Model 3)\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Workbench
Run in Workbench\n", + " \n", + " 		\n", + " \n", + " \"Google
Run in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rzbW4muObGwo" + }, + "source": [ + "## Overview\n", + "\n", + "This notebook demonstrates deploying **SAM 3** on Vertex AI for:\n", + "- **Image Segmentation** (text-prompted)\n", + "- **Point-Click Segmentation** (coordinate-based)\n", + "- **Video Segmentation** (object tracking)\n", + "\n", + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n", + "\n", + "### File a bug\n", + "\n", + "File a bug on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new) if you encounter any issue with the notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_TUvfvA-bGwo" + }, + "source": [ + "## Setup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "yFIKRg-pbGwo" + }, + "outputs": [], + "source": [ + "%pip install --upgrade --quiet 'google-cloud-aiplatform>=1.106.0' 'google-cloud-storage' 'pycocotools' 'numpy<2.0' 'opencv-python-headless' 'matplotlib' 'Pillow==11.3.0'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "fTVQfwSJbGwo" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "0YdXF0EGk1Aj" + }, + "outputs": [], + "source": [ + "import base64\n", + "import io\n", + "import re\n", + "import subprocess\n", + "import tempfile\n", + "import uuid\n", + "\n", + "import cv2\n", + "import matplotlib\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import requests\n", + "from google.cloud import aiplatform, storage\n", + "from PIL import Image, ImageDraw\n", + "from pycocotools import mask as mask_utils" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "vnTVbHMibGwo" + }, + "outputs": [], + "source": [ + "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "# @markdown 2. **[Optional]** [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n", + "\n", + "BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n", + "\n", + "# @markdown 3. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n", + "\n", + "REGION = \"\" # @param {type:\"string\"}\n", + "\n", + "\n", + "# Get the default cloud project id.\n", + "PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n", + "\n", + "# Get the default region for launching jobs.\n", + "if not REGION:\n", + " if not os.environ.get(\"GOOGLE_CLOUD_REGION\"):\n", + " raise ValueError(\n", + " \"REGION must be set. See\"\n", + " \" https://cloud.google.com/vertex-ai/docs/general/locations for\"\n", + " \" available cloud locations.\"\n", + " )\n", + " REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n", + "\n", + "# Enable the Vertex AI API and Compute Engine API, if not already.\n", + "print(\"Enabling Vertex AI API and Compute Engine API.\")\n", + "! gcloud services enable aiplatform.googleapis.com compute.googleapis.com\n", + "\n", + "# Initialize Vertex AI API.\n", + "print(\"Initializing Vertex AI API.\")\n", + "aiplatform.init(project=PROJECT_ID, location=REGION)\n", + "! gcloud config set project $PROJECT_ID\n", + "\n", + "models, endpoints = {}, {}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "i7Z2h9aJbGwo" + }, + "outputs": [], + "source": [ + "# GCS bucket for large images/videos (required for files > 1.1MB)\n", + "GCS_BUCKET = BUCKET_URI if BUCKET_URI and BUCKET_URI != \"gs://\" else None\n", + "\n", + "# Hugging Face token (required for gated model facebook/sam3)\n", + "HF_TOKEN = \"\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "pm0Ybff-bGwo" + }, + "outputs": [], + "source": [ + "# SAM3 Configuration\n", + "SAM3_DOCKER_URI = \"us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/pytorch-inference.cu125.0-4.ubuntu2204.py310\"\n", + "DEFAULT_MASK_BLUR_SIGMA = 3.5\n", + "\n", + "# GPU Configuration: (accelerator_type, gpu_count) -> machine_type\n", + "GPU_MACHINE_TYPE_MAP = {\n", + " (\"NVIDIA_L4\", 1): \"g2-standard-12\",\n", + " (\"NVIDIA_L4\", 2): \"g2-standard-24\",\n", + " (\"NVIDIA_L4\", 4): \"g2-standard-48\",\n", + " (\"NVIDIA_L4\", 8): \"g2-standard-96\",\n", + " (\"NVIDIA_H100_80GB\", 1): \"a3-highgpu-1g\",\n", + " (\"NVIDIA_H100_80GB\", 2): \"a3-highgpu-2g\",\n", + " (\"NVIDIA_H100_80GB\", 4): \"a3-highgpu-4g\",\n", + " (\"NVIDIA_H100_80GB\", 8): \"a3-highgpu-8g\",\n", + "}\n", + "\n", + "\n", + "def deploy_sam3_model(\n", + " model_id: str,\n", + " accelerator_type: str = \"NVIDIA_L4\",\n", + " accelerator_count: int = 1,\n", + " use_dedicated_endpoint: bool = True,\n", + " hf_token: str = None,\n", + ") -> tuple:\n", + " \"\"\"Deploy SAM3 model to Vertex AI endpoint.\"\"\"\n", + " machine_type = GPU_MACHINE_TYPE_MAP[(accelerator_type, accelerator_count)]\n", + "\n", + " endpoint = aiplatform.Endpoint.create(\n", + " display_name=\"sam3-endpoint-notebook\",\n", + " dedicated_endpoint_enabled=use_dedicated_endpoint,\n", + " )\n", + "\n", + " env_vars = {\n", + " \"MODEL_ID\": model_id,\n", + " \"TASK\": \"mask-generation\",\n", + " \"DEPLOY_SOURCE\": \"notebook\",\n", + " }\n", + " if hf_token:\n", + " env_vars[\"HF_TOKEN\"] = hf_token\n", + "\n", + " model = aiplatform.Model.upload(\n", + " display_name=\"sam3-notebook\",\n", + " serving_container_image_uri=SAM3_DOCKER_URI,\n", + " serving_container_ports=[8080],\n", + " serving_container_predict_route=\"/predict\",\n", + " serving_container_health_route=\"/health\",\n", + " serving_container_environment_variables=env_vars,\n", + " serving_container_shared_memory_size_mb=(16 * 1024),\n", + " serving_container_deployment_timeout=7200,\n", + " )\n", + "\n", + " model.deploy(\n", + " endpoint=endpoint,\n", + " machine_type=machine_type,\n", + " accelerator_type=accelerator_type,\n", + " accelerator_count=accelerator_count,\n", + " min_replica_count=1,\n", + " max_replica_count=1,\n", + " deploy_request_timeout=1800,\n", + " )\n", + " return model, endpoint\n", + "\n", + "\n", + "def _get_auth_headers():\n", + " token = (\n", + " subprocess.check_output([\"gcloud\", \"auth\", \"print-access-token\"])\n", + " .decode()\n", + " .strip()\n", + " )\n", + " return {\"Authorization\": f\"Bearer {token}\", \"Content-Type\": \"application/json\"}\n", + "\n", + "\n", + "def _get_dedicated_dns(endpoint):\n", + " try:\n", + " if (\n", + " hasattr(endpoint, \"gca_resource\")\n", + " and endpoint.gca_resource.dedicated_endpoint_dns\n", + " ):\n", + " return endpoint.gca_resource.dedicated_endpoint_dns\n", + " except:\n", + " pass\n", + " return None\n", + "\n", + "\n", + "def call_sam3_endpoint(payload, endpoint, timeout=180):\n", + " \"\"\"Call SAM3 Vertex AI endpoint with automatic dedicated DNS resolution.\"\"\"\n", + " endpoint_id = endpoint.name.split(\"/\")[-1]\n", + " dns = _get_dedicated_dns(endpoint)\n", + " host = dns if dns else f\"{REGION}-aiplatform.googleapis.com\"\n", + " url = f\"https://{host}/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_id}:predict\"\n", + "\n", + " response = requests.post(\n", + " url, json=payload, headers=_get_auth_headers(), timeout=timeout\n", + " )\n", + " if response.status_code == 400 and \"dedicated domain name\" in response.text:\n", + " dns = re.search(r\"dedicated domain name '([^']+)'\", response.text).group(1)\n", + " url = f\"https://{dns}/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_id}:predict\"\n", + " response = requests.post(\n", + " url, json=payload, headers=_get_auth_headers(), timeout=timeout\n", + " )\n", + "\n", + " if response.status_code != 200:\n", + " raise RuntimeError(\n", + " f\"Vertex AI error {response.status_code}: {response.text[:500]}\"\n", + " )\n", + " return response.json()\n", + "\n", + "\n", + "def decode_rle_mask(rle):\n", + " counts = (\n", + " rle[\"counts\"].encode(\"utf-8\")\n", + " if isinstance(rle[\"counts\"], str)\n", + " else rle[\"counts\"]\n", + " )\n", + " return mask_utils.decode({\"size\": rle[\"size\"], \"counts\": counts}).astype(np.uint8)\n", + "\n", + "\n", + "def image_to_base64(img):\n", + " buf = io.BytesIO()\n", + " img.save(buf, format=\"PNG\")\n", + " return base64.b64encode(buf.getvalue()).decode()\n", + "\n", + "\n", + "def load_image(path_or_url):\n", + " if path_or_url.startswith((\"http://\", \"https://\")):\n", + " return Image.open(\n", + " io.BytesIO(requests.get(path_or_url, timeout=30).content)\n", + " ).convert(\"RGB\")\n", + " return Image.open(path_or_url).convert(\"RGB\")\n", + "\n", + "\n", + "def upload_to_gcs(data, gcs_uri, prefix, content_type, is_file=False):\n", + " \"\"\"Upload data or file to GCS. Returns gs:// URI.\"\"\"\n", + " bucket_name, path = (\n", + " gcs_uri[5:].split(\"/\", 1) if \"/\" in gcs_uri[5:] else (gcs_uri[5:], \"\")\n", + " )\n", + " ext = \".mp4\" if \"video\" in content_type else \".png\"\n", + " blob_path = f\"{path}/{prefix}-{uuid.uuid4().hex[:8]}{ext}\".lstrip(\"/\")\n", + "\n", + " blob = storage.Client().bucket(bucket_name).blob(blob_path)\n", + " if is_file:\n", + " blob.upload_from_filename(data, content_type=content_type)\n", + " else:\n", + " blob.upload_from_string(data, content_type=content_type)\n", + " return f\"gs://{bucket_name}/{blob_path}\"\n", + "\n", + "\n", + "def delete_gcs_file(gcs_uri):\n", + " try:\n", + " path = gcs_uri[5:]\n", + " bucket_name, blob_path = path.split(\"/\", 1)\n", + " storage.Client().bucket(bucket_name).blob(blob_path).delete()\n", + " except:\n", + " pass\n", + "\n", + "\n", + "def apply_mask_overlay(image, masks, opacity=0.5):\n", + " \"\"\"Overlay colored masks on image.\"\"\"\n", + " if isinstance(image, np.ndarray):\n", + " image = Image.fromarray(image)\n", + " image = image.convert(\"RGBA\")\n", + " if not masks:\n", + " return image.convert(\"RGB\")\n", + "\n", + " cmap = matplotlib.colormaps[\"rainbow\"].resampled(len(masks))\n", + " composite = Image.new(\"RGBA\", image.size, (0, 0, 0, 0))\n", + "\n", + " for i, mask in enumerate(masks):\n", + " color = tuple(int(c * 255) for c in cmap(i)[:3])\n", + " mask_img = Image.fromarray((mask * 255).astype(np.uint8))\n", + " if mask_img.size != image.size:\n", + " mask_img = mask_img.resize(image.size, Image.NEAREST)\n", + " fill = Image.new(\"RGBA\", image.size, color + (0,))\n", + " fill.putalpha(mask_img.point(lambda v: int(v * opacity) if v > 0 else 0))\n", + " composite = Image.alpha_composite(composite, fill)\n", + "\n", + " return Image.alpha_composite(image, composite).convert(\"RGB\")\n", + "\n", + "\n", + "def draw_points(image, points):\n", + " \"\"\"Draw red circles at click points.\"\"\"\n", + " if isinstance(image, np.ndarray):\n", + " image = Image.fromarray(image)\n", + " img = image.copy()\n", + " draw = ImageDraw.Draw(img)\n", + " for x, y in points:\n", + " draw.ellipse((x - 8, y - 8, x + 8, y + 8), fill=\"red\", outline=\"white\", width=4)\n", + " return img\n", + "\n", + "\n", + "def segment_image(\n", + " image_path_or_url,\n", + " text_prompt,\n", + " endpoint,\n", + " gcs_bucket=None,\n", + " blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n", + "):\n", + " \"\"\"Text-prompted image segmentation. Returns (original, overlay, masks).\"\"\"\n", + " img = load_image(image_path_or_url)\n", + " img_b64 = image_to_base64(img)\n", + " gcs_uri = None\n", + "\n", + " # Upload large images to GCS\n", + " if len(img_b64) * 0.75 > 1.1 * 1024 * 1024:\n", + " if not gcs_bucket:\n", + " raise RuntimeError(\"Image too large. Set GCS_BUCKET for large images.\")\n", + " buf = io.BytesIO()\n", + " img.save(buf, format=\"PNG\")\n", + " gcs_uri = upload_to_gcs(buf.getvalue(), gcs_bucket, \"sam3-img\", \"image/png\")\n", + " payload = {\n", + " \"instances\": [\n", + " {\"image\": gcs_uri, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n", + " ],\n", + " \"parameters\": {\"mask_format\": \"rle\"},\n", + " }\n", + " else:\n", + " payload = {\n", + " \"instances\": [\n", + " {\"image\": img_b64, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n", + " ],\n", + " \"parameters\": {\"mask_format\": \"rle\"},\n", + " }\n", + "\n", + " try:\n", + " result = call_sam3_endpoint(payload, endpoint)\n", + " finally:\n", + " if gcs_uri:\n", + " delete_gcs_file(gcs_uri)\n", + "\n", + " masks = [\n", + " decode_rle_mask(rle)\n", + " for rle in result.get(\"predictions\", [{}])[0].get(\"masks_rle\", [])\n", + " ]\n", + " return img, apply_mask_overlay(img, masks), masks\n", + "\n", + "\n", + "def segment_by_points(\n", + " image_path_or_url,\n", + " points,\n", + " endpoint,\n", + " gcs_bucket=None,\n", + " blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n", + "):\n", + " \"\"\"Point-click segmentation. Returns (original, overlay with points, masks).\"\"\"\n", + " img = load_image(image_path_or_url)\n", + " img_b64 = image_to_base64(img)\n", + " gcs_uri = None\n", + "\n", + " if len(img_b64) * 0.75 > 1.1 * 1024 * 1024:\n", + " if not gcs_bucket:\n", + " raise RuntimeError(\"Image too large. Set GCS_BUCKET for large images.\")\n", + " buf = io.BytesIO()\n", + " img.save(buf, format=\"PNG\")\n", + " gcs_uri = upload_to_gcs(buf.getvalue(), gcs_bucket, \"sam3-click\", \"image/png\")\n", + " payload = {\n", + " \"instances\": [\n", + " {\n", + " \"image\": gcs_uri,\n", + " \"input_points\": points,\n", + " \"mask_blur_sigma\": blur_sigma,\n", + " }\n", + " ],\n", + " \"parameters\": {\"mask_format\": \"rle\"},\n", + " }\n", + " else:\n", + " payload = {\n", + " \"instances\": [\n", + " {\n", + " \"image\": img_b64,\n", + " \"input_points\": points,\n", + " \"mask_blur_sigma\": blur_sigma,\n", + " }\n", + " ],\n", + " \"parameters\": {\"mask_format\": \"rle\"},\n", + " }\n", + "\n", + " try:\n", + " result = call_sam3_endpoint(payload, endpoint)\n", + " finally:\n", + " if gcs_uri:\n", + " delete_gcs_file(gcs_uri)\n", + "\n", + " masks = [\n", + " decode_rle_mask(rle)\n", + " for rle in result.get(\"predictions\", [{}])[0].get(\"masks_rle\", [])\n", + " ]\n", + " overlay = draw_points(apply_mask_overlay(img, masks), points)\n", + " return img, overlay, masks\n", + "\n", + "\n", + "def segment_video(\n", + " video_path,\n", + " text_prompt,\n", + " endpoint,\n", + " gcs_bucket,\n", + " frame_limit=60,\n", + " timeout=1600,\n", + " blur_sigma=DEFAULT_MASK_BLUR_SIGMA,\n", + "):\n", + " \"\"\"Video segmentation. Returns (output_path, sample_frames, status).\"\"\"\n", + " cap = cv2.VideoCapture(video_path)\n", + " fps, w, h = (\n", + " cap.get(cv2.CAP_PROP_FPS),\n", + " int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),\n", + " int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)),\n", + " )\n", + "\n", + " frames = []\n", + " while cap.isOpened() and len(frames) < frame_limit:\n", + " ret, frame = cap.read()\n", + " if not ret:\n", + " break\n", + " frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))\n", + " cap.release()\n", + "\n", + " # Write temp video and upload to GCS\n", + " tmp = tempfile.mktemp(suffix=\".mp4\")\n", + " writer = cv2.VideoWriter(tmp, cv2.VideoWriter_fourcc(*\"mp4v\"), fps, (w, h))\n", + " for f in frames:\n", + " writer.write(cv2.cvtColor(f, cv2.COLOR_RGB2BGR))\n", + " writer.release()\n", + "\n", + " gcs_uri = upload_to_gcs(tmp, gcs_bucket, \"sam3-vid\", \"video/mp4\", is_file=True)\n", + " os.unlink(tmp)\n", + "\n", + " try:\n", + " payload = {\n", + " \"instances\": [\n", + " {\"video\": gcs_uri, \"text\": text_prompt, \"mask_blur_sigma\": blur_sigma}\n", + " ],\n", + " \"parameters\": {\"mask_format\": \"rle\"},\n", + " }\n", + " result = call_sam3_endpoint(payload, endpoint, timeout=max(timeout, 1600))\n", + " finally:\n", + " delete_gcs_file(gcs_uri)\n", + "\n", + " masks_video = result.get(\"predictions\", [{}])[0].get(\"masks_rle_video\", [])\n", + "\n", + " # Create output video\n", + " out_path = tempfile.mktemp(suffix=\".mp4\")\n", + " writer = cv2.VideoWriter(out_path, cv2.VideoWriter_fourcc(*\"mp4v\"), fps, (w, h))\n", + " overlay_frames = []\n", + "\n", + " for i, frame_masks in enumerate(masks_video):\n", + " if i >= len(frames):\n", + " break\n", + " decoded = [decode_rle_mask(rle) for rle in frame_masks] if frame_masks else []\n", + " overlay = apply_mask_overlay(Image.fromarray(frames[i]), decoded)\n", + " overlay_frames.append(overlay)\n", + " writer.write(cv2.cvtColor(np.array(overlay), cv2.COLOR_RGB2BGR))\n", + " writer.release()\n", + "\n", + " return out_path, overlay_frames, f\"Processed {len(masks_video)} frames\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pZyx8HJmbGwo" + }, + "source": [ + "## Deploy SAM3 Model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "vQEX7jYGbGwo" + }, + "outputs": [], + "source": [ + "MODEL_ID = \"facebook/sam3\" # @param [\"facebook/sam3\"] {isTemplate:true}\n", + "ACCELERATOR_TYPE = \"NVIDIA_L4\" # @param [\"NVIDIA_L4\", \"NVIDIA_H100_80GB\"] {isTemplate:true}\n", + "ACCELERATOR_COUNT = 1 # @param [1, 2, 4, 8]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "3XeDOlk2bGwo" + }, + "outputs": [], + "source": [ + "# @markdown Set `use_dedicated_endpoint` to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint).\n", + "use_dedicated_endpoint = True # @param {type:\"boolean\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "X0NV1guUbGwo" + }, + "outputs": [], + "source": [ + "models[\"sam3\"], endpoints[\"sam3\"] = deploy_sam3_model(\n", + " MODEL_ID, ACCELERATOR_TYPE, ACCELERATOR_COUNT, use_dedicated_endpoint, HF_TOKEN\n", + ")\n", + "endpoint = endpoints[\"sam3\"]\n", + "print(f\"Endpoint deployed: {endpoint.resource_name}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wfxEoBhRbGwo" + }, + "source": [ + "### Connect to Existing Endpoint (Optional)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "vkVvyMv-bGwo" + }, + "outputs": [], + "source": [ + "# ENDPOINT_ID = \"YOUR_ENDPOINT_ID\" # Uncomment to use existing endpoint\n", + "# endpoint = aiplatform.Endpoint(f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\")\n", + "# endpoints[\"sam3\"] = endpoint" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qwsDVT6xbGwo" + }, + "source": [ + "## Image Segmentation (Text-Prompted)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "anaMNUePbGwo" + }, + "outputs": [], + "source": [ + "image_url = \"http://images.cocodataset.org/val2017/000000039769.jpg\" # @param {type:\"string\"}\n", + "text_prompt = \"cat\" # @param {type:\"string\"}\n", + "\n", + "original, segmented, masks = segment_image(image_url, text_prompt, endpoint, GCS_BUCKET)\n", + "print(f\"Found {len(masks)} mask(s)\")\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", + "axes[0].imshow(original)\n", + "axes[0].set_title(\"Original\")\n", + "axes[0].axis(\"off\")\n", + "axes[1].imshow(segmented)\n", + "axes[1].set_title(f\"'{text_prompt}' ({len(masks)} masks)\")\n", + "axes[1].axis(\"off\")\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iVsaZ8rFbGwo" + }, + "source": [ + "## Point-Click Segmentation" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "cGleQYk8bGwo" + }, + "outputs": [], + "source": [ + "click_image = \"http://images.cocodataset.org/val2017/000000039769.jpg\" # @param {type:\"string\"}\n", + "click_points = [[220, 300], [400, 350]] # @param {type:\"raw\"}\n", + "\n", + "original, segmented, masks = segment_by_points(\n", + " click_image, click_points, endpoint, GCS_BUCKET\n", + ")\n", + "print(f\"Found {len(masks)} mask(s) for {len(click_points)} point(s)\")\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", + "axes[0].imshow(draw_points(original, click_points))\n", + "axes[0].set_title(\"Click Points\")\n", + "axes[0].axis(\"off\")\n", + "axes[1].imshow(segmented)\n", + "axes[1].set_title(f\"Segmentation ({len(masks)} masks)\")\n", + "axes[1].axis(\"off\")\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yT351_IvbGwp" + }, + "source": [ + "## Video Segmentation\n", + "\n", + "> **Note:** Video segmentation requires a GCS bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "nCBEWrE-bGwp" + }, + "outputs": [], + "source": [ + "video_url = \"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4\" # @param {type:\"string\"}\n", + "video_prompt = \"person\" # @param {type:\"string\"}\n", + "frame_limit = 60 # @param {type:\"integer\"}\n", + "\n", + "if video_url and GCS_BUCKET:\n", + " # Download video from URL to temporary file\n", + " if video_url.startswith((\"http://\", \"https://\")):\n", + " print(f\"Downloading video from {video_url}...\")\n", + " video_response = requests.get(video_url, timeout=400, stream=True)\n", + " video_response.raise_for_status()\n", + " video_path = tempfile.mktemp(suffix=\".mp4\")\n", + " with open(video_path, \"wb\") as f:\n", + " for chunk in video_response.iter_content(chunk_size=8192):\n", + " f.write(chunk)\n", + " print(f\"Downloaded to temporary file: {video_path}\")\n", + " cleanup_temp_video = True\n", + " else:\n", + " video_path = video_url\n", + " cleanup_temp_video = False\n", + "\n", + " try:\n", + " out_path, frames, status = segment_video(\n", + " video_path, video_prompt, endpoint, GCS_BUCKET, frame_limit\n", + " )\n", + " print(f\"{status}. Output: {out_path}\")\n", + "\n", + " # Display sample frames\n", + " if frames:\n", + " indices = np.linspace(0, len(frames) - 1, min(6, len(frames)), dtype=int)\n", + " fig, axes = plt.subplots(2, 3, figsize=(15, 8))\n", + " for i, ax in enumerate(axes.flat):\n", + " if i < len(indices):\n", + " ax.imshow(frames[indices[i]])\n", + " ax.set_title(f\"Frame {indices[i]}\")\n", + " ax.axis(\"off\")\n", + " plt.suptitle(f\"Video: '{video_prompt}'\")\n", + " plt.tight_layout()\n", + " plt.show()\n", + " finally:\n", + " if cleanup_temp_video and os.path.exists(video_path):\n", + " os.unlink(video_path)\n", + "else:\n", + " print(\"Set video_url and ensure GCS_BUCKET (BUCKET_URI) is configured.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "y1zyxQZFbGwp" + }, + "source": [ + "## Clean Up" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "uL-KrhjCbGwp" + }, + "outputs": [], + "source": [ + "# @markdown Delete the experiment models and endpoints to recycle the resources\n", + "# @markdown and avoid unnecessary continuous charges that may incur.\n", + "\n", + "# Undeploy model and delete endpoint.\n", + "for endpoint in endpoints.values():\n", + " endpoint.delete(force=True)\n", + "\n", + "# Delete models.\n", + "for model in models.values():\n", + " model.delete()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "qQTsiBMqbGwp" + }, + "outputs": [], + "source": [ + "# @markdown Delete temporary GCS buckets.\n", + "\n", + "delete_bucket = False # @param {type:\"boolean\"}\n", + "if delete_bucket:\n", + " ! gsutil -m rm -r $BUCKET_NAME" + ] + } + ], + "metadata": { + "colab": { + "name": "model_garden_pytorch_sam3.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}