feat:New MARS8 model (#4429)

* feat:New MARS8 model

* feat:New MARS8 model
This commit is contained in:
Sam-Deciga
2026-01-27 10:43:43 -05:00
committed by GitHub
parent 5078c44eb8
commit 665547f790
@@ -0,0 +1,686 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "9A9NkTRTfo2I"
},
"outputs": [],
"source": [
"# Copyright 2026 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IPprg6Oz0QDs"
},
"source": [
"# Getting Started with Camb AI's Models"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8fK_rdvvx1iZ"
},
"source": [
"## Overview\n",
"\n",
"### Camb AI on Vertex AI\n",
"\n",
"You can deploy the Camb AI models in your own endpoint.\n",
"\n",
"### Available Camb AI models\n",
"\n",
"#### MARS8\n",
"\n",
"MARS8 is CAMB.AI's latest speech synthesis model. It's aimed towards high quality, multilingual, low latency TTS outputs.\n",
"It features voicecloning and fine-grained pronunciation control in over 30 languages.\n",
"Deployed on Google Cloud’s Vertex AI Launchpad, it brings latencies of less than 400ms with Blackwell 6000 GPUs, with an architectural focus on speed and voice cloning capabilites.\n",
"\n",
"#### MARS7\n",
"(Multilingual AutoRegressive Speech 7) is the prior generation in CAMB.AI’s MARS series of speech synthesis models. MARS7 creates hyper-realistic, prosodic, multilingual text-to-speech (TTS) outputs, featuring optional voice cloning and fine-grained emotional control. Deployed on Google Cloud’s Vertex AI Launchpad, it brings near real-time latency, with an architectural focus on parameter efficiency and global context understanding.\n",
"\n",
"## Objective\n",
"This notebook shows how to use **Vertex AI API** to deploy the Camb AI models.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lNZFf33uusH9"
},
"source": [
"## Vertex AI API"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nwYvaaW25jYS"
},
"source": [
"## Get Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0660e339bf3f"
},
"source": [
"### Install required packages\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "754611260f53"
},
"outputs": [],
"source": [
"! pip3 install -U -q httpx soundfile\n",
"! command -v jq >/dev/null 2>&1 || { echo >&2 \"jq is not installed. Installing jq...\"; sudo apt-get update -y && sudo apt-get install -y jq; }"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b9f4c57a43f6"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "3b9119a60525"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "u8Mn7YGTfgpc"
},
"outputs": [],
"source": [
"import sys\n",
"import tempfile\n",
"\n",
"import IPython.display as ipd\n",
"import soundfile as sf\n",
"from IPython.display import Audio\n",
"\n",
"\n",
"def display_flac_in_notebook(path_to_flac):\n",
" data, samplerate = sf.read(path_to_flac)\n",
" with tempfile.NamedTemporaryFile(suffix=\".wav\", delete=False) as tmp:\n",
" sf.write(tmp.name, data, samplerate, format=\"WAV\")\n",
" return Audio(filename=tmp.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e767418763cd"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6a5bea26f60f"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "c97be6a73155"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2fxZn4SAbxdl"
},
"source": [
"#### Select one of Camb AI models"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "Y8X70FTSbx7U"
},
"outputs": [],
"source": [
"PUBLISHER_NAME = \"cambai\" # @param {type:\"string\"}\n",
"PUBLISHER_MODEL_NAME = \"mars8\"\n",
"available_regions = [\"us-central1\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bpuX3sKtexlK"
},
"source": [
"#### Select a location and a version from the dropdown"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dHl8xW45ex_O"
},
"outputs": [],
"source": [
"import ipywidgets as widgets\n",
"from IPython.display import display\n",
"\n",
"dropdown_loc = widgets.Dropdown(\n",
" options=available_regions,\n",
" description=\"Select a location:\",\n",
" font_weight=\"bold\",\n",
" style={\"description_width\": \"initial\"},\n",
")\n",
"\n",
"\n",
"def dropdown_loc_eventhandler(change):\n",
" global LOCATION\n",
" if change[\"type\"] == \"change\" and change[\"name\"] == \"value\":\n",
" LOCATION = change.new\n",
" print(\"Selected:\", change.new)\n",
"\n",
"\n",
"LOCATION = dropdown_loc.value\n",
"dropdown_loc.observe(dropdown_loc_eventhandler, names=\"value\")\n",
"display(dropdown_loc)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3q58icinBjoK"
},
"source": [
"#### Set Google Cloud project and model information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "hltNx33t6cSZ"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"ENDPOINT = f\"https://{LOCATION}-aiplatform.googleapis.com\"\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" raise ValueError(\"Please set your PROJECT_ID\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4NAstKRFBt4N"
},
"source": [
"#### Import required libraries"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "QZEFLE6a6bqy"
},
"outputs": [],
"source": [
"import json\n",
"import time"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qjsDpa8jlTRu"
},
"source": [
"### Upload Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "y1R2BRsBlu-k"
},
"outputs": [],
"source": [
"UPLOAD_MODEL_PAYLOAD = {\n",
" \"model\": {\n",
" \"displayName\": \"ModelGarden_LaunchPad_Model_\" + time.strftime(\"%Y%m%d-%H%M%S\"),\n",
" \"baseModelSource\": {\n",
" \"modelGardenSource\": {\n",
" \"publicModelName\": f\"publishers/{PUBLISHER_NAME}/models/{PUBLISHER_MODEL_NAME}\",\n",
" }\n",
" },\n",
" }\n",
"}\n",
"\n",
"request = json.dumps(UPLOAD_MODEL_PAYLOAD)\n",
"\n",
"! curl -X POST -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{LOCATION}/models:upload -d '{request}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V2j0nVGwlf9b"
},
"source": [
"#### Get Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bxwM0GXTmQhh"
},
"outputs": [],
"source": [
"MODEL_ID = -1 # @param {type: \"number\"}\n",
"# copy the model id from the upload in the previous cell into the model id above\n",
"! curl -X GET -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/models/{MODEL_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3q3ygq8VlZAp"
},
"source": [
"### Create the endpoint"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "O1ChDOt7mPBQ"
},
"outputs": [],
"source": [
"CREATE_ENDPOINT_PAYLOAD = {\n",
" \"displayName\": \"ModelGarden_LaunchPad_Endpoint_\" + time.strftime(\"%Y%m%d-%H%M%S\"),\n",
"}\n",
"\n",
"request = json.dumps(CREATE_ENDPOINT_PAYLOAD)\n",
"\n",
"! curl -X POST -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/endpoints -d '{request}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GuMZCdhmlpCE"
},
"source": [
"#### Get Endpoint"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tHq_cLT6mPp_"
},
"outputs": [],
"source": [
"ENDPOINT_ID = -1 # @param {type: \"number\"}\n",
"# copy the endpoint id from the create endpoint in the previous cell into the endpoint id above\n",
"! curl -X GET -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "G0amEPXolbP7"
},
"source": [
"### Deploy Model"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "Ucj-Xa-fpGrg"
},
"outputs": [],
"source": [
"MACHINE_TYPE = \"g4-standard-48\" # @param {type: \"string\"}\n",
"ACCELERATOR_TYPE = \"NVIDIA_RTX_PRO_6000\" # @param {type: \"string\"}\n",
"ACCELERATOR_COUNT = 1 # @param {type: \"number\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VGTyCQQhlrAR"
},
"outputs": [],
"source": [
"# Try CURL, if it fails, use stubby command in the next cell.\n",
"DEPLOY_PAYLOAD = {\n",
" \"deployedModel\": {\n",
" \"model\": f\"projects/{PROJECT_ID}/locations/{LOCATION}/models/{MODEL_ID}\",\n",
" \"displayName\": \"ModelGarden_LaunchPad_DeployedModel_\"\n",
" + time.strftime(\"%Y%m%d-%H%M%S\"),\n",
" \"dedicatedResources\": {\n",
" \"machineSpec\": {\n",
" \"machineType\": MACHINE_TYPE,\n",
" \"acceleratorType\": ACCELERATOR_TYPE,\n",
" \"acceleratorCount\": ACCELERATOR_COUNT,\n",
" },\n",
" \"minReplicaCount\": 1,\n",
" \"maxReplicaCount\": 1,\n",
" },\n",
" },\n",
" \"trafficSplit\": {\"0\": 100},\n",
"}\n",
"\n",
"request = json.dumps(DEPLOY_PAYLOAD)\n",
"print(\"Request payload to Deploy Model:\")\n",
"print(json.dumps(DEPLOY_PAYLOAD, indent=2))\n",
"print(\"\\nResult:\")\n",
"! curl -X POST -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}:deployModel -d '{request}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ahw-uFjCAbo"
},
"source": [
"### Audio generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61107099357a"
},
"source": [
"#### Unary call\n",
"\n",
"Sends a POST request to the specified API endpoint to get a response from the model for a joke using the provided payload."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "fqq3G6fnDxzR"
},
"outputs": [],
"source": [
"# load in env variables\n",
"import os\n",
"\n",
"os.environ[\"ENDPOINT\"] = ENDPOINT\n",
"os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"os.environ[\"LOCATION\"] = LOCATION\n",
"os.environ[\"ENDPOINT_ID\"] = str(ENDPOINT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_I8EU083e1jy"
},
"outputs": [],
"source": [
"%%bash\n",
"# download the ref file\n",
"curl -o ref.flac \"https://storage.googleapis.com/cambai-prod-public/public/ref.flac\"\n",
"# base64 encode the ref and save the payload:\n",
"b64=$(base64 -i ref.flac | tr -d '\\n')\n",
"cat > body.json <<EOF\n",
"{\n",
" \"text\": \"The quick brown fox jumps over the lazy dog.\",\n",
" \"reference_audio\": \"$b64\",\n",
" \"reference_language\": \"en-us\",\n",
" \"output_duration\": null,\n",
" \"language\": \"en-us\"\n",
"}\n",
"EOF\n",
"\n",
"# send the request, saving the output to a .flac file:\n",
"curl -X POST ${ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION}/endpoints/${ENDPOINT_ID}:rawPredict \\\n",
" -H \"Content-Type: application/json\" \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -d @body.json > output.flac\n",
"# NOTE: the first request to a model endpoint pod might be slow as the model compiles kernels"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XBMjtnUjflpz"
},
"outputs": [],
"source": [
"# play the output audio\n",
"ipd.Audio(\"output.flac\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GhUFddWsDxzR"
},
"source": [
"#### Unary Call with Vertex SDK"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"id": "FJuOQuS1DxzS"
},
"outputs": [],
"source": [
"import base64\n",
"from pathlib import Path\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
"endpoint = aiplatform.Endpoint(endpoint_name=str(ENDPOINT_ID))\n",
"\n",
"# encode reference audio as base64 string:\n",
"file_path = \"ref.flac\"\n",
"encoded_str = base64.b64encode(Path(file_path).read_bytes()).decode(\"utf-8\")\n",
"\n",
"# define payload:\n",
"data = {\n",
" \"text\": \"The quick brown fox jumps over the lazy dog.\",\n",
" \"language\": \"en-us\",\n",
" \"output_duration\": None,\n",
" \"reference_language\": \"en-us\",\n",
" \"reference_audio\": encoded_str,\n",
"}\n",
"\n",
"prediction = []\n",
"for chunk in endpoint.raw_predict(\n",
" body=json.dumps(data).encode(\"utf-8\"),\n",
" headers={\"Content-Type\": \"application/json\"},\n",
" use_dedicated_endpoint=True,\n",
"):\n",
" if chunk:\n",
" prediction.append(chunk)\n",
"full_audio_bytes = b\"\".join(prediction)\n",
"# save output to a local file\n",
"with open(\"output.flac\", \"wb\") as file:\n",
" file.write(full_audio_bytes)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Oo_NeG4LgJsg"
},
"outputs": [],
"source": [
"# play the output audio\n",
"ipd.Audio(\"output.flac\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WmQ-CJs4gbPI"
},
"source": [
"#### Unary call with streaming"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gGfJjOlWgYsZ"
},
"outputs": [],
"source": [
"import base64\n",
"import subprocess\n",
"import time\n",
"from pathlib import Path\n",
"\n",
"import httpx\n",
"\n",
"url = f\"https://{LOCATION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}:rawPredict\"\n",
"# authenticate\n",
"res = subprocess.run([\"gcloud\", \"auth\", \"print-access-token\"], capture_output=True)\n",
"access_token = res.stdout.decode(\"utf-8\").strip()\n",
"headers = {\"Authorization\": f\"Bearer {access_token}\", \"Accept\": \"text/event-stream\"}\n",
"# Define reference audio to use for cloning (base64 encoded)\n",
"file_path = \"ref.flac\"\n",
"encoded_str = base64.b64encode(Path(file_path).read_bytes()).decode(\"utf-8\")\n",
"\n",
"data = {\n",
" \"text\": \"そしてカールソンは、あの軽やかなスケーティングで、まるで幽霊のようにニュートラルゾーンを縫うように駆け抜ける.\", # text to synthesize\n",
" \"language\": \"ja-jp\",\n",
" \"output_duration\": None,\n",
" \"reference_language\": \"en-us\",\n",
" \"reference_audio\": encoded_str,\n",
"}\n",
"\n",
"st = time.time()\n",
"\n",
"prediction = []\n",
"wavs = []\n",
"with httpx.stream(\"POST\", url, headers=headers, json=data, timeout=300) as r:\n",
" print(r.status_code, r.headers)\n",
" dt = time.time()\n",
" for chunk in r.iter_bytes(4096 * 16):\n",
" if chunk:\n",
" # each chunk is a bytes object of the next audio chunk.\n",
" # If you want you can render the output piece by piece, or use an async receiver.\n",
" print(\n",
" f\"Received chunk of size {len(chunk)} at {time.time() - st:.2f}s. w/o network delay: {time.time() - dt:.2f}s\"\n",
" )\n",
" prediction.append(chunk)\n",
"et = time.time()\n",
"full_audio_bytes = b\"\".join(prediction)\n",
"# Check the response status code\n",
"if r.status_code == 200:\n",
" print(f\"Request successful! Took {et-st:.2f}s\")\n",
"else:\n",
" print(\"Request failed with status code\", r.status_code)\n",
"# display or save full output\n",
"with open(\"output.flac\", \"wb\") as file:\n",
" file.write(full_audio_bytes)\n",
"ipd.Audio(full_audio_bytes)"
]
}
],
"metadata": {
"colab": {
"name": "cambai_intro_mars8.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}