xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: mistral ai sdk support for vertexai (#3601)

Browse Source 
* feat: mistral ai sdk support fro vertexai

* feat: mistral ai sdk support fro vertexai, token fix

* feat: mistral ai sdk support fro vertexai, excep handling
This commit is contained in:
Sujit Khasnis
2024-10-01 14:59:10 +00:00
committed by GitHub
parent daf92aa672
commit c5ec7e5d83
1 changed files with 426 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/generative_ai/mistralai_intro.ipynb
+426 -43
View File
@@ -93,7 +93,7 @@
"- Mistral on Model Garden supports the same API calls as Mistral’s own API endpoints, except for the `safe_prompt` parameter that will return an error if specified in the input. So do not include `safe_prompt` in input requests.\n",
"- Documentation links\n",
" - [Mistral APIs](https://docs.mistral.ai/api/)\n",
" - [Chat Completions](https://docs.mistral.ai/api/#operation/createChatCompletion) operations supported by Mistral Large, Mistral Nemo and Codestral\n",
" - [Chat Completion](https://docs.mistral.ai/api/#operation/createChatCompletion) operations supported by Mistral Large, Mistral Nemo and Codestral\n",
" - [Fill-in-the-middle](https://docs.mistral.ai/api/#operation/createFIMCompletion) operations supported by Codestral"
]
},
@@ -146,7 +146,7 @@
"id": "2fxZn4SAbxdl"
},
"source": [
"### Select one of Mistral AI models"
"#### Select one of Mistral AI models"
]
},
{
@@ -175,7 +175,7 @@
"id": "bpuX3sKtexlK"
},
"source": [
"### Select a location and a version from the dropdown"
"#### Select a location and a version from the dropdown"
]
},
{
@@ -233,7 +233,7 @@
"id": "3q58icinBjoK"
},
"source": [
"### Set Google Cloud project and model information\n",
"#### Set Google Cloud project and model information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
@@ -260,7 +260,7 @@
"id": "4NAstKRFBt4N"
},
"source": [
"### Import required libraries"
"#### Import required libraries"
]
},
{
@@ -283,7 +283,7 @@
"id": "vVVnOhvE_PA6"
},
"source": [
"## Sample Requests"
"### Sample Requests"
]
},
{
@@ -292,7 +292,7 @@
"id": "5ahw-uFjCAbo"
},
"source": [
"### Text generation"
"#### Text generation"
]
},
{
@@ -301,7 +301,7 @@
"id": "61107099357a"
},
"source": [
"#### Unary call\n",
"##### Unary call\n",
"\n",
"Sends a POST request to the specified API endpoint to get a response from the model using the provided payload."
]
@@ -384,7 +384,7 @@
"id": "e6f52fae9379"
},
"source": [
"#### Streaming call\n",
"##### Streaming call\n",
"\n",
"Sends a POST request to the specified API endpoint to stream a response from the model using the provided payload."
]
@@ -414,9 +414,9 @@
"id": "X6HolebUhShT"
},
"source": [
"### Code generation\n",
"#### Code generation\n",
"\n",
"Mistral Large, Mistral Nemo and Codestral support code generation with the Chat Completions operations covered above.\n",
"Mistral Large, Mistral Nemo and Codestral support code generation with the Chat Completion operations covered above.\n",
"\n",
"With Codestral, you can also do Fill-in-the-middle operations."
]
@@ -427,7 +427,7 @@
"id": "deUZQwgSheEr"
},
"source": [
"#### Fill-in-the-middle (FIM)\n",
"##### Fill-in-the-middle (FIM)\n",
"With this feature, users can define the starting point of the code using a `prompt`, and the ending point of the code using an optional `suffix` and an optional `stop`.\n",
"\n",
"The Codestral model will then generate the code that fits in between, making it ideal for tasks that require a specific piece of code to be generated.\n",
@@ -485,7 +485,6 @@
"outputs": [],
"source": [
"MODEL = \"codestral\"\n",
"SELECTED_MODEL_VERSION = \"\"\n",
"\n",
"# Get the access token\n",
"process = subprocess.Popen(\n",
@@ -501,7 +500,7 @@
"}\n",
"\n",
"# Replace with your actual values\n",
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}{SELECTED_MODEL_VERSION}:rawPredict\"\n",
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}:rawPredict\"\n",
"data = {\n",
" \"model\": MODEL,\n",
" \"prompt\": \"def f(\",\n",
@@ -528,10 +527,40 @@
{
"cell_type": "markdown",
"metadata": {
"id": "qsJLkqyyztR_"
"id": "-G8kI3S_FUjM"
},
"source": [
"## Model Capabilities"
"## Using Mistral AI's Vertex SDK for *Python*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pdzqqcJKOYHa"
},
"source": [
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_TGDKo-vFn5F"
},
"source": [
"### Install Mistral's Vertex SDK for Python and other required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2AdfzWYfFw3u"
},
"outputs": [],
"source": [
"! pip3 install -U -q 'mistralai[gcp]>=1.0.3'\n",
"! pip3 install -U -q httpx"
]
},
{
@@ -543,6 +572,8 @@
"outputs": [],
"source": [
"# Get the access token\n",
"import subprocess\n",
"\n",
"process = subprocess.Popen(\n",
" \"gcloud auth print-access-token\", stdout=subprocess.PIPE, shell=True\n",
")\n",
@@ -555,6 +586,382 @@
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mTbP9RNlF1nO"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FCjYtEc8F437"
},
"outputs": [],
"source": [
"# Restart kernel after installs so that your environment can access the new packages\n",
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Fgq8ZhyuF8wx"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3nepikRF_Gt"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-qgHPk0IGBZX"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ICjT5ZNGF2c"
},
"source": [
"#### Select one of Mistral AI models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NBRTMXhmGIlN"
},
"outputs": [],
"source": [
"MODEL = \"mistral-large\" # @param [\"mistral-large\", \"mistral-nemo\", \"codestral\"]\n",
"if MODEL == \"mistral-large\":\n",
" available_regions = [\"europe-west4\", \"us-central1\"]\n",
" available_versions = [\"2407\"]\n",
"elif MODEL == \"mistral-nemo\":\n",
" available_regions = [\"europe-west4\", \"us-central1\"]\n",
" available_versions = [\"2407\"]\n",
"elif MODEL == \"codestral\":\n",
" available_regions = [\"europe-west4\", \"us-central1\"]\n",
" available_versions = [\"2405\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "227_bG2OGNec"
},
"source": [
"#### Select a location and a version from the dropdown"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uoP1c89sGP7k"
},
"outputs": [],
"source": [
"import ipywidgets as widgets\n",
"from IPython.display import display\n",
"\n",
"dropdown_loc = widgets.Dropdown(\n",
" options=available_regions,\n",
" description=\"Select a location:\",\n",
" font_weight=\"bold\",\n",
" style={\"description_width\": \"initial\"},\n",
")\n",
"\n",
"dropdown_ver = widgets.Dropdown(\n",
" options=available_versions,\n",
" description=\"Select the model version (optional):\",\n",
" font_weight=\"bold\",\n",
" style={\"description_width\": \"initial\"},\n",
")\n",
"\n",
"\n",
"def dropdown_loc_eventhandler(change):\n",
" global LOCATION\n",
" if change[\"type\"] == \"change\" and change[\"name\"] == \"value\":\n",
" LOCATION = change.new\n",
" print(\"Selected:\", change.new)\n",
"\n",
"\n",
"def dropdown_ver_eventhandler(change):\n",
" global MODEL_VERSION\n",
" if change[\"type\"] == \"change\" and change[\"name\"] == \"value\":\n",
" MODEL_VERSION = change.new\n",
" print(\"Selected:\", change.new)\n",
"\n",
"\n",
"LOCATION = dropdown_loc.value\n",
"dropdown_loc.observe(dropdown_loc_eventhandler, names=\"value\")\n",
"display(dropdown_loc)\n",
"\n",
"MODEL_VERSION = dropdown_ver.value\n",
"dropdown_ver.observe(dropdown_ver_eventhandler, names=\"value\")\n",
"display(dropdown_ver)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mghCnsVbK5YP"
},
"source": [
"#### Set Google Cloud project and model information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NB-_kGiJLDcT"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"ENDPOINT = f\"https://{LOCATION}-aiplatform.googleapis.com\"\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" raise ValueError(\"Please set your PROJECT_ID\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yD9oKNOGMA18"
},
"source": [
"#### Import required libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hgmj4wwLMYgd"
},
"outputs": [],
"source": [
"import json\n",
"\n",
"import requests"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pQnpW1WLO3uX"
},
"source": [
"### Sample Requests"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tH0QFYIXMZNZ"
},
"source": [
"#### Text generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tm2rUyXtPgp2"
},
"source": [
"##### Unary call\n",
"\n",
"Initializes a client for Mistral AI's Vertex AI, sends a request to generate the content, and prints the response in a formatted JSON"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cRqiGVUdMgXs"
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"from mistralai_gcp import MistralGoogleCloud\n",
"\n",
"client = MistralGoogleCloud(\n",
" access_token=access_token, region=LOCATION, project_id=PROJECT_ID\n",
")\n",
"\n",
"try:\n",
" resp = client.chat.complete(\n",
" model=f\"{MODEL}-{MODEL_VERSION}\",\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"Who is the best French painter? Answer in one short sentence.\",\n",
" }\n",
" ],\n",
" )\n",
" print(resp.choices[0].message.content)\n",
"\n",
"except Exception as e:\n",
" print(f\"An error occurred: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EWkLsn5GQrqu"
},
"source": [
"##### Streaming call\n",
"\n",
"Initializes a client for Mistral AI's Vertex AI, sends a streaming request to generate the content, and continuously prints the received text as it is streamed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2uOfrmmRQ-rJ"
},
"outputs": [],
"source": [
"from mistralai_gcp import MistralGoogleCloud\n",
"\n",
"client = MistralGoogleCloud(\n",
" access_token=access_token, region=LOCATION, project_id=PROJECT_ID\n",
")\n",
"\n",
"try:\n",
" stream = client.chat.stream(\n",
" model=f\"{MODEL}-{MODEL_VERSION}\",\n",
" max_tokens=1024,\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"Who is the best French painter? Answer in one short sentence.\",\n",
" }\n",
" ],\n",
" )\n",
"\n",
" for chunk in stream:\n",
" print(chunk.data.choices[0].delta.content)\n",
"\n",
"except Exception as e:\n",
" print(f\"An error occurred: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PEZcdznmRxT6"
},
"source": [
"#### Code generation\n",
"\n",
"Mistral Large, Mistral Nemo and Codestral support code generation with the Chat Completion operations covered above.\n",
"\n",
"With Codestral, you can also do Fill-in-the-middle operations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BWl3pMDOXdEZ"
},
"source": [
"##### Fill-in-the-middle (FIM)\n",
"With this feature, users can define the starting point of the code using a `prompt`, and the ending point of the code using an optional `suffix` and an optional `stop`.\n",
"\n",
"The Codestral model will then generate the code that fits in between, making it ideal for tasks that require a specific piece of code to be generated.\n",
"\n",
"More information on FIM:\n",
"- [Mistral API Documentation FIM](https://docs.mistral.ai/api/#operation/createFIMCompletion)\n",
"- [Mistral FIM Documentation](https://docs.mistral.ai/capabilities/code_generation/#fill-in-the-middle-endpoint)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "o61l9pOsTUc-"
},
"outputs": [],
"source": [
"from mistralai_gcp import MistralGoogleCloud\n",
"\n",
"client = MistralGoogleCloud(\n",
" access_token=access_token, region=LOCATION, project_id=PROJECT_ID\n",
")\n",
"\n",
"MODEL = \"codestral\"\n",
"MODEL_VERSION = \"2405\"\n",
"\n",
"try:\n",
" resp = client.fim.complete(\n",
" model=f\"{MODEL}-{MODEL_VERSION}\",\n",
" prompt=\"def count_words_in_file(file_path: str) -> int\",\n",
" suffix=\"return n_words\",\n",
" )\n",
"\n",
" print(resp.choices[0].message.content)\n",
"\n",
"except Exception as e:\n",
" print(f\"An error occurred: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qsJLkqyyztR_"
},
"source": [
"## Model Capabilities"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -705,18 +1112,6 @@
"#### Step 2. Model: Generate the right tool and arguments with Mistral Large"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XyrgEDRu2gc2"
},
"outputs": [],
"source": [
"MODEL = \"mistral-large\"\n",
"SELECTED_MODEL_VERSION = \"\""
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -725,7 +1120,7 @@
},
"outputs": [],
"source": [
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}{SELECTED_MODEL_VERSION}:rawPredict\"\n",
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}:rawPredict\"\n",
"data = {\n",
" \"model\": MODEL,\n",
" \"messages\": [\n",
@@ -837,18 +1232,6 @@
"* See Mistral's API [documentation](https://docs.mistral.ai/api/#operation/createChatCompletion)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "M2-4H54gnizf"
},
"outputs": [],
"source": [
"MODEL = \"mistral-large\"\n",
"SELECTED_MODEL_VERSION = \"\""
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -869,7 +1252,7 @@
"}\n",
"\n",
"request = json.dumps(PAYLOAD)\n",
"!curl -X POST -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}{SELECTED_MODEL_VERSION}:rawPredict -d '{request}'"
"!curl -X POST -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" {ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}:rawPredict -d '{request}'"
]
},
{
@@ -897,7 +1280,7 @@
"access_token = access_token_bytes.decode(\"utf-8\").strip() # Strip newline\n",
"\n",
"# Replace with your actual values\n",
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}{SELECTED_MODEL_VERSION}:rawPredict\"\n",
"url = f\"{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/mistralai/models/{MODEL}:rawPredict\"\n",
"data = {\n",
" \"model\": MODEL,\n",
" \"messages\": [\n",