mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-27 07:31:58 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Andrew Ferlitsch
|
51f342f88f |
@@ -55,4 +55,3 @@
|
||||
/notebooks/community/model_garden/model_garden_pytorch_clip.ipynb @xiangxu-google
|
||||
/notebooks/community/model_garden/model_garden_pytorch_owlvit.ipynb @xiangxu-google
|
||||
/notebooks/community/model_garden/model_garden_pytorch_layoutml_document_qa.ipynb @xiangxu-google
|
||||
/notebooks/community/model_garden/model_garden_pytorch_blip2.ipynb @xiangxu-google
|
||||
|
||||
@@ -1,468 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7d9bbf86da5e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2023 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2bd716bf3e39"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI Model Garden - BLIP2\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_blip2.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_blip2.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_blip2.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
"Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" (a Python-3 CPU notebook is recommended)\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d8cd12648da4"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates deploying the pre-trained [BLIP2](https://huggingface.co/Salesforce/blip2-opt-2.7b) model on Vertex AI for online prediction.\n",
|
||||
"\n",
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"- Upload the model to [Model Registry](https://cloud.google.com/vertex-ai/docs/model-registry/introduction).\n",
|
||||
"- Deploy the model on [Endpoint](https://cloud.google.com/vertex-ai/docs/predictions/using-private-endpoints).\n",
|
||||
"- Run online predictions for image captioning.\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "264c07757582"
|
||||
},
|
||||
"source": [
|
||||
"## Setup environment\n",
|
||||
"\n",
|
||||
"**NOTE**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d73ffa0c0b83"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2707b02ef5df"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip3 install --upgrade google-cloud-aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b60a4d7100bf"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
"google_auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0f826ff482a2"
|
||||
},
|
||||
"source": [
|
||||
"### Setup Google Cloud project\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
|
||||
"\n",
|
||||
"1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8958ebc71868"
|
||||
},
|
||||
"source": [
|
||||
"Fill following variables for experiments environment:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "9db30f827a65"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Cloud project id.\n",
|
||||
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# The region you want to launch jobs in.\n",
|
||||
"REGION = \"us-central1\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# The Cloud Storage bucket for storing experiments output. Fill it without the 'gs://' prefix.\n",
|
||||
"GCS_BUCKET = \"\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "92f16e22c20b"
|
||||
},
|
||||
"source": [
|
||||
"Initialize Vertex AI API:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1680c257acfb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import aiplatform\n",
|
||||
"\n",
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=GCS_BUCKET)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6ca48b699d17"
|
||||
},
|
||||
"source": [
|
||||
"### Define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "de9882ea89ea"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The pre-built serving docker image.\n",
|
||||
"# The model artifacts are embedded within the container, except for model weights which will be downloaded during deployment.\n",
|
||||
"SERVE_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai-restricted/vertex-vision-model-garden-dockers/pytorch-transformers-serve\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "10188266a5cd"
|
||||
},
|
||||
"source": [
|
||||
"### Define common functions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "cac4478ae098"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import base64\n",
|
||||
"import os\n",
|
||||
"from datetime import datetime\n",
|
||||
"from io import BytesIO\n",
|
||||
"\n",
|
||||
"import requests\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from PIL import Image\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def create_job_name(prefix):\n",
|
||||
" user = os.environ.get(\"USER\")\n",
|
||||
" now = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
|
||||
" job_name = f\"{prefix}-{user}-{now}\"\n",
|
||||
" return job_name\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def download_image(url):\n",
|
||||
" response = requests.get(url)\n",
|
||||
" return Image.open(BytesIO(response.content))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def image_to_base64(image, format=\"JPEG\"):\n",
|
||||
" buffer = BytesIO()\n",
|
||||
" image.save(buffer, format=format)\n",
|
||||
" image_str = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n",
|
||||
" return image_str\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def base64_to_image(image_str):\n",
|
||||
" image = Image.open(BytesIO(base64.b64decode(image_str)))\n",
|
||||
" return image\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def image_grid(imgs, rows=2, cols=2):\n",
|
||||
" w, h = imgs[0].size\n",
|
||||
" grid = Image.new(\"RGB\", size=(cols * w, rows * h))\n",
|
||||
" for i, img in enumerate(imgs):\n",
|
||||
" grid.paste(img, box=(i % cols * w, i // cols * h))\n",
|
||||
" return grid\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def deploy_model(model_id, task):\n",
|
||||
" model_name = \"blip-image-captioning\"\n",
|
||||
" endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
|
||||
" serving_env = {\n",
|
||||
" \"MODEL_ID\": model_id,\n",
|
||||
" \"TASK\": task,\n",
|
||||
" }\n",
|
||||
" # If the model_id is a GCS path, use artifact_uri to pass it to serving docker.\n",
|
||||
" artifact_uri = model_id if model_id.startswith(\"gs://\") else None\n",
|
||||
" model = aiplatform.Model.upload(\n",
|
||||
" display_name=model_name,\n",
|
||||
" serving_container_image_uri=SERVE_DOCKER_URI,\n",
|
||||
" serving_container_ports=[7080],\n",
|
||||
" serving_container_predict_route=\"/predictions/transformers_serving\",\n",
|
||||
" serving_container_health_route=\"/ping\",\n",
|
||||
" serving_container_environment_variables=serving_env,\n",
|
||||
" artifact_uri=artifact_uri,\n",
|
||||
" )\n",
|
||||
" model.deploy(\n",
|
||||
" endpoint=endpoint,\n",
|
||||
" machine_type=\"n1-standard-8\",\n",
|
||||
" accelerator_type=\"NVIDIA_TESLA_T4\",\n",
|
||||
" accelerator_count=1,\n",
|
||||
" deploy_request_timeout=1800,\n",
|
||||
" )\n",
|
||||
" return model, endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d2d72ecdb8c9"
|
||||
},
|
||||
"source": [
|
||||
"## Upload and deploy models"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "9448c5f545fa"
|
||||
},
|
||||
"source": [
|
||||
"This section uploads the pre-trained model to Model Registry and deploys it on the Endpoint with 1 T4 GPU.\n",
|
||||
"\n",
|
||||
"The model deployment step will take ~15 minutes to complete.\n",
|
||||
"\n",
|
||||
"Once deployed, you can send images to get descriptions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "19e8aeec969c"
|
||||
},
|
||||
"source": [
|
||||
"### Image captioning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b4b46c28d8b1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model, endpoint = deploy_model(\n",
|
||||
" model_id=\"Salesforce/blip2-opt-2.7b\", task=\"image-to-text\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "12893aa2c5af"
|
||||
},
|
||||
"source": [
|
||||
"NOTE: The model weights will be downloaded after the deployment succeeds. When the model is very large it could add 5~15mins additional time before the endpoint is ready for prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "6be655247cb1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"image = download_image(\"http://images.cocodataset.org/val2017/000000039769.jpg\")\n",
|
||||
"display(image)\n",
|
||||
"\n",
|
||||
"instances = [\n",
|
||||
" {\"image\": image_to_base64(image)},\n",
|
||||
"]\n",
|
||||
"preds = endpoint.predict(instances=instances).predictions\n",
|
||||
"print(preds)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2ccf3714dbe9"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Undeploy model and delete endpoint.\n",
|
||||
"endpoint.delete(force=True)\n",
|
||||
"\n",
|
||||
"# Delete models.\n",
|
||||
"model.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "01e11d85d504"
|
||||
},
|
||||
"source": [
|
||||
"### VQA (Visual-Question-Answering)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "26018d961cf9"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model, endpoint = deploy_model(\n",
|
||||
" model_id=\"Salesforce/blip2-opt-2.7b\", task=\"visual-question-answering\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0ac7f8d945e3"
|
||||
},
|
||||
"source": [
|
||||
"NOTE: The model weights will be downloaded after the deployment succeeds. When the model is very large it could add 5~15mins additional time before the endpoint is ready for prediction."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "f19c342829fd"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"image = download_image(\n",
|
||||
" \"https://media.newyorker.com/cartoons/63dc6847be24a6a76d90eb99/master/w_1160,c_limit/230213_a26611_838.jpg\"\n",
|
||||
")\n",
|
||||
"display(image)\n",
|
||||
"\n",
|
||||
"question = \"Question: What are they doing? Answer:\"\n",
|
||||
"instances = [\n",
|
||||
" {\"image\": image_to_base64(image), \"text\": question},\n",
|
||||
"]\n",
|
||||
"preds = endpoint.predict(instances=instances).predictions\n",
|
||||
"print(question)\n",
|
||||
"print(preds)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "712eb9d0b336"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Undeploy model and delete endpoint.\n",
|
||||
"endpoint.delete(force=True)\n",
|
||||
"\n",
|
||||
"# Delete models.\n",
|
||||
"model.delete()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "model_garden_pytorch_blip2.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@@ -273,11 +273,7 @@
|
||||
"\n",
|
||||
"def draw_image_with_boxes(image, boxes):\n",
|
||||
" fig, ax = plt.subplots()\n",
|
||||
" plt.axis(\"off\")\n",
|
||||
" ax.imshow(image)\n",
|
||||
" if len(boxes) == 0:\n",
|
||||
" return\n",
|
||||
" boxes = boxes[\"boxes\"]\n",
|
||||
" for box in boxes:\n",
|
||||
" x, y = box[\"xmin\"], box[\"ymin\"]\n",
|
||||
" width, height = box[\"xmax\"] - x, box[\"ymax\"] - y\n",
|
||||
@@ -285,6 +281,7 @@
|
||||
" (x, y), width, height, linewidth=2, edgecolor=\"yellow\", facecolor=\"none\"\n",
|
||||
" )\n",
|
||||
" ax.add_patch(rect)\n",
|
||||
" plt.axis(\"off\")\n",
|
||||
" plt.show()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -365,7 +362,7 @@
|
||||
" {\"image\": image_to_base64(image), \"text\": \"cat\"},\n",
|
||||
"]\n",
|
||||
"preds = endpoint.predict(instances=instances).predictions\n",
|
||||
"draw_image_with_boxes(image, preds[0])\n",
|
||||
"draw_image_with_boxes(image, preds[0][\"boxes\"])\n",
|
||||
"print(preds)"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -242,7 +242,22 @@
|
||||
"source": [
|
||||
"## Notes about service account and permission\n",
|
||||
"\n",
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts have the required roles listed in the [Service accounts for Tabular Workflow for End-to-End AutoML documentation](https://cloud.google.com/vertex-ai/docs/tabular-data/tabular-workflows/service-accounts#e2e-automl)."
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n",
|
||||
"\n",
|
||||
"|Service account email|Description|Roles|\n",
|
||||
"|---|---|---|\n",
|
||||
"|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Developer, Dataflow Worker, Storage Admin, BigQuery Data Editor, Vertex AI User, Service Account User|\n",
|
||||
"|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"1. Goto https://console.cloud.google.com/iam-admin/iam.\n",
|
||||
"2. Check the \"Include Google-provided role grants\" checkbox.\n",
|
||||
"3. Find the above emails.\n",
|
||||
"4. Grant the corresponding roles.\n",
|
||||
"\n",
|
||||
"### Using data source from a different project\n",
|
||||
"- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n",
|
||||
"- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+1
-1
@@ -1577,7 +1577,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "sdk-custom-scikit-learn-prebuilt-container.ipynb",
|
||||
"name": "UJ10 Vertex SDK Custom Scikit-Learn with pre-built training container.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -201,8 +201,8 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade {USER_FLAG} google-cloud-aiplatform \\\n",
|
||||
" google-cloud-storage \\\n",
|
||||
" 'kfp<2' \\\n",
|
||||
" 'google-cloud-pipeline-components<2' -q"
|
||||
" kfp \\\n",
|
||||
" google-cloud-pipeline-components -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -165,11 +165,11 @@
|
||||
"# Install the packages\n",
|
||||
"USER=''\n",
|
||||
"! pip3 install {USER} --upgrade google-cloud-aiplatform \\\n",
|
||||
" 'google-cloud-pipeline-components<2'\n",
|
||||
" google-cloud-pipeline-components\n",
|
||||
"! pip3 install {USER} tensorflow==2.5 \\\n",
|
||||
" tensorflow_hub\n",
|
||||
" \n",
|
||||
"! pip3 install {USER} --upgrade 'kfp<2'"
|
||||
"! pip3 install {USER} --upgrade kfp"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -175,7 +175,7 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"! pip3 install -U google-cloud-storage {USER_FLAG} -q\n",
|
||||
"! pip3 install {USER_FLAG} 'kfp<2' 'google-cloud-pipeline-components<2' --upgrade -q"
|
||||
"! pip3 install {USER_FLAG} kfp google-cloud-pipeline-components --upgrade -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -197,10 +197,10 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"! pip3 install -U google-cloud-storage {USER_FLAG} -q\n",
|
||||
"! pip3 install {USER_FLAG} 'kfp<2' 'google-cloud-pipeline-components<2' --upgrade -q\n",
|
||||
"! pip3 install {USER_FLAG} kfp google-cloud-pipeline-components --upgrade -q\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade --force-reinstall $USER_FLAG tensorflow 'kfp<2' google-cloud-aiplatform google-cloud-storage 'google-cloud-pipeline-components<2' -q"
|
||||
"! pip3 install --upgrade --force-reinstall $USER_FLAG tensorflow kfp google-cloud-aiplatform google-cloud-storage google-cloud-pipeline-components -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -224,8 +224,8 @@
|
||||
" google-cloud-bigquery \\\n",
|
||||
" pandas \\\n",
|
||||
" pyarrow \\\n",
|
||||
" 'kfp<2' \\\n",
|
||||
" 'google-cloud-pipeline-components<2' {USER_FLAG} -q \n",
|
||||
" kfp \\\n",
|
||||
" google-cloud-pipeline-components {USER_FLAG} -q \n",
|
||||
"! pip3 install db-dtypes {USER_FLAG} -q"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -149,8 +149,8 @@
|
||||
"! pip3 install -U tensorflow-transform==1.2 -q\n",
|
||||
"! pip3 install -U tensorflow-io==0.18 -q\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] -q\n",
|
||||
"! pip3 install --upgrade 'google-cloud-pipeline-components<2' -q\n",
|
||||
"! pip3 install --upgrade 'kfp<2' -q"
|
||||
"! pip3 install --upgrade google-cloud-pipeline-components -q\n",
|
||||
"! pip3 install --upgrade kfp -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -138,8 +138,8 @@
|
||||
"import os\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" 'google-cloud-pipeline-components<2' --quiet\n",
|
||||
"! pip3 install --upgrade 'kfp<2' --quiet\n",
|
||||
" google-cloud-pipeline-components --quiet\n",
|
||||
"! pip3 install --upgrade kfp --quiet\n",
|
||||
"! pip3 install --upgrade tensorflow==2.7 --quiet"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -193,7 +193,7 @@
|
||||
"if IS_WORKBENCH_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade --quiet {USER_FLAG} google-cloud-aiplatform 'kfp<2' 'google-cloud-pipeline-components<2' google-cloud-storage"
|
||||
"! pip3 install --upgrade --quiet {USER_FLAG} google-cloud-aiplatform kfp google-cloud-pipeline-components google-cloud-storage"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -198,7 +198,7 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" google-cloud-storage \\\n",
|
||||
" $USER 'kfp<2' 'google-cloud-pipeline-components<2' -q "
|
||||
" $USER kfp google-cloud-pipeline-components -q "
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -196,7 +196,7 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"! pip3 install -U google-cloud-storage $USER_FLAG -q\n",
|
||||
"! pip3 install $USER 'kfp<2' 'google-cloud-pipeline-components<2' --upgrade -q"
|
||||
"! pip3 install $USER kfp google-cloud-pipeline-components --upgrade -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+2
-2
@@ -198,8 +198,8 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade {USER_FLAG} -q google-cloud-aiplatform \\\n",
|
||||
" google-cloud-storage {USER_FLAG} \\\n",
|
||||
" 'kfp<2' \\\n",
|
||||
" 'google-cloud-pipeline-components<2'"
|
||||
" kfp \\\n",
|
||||
" google-cloud-pipeline-components"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+2
-2
@@ -199,8 +199,8 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" 'google-cloud-pipeline-components<2' \\\n",
|
||||
" 'kfp<2' $USER_FLAG -q"
|
||||
" google-cloud-pipeline-components \\\n",
|
||||
" kfp $USER_FLAG -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -203,7 +203,7 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"! pip3 install -U google-cloud-storage {USER_FLAG} -q\n",
|
||||
"! pip3 install {USER_FLAG} 'kfp<2' 'google-cloud-pipeline-components<2' --upgrade -q"
|
||||
"! pip3 install {USER_FLAG} kfp google-cloud-pipeline-components --upgrade -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -201,7 +201,7 @@
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
|
||||
"! pip3 install -U google-cloud-storage {USER_FLAG} -q\n",
|
||||
"! pip3 install {USER_FLAG} 'kfp<2' 'google-cloud-pipeline-components<2' --upgrade -q\n",
|
||||
"! pip3 install {USER_FLAG} kfp google-cloud-pipeline-components --upgrade -q\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! pip3 install --upgrade matplotlib $USER_FLAG -q"
|
||||
|
||||
@@ -168,11 +168,11 @@
|
||||
"# Install the packages\n",
|
||||
"USER=''\n",
|
||||
"! pip3 install {USER} --upgrade google-cloud-aiplatform \\\n",
|
||||
" 'google-cloud-pipeline-components<2'\n",
|
||||
" google-cloud-pipeline-components\n",
|
||||
"! pip3 install {USER} tensorflow==2.5 \\\n",
|
||||
" tensorflow_hub\n",
|
||||
"\n",
|
||||
"! pip3 install {USER} --upgrade 'kfp<2'"
|
||||
"! pip3 install {USER} --upgrade kfp"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -128,8 +128,8 @@
|
||||
"# Install the packages\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
|
||||
" google-cloud-storage \\\n",
|
||||
" 'kfp<2' \\\n",
|
||||
" 'google-cloud-pipeline-components<2'"
|
||||
" kfp \\\n",
|
||||
" google-cloud-pipeline-components"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
-1174
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -268,7 +268,22 @@
|
||||
"source": [
|
||||
"## Notes about service account and permission\n",
|
||||
"\n",
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts have the required roles listed in the [Service accounts for Tabular Workflow for TabNet, and Tabular Workflow for Wide & Deep, and Prophet documentation](https://cloud.google.com/vertex-ai/docs/tabular-data/tabular-workflows/service-accounts#fte-workflow)."
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n",
|
||||
"\n",
|
||||
"|Service account email|Description|Roles|\n",
|
||||
"|---|---|---|\n",
|
||||
"|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Admin, Dataflow Worker, Storage Admin, BigQuery Admin, Vertex AI User|\n",
|
||||
"|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"1. Goto https://console.cloud.google.com/iam-admin/iam.\n",
|
||||
"2. Check the \"Include Google-provided role grants\" checkbox.\n",
|
||||
"3. Find the above emails.\n",
|
||||
"4. Grant the corresponding roles.\n",
|
||||
"\n",
|
||||
"### Using data source from a different project\n",
|
||||
"- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n",
|
||||
"- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -268,7 +268,22 @@
|
||||
"source": [
|
||||
"## Notes about service account and permission\n",
|
||||
"\n",
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts have the required roles listed in the [Service accounts for Tabular Workflow for TabNet, and Tabular Workflow for Wide & Deep, and Prophet documentation](https://cloud.google.com/vertex-ai/docs/tabular-data/tabular-workflows/service-accounts#fte-workflow)."
|
||||
"**By default no configuration is required**, if you run into any permission related issue, please make sure the service accounts above have the required roles:\n",
|
||||
"\n",
|
||||
"|Service account email|Description|Roles|\n",
|
||||
"|---|---|---|\n",
|
||||
"|PROJECT_NUMBER-compute@developer.gserviceaccount.com|Compute Engine default service account|Dataflow Admin, Dataflow Worker, Storage Admin, BigQuery Admin, Vertex AI User|\n",
|
||||
"|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"1. Goto https://console.cloud.google.com/iam-admin/iam.\n",
|
||||
"2. Check the \"Include Google-provided role grants\" checkbox.\n",
|
||||
"3. Find the above emails.\n",
|
||||
"4. Grant the corresponding roles.\n",
|
||||
"\n",
|
||||
"### Using data source from a different project\n",
|
||||
"- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n",
|
||||
"- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user