xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updates sdk_automl_video_action_recognition_batch.ipynb (#1233)

Browse Source 
File came in regression log. Unable to download executed notebook. When ran in local, file executed successfully.
**Changes made**:

Update notebook according to template

- [X] Use the [notebook template](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb) as a starting point.
- [X] Follow the style and grammar rules outlined in the above notebook template.
- [X] Verify the notebook runs successfully in Colab since the automated tests cannot guarantee this even when it passes.
- [X] Passes all the required automated checks. You can locally test for formatting and linting with these [instructions](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/CONTRIBUTING.md#code-quality-checks).
- [ ] You have consulted with a tech writer to see if tech writer review is necessary. If so, the notebook has been reviewed by a tech writer, and they have approved it.
- [X] This notebook has been added to the [CODEOWNERS](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/CODEOWNERS) file under the `Official Notebooks` section, pointing to the author or the author's team.
- [X] The Jupyter notebook cleans up any artifacts it has created (datasets, ML models, endpoints, etc) so as not to eat up unnecessary resources.

<br>
This commit is contained in:
sudarshan-SpringML
2022-11-30 00:04:13 +00:00
committed by GitHub
parent 1aa94377f5
commit 32ac97982c
1 changed files with 145 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb
+145 -142
View File
@@ -29,9 +29,10 @@
"id": "title"
},
"source": [
"# Vertex SDK: AutoML training video action recognition model for batch prediction\n",
"# Vertex AI SDK: AutoML training video action recognition model for batch prediction\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
@@ -44,10 +45,11 @@
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/automl/sdk_automl_video_action_recognition_batch.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" </td> \n",
"</table>\n",
"<br/><br/><br/>"
]
@@ -74,9 +76,16 @@
"\n",
"In this tutorial, you learn how to create an AutoML video action recognition model from a Python script, and then do a batch prediction using the Vertex AI SDK. You can alternatively create and deploy models using the `gcloud` command-line tool or online using the Cloud Console.\n",
"\n",
"This tutorial uses the following Google Cloud ML services and resources:\n",
"\n",
"- Vertex AI Dataset\n",
"- Vertex AI Model\n",
"- Vertex AI Batch Prediction\n",
"\n",
"\n",
"The steps performed include:\n",
"\n",
"- Create a Vertex `Dataset` resource.\n",
"- Create a `Vertex AI Dataset` resource.\n",
"- Train the model.\n",
"- View the model evaluation.\n",
"- Make a batch prediction.\n",
@@ -96,7 +105,7 @@
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset from MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model will predict the start frame where an action of golf swing begins."
"The dataset used for this tutorial is the golf swing recognition portion of the [Human Motion dataset from MIT](http://cbcl.mit.edu/publications/ps/Kuehne_etal_iccv11.pdf). The version of the dataset you use in this tutorial is stored in a public Cloud Storage bucket. The trained model will predict the start frame where an action of golf swing begins."
]
},
{
@@ -127,29 +136,38 @@
"source": [
"### Set up your local development environment\n",
"\n",
"If you are using Colab or Google Cloud Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
"all the requirements to run this notebook.\n",
"\n",
"Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
"You need the following:\n",
"\n",
"- The Cloud Storage SDK\n",
"- Git\n",
"- Python 3\n",
"- virtualenv\n",
"- Jupyter notebook running in a virtual environment with Python 3\n",
"* The Google Cloud SDK\n",
"* Git\n",
"* Python 3\n",
"* virtualenv\n",
"* Jupyter notebook running in a virtual environment with Python 3\n",
"\n",
"The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
"The Google Cloud guide to [Setting up a Python development\n",
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
"for meeting these requirements. The following steps provide a condensed set of\n",
"instructions:\n",
"\n",
"1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
"\n",
"2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
"\n",
"3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"1. [Install\n",
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"\n",
"4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
"command-line in a terminal shell.\n",
"\n",
"5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"\n",
"6. Open this notebook in the Jupyter Notebook Dashboard.\n"
"1. Open this notebook in the Jupyter Notebook Dashboard."
]
},
{
@@ -160,7 +178,7 @@
"source": [
"## Installation\n",
"\n",
"Install the latest version of Vertex SDK for Python."
"Install the following packages required to execute this notebook. \n"
]
},
{
@@ -173,35 +191,17 @@
"source": [
"import os\n",
"\n",
"# Google Cloud Notebook\n",
"if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"else:\n",
" USER_FLAG = \"\"\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "install_storage"
},
"source": [
"Install the latest GA version of *google-cloud-storage* library as well.\n",
"\n",
"**Note**: You may encounter a PIP dependency error during the installation of the Google Cloud Storage package. This can be ignored as it will not affect the proper running of this script."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_storage"
},
"outputs": [],
"source": [
"! pip3 install -U google-cloud-storage $USER_FLAG"
"! pip3 install --upgrade --quiet {USER_FLAG} google-cloud-aiplatform google-cloud-storage"
]
},
{
@@ -212,7 +212,7 @@
"source": [
"### Restart the kernel\n",
"\n",
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages. The following cell will restart the kernel."
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
]
},
{
@@ -223,6 +223,7 @@
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
@@ -241,26 +242,33 @@
"source": [
"## Before you begin\n",
"\n",
"### GPU runtime\n",
"\n",
"This tutorial does not require a GPU runtime.\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
"1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). \n",
"\n",
"4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
"\n",
"5. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5aee4379e8e5"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
]
},
{
@@ -309,7 +317,7 @@
"#### Region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
@@ -317,7 +325,7 @@
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
@@ -340,9 +348,9 @@
"id": "timestamp"
},
"source": [
"#### Timestamp\n",
"#### UUID\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial.\n"
]
},
{
@@ -353,9 +361,16 @@
},
"outputs": [],
"source": [
"from datetime import datetime\n",
"import random\n",
"import string\n",
"\n",
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
"\n",
"# Generate a uuid of a specifed length(default=8)\n",
"def generate_uuid(length: int = 8) -> str:\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
@@ -366,23 +381,31 @@
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
"authenticated. \n",
"\n",
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"**Click Create service account**.\n",
"2. Click **Create service account**.\n",
"\n",
"In the **Service account name** field, enter a name, and click **Create**.\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"Click Create. A JSON file that contains your key downloads to your local environment.\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
]
},
{
@@ -401,8 +424,11 @@
"import os\n",
"import sys\n",
"\n",
"# If on Google Cloud Notebook, then don't execute this code\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
@@ -425,9 +451,9 @@
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"When you run a Vertex AI pipeline job using the Cloud SDK, your job stores the pipeline artifacts to a Cloud Storage bucket. In this tutorial, you create a Vertex AI Pipeline job that saves the artifacts like evaluation metrics and feature attributes to a Cloud Storage bucket.\n",
"\n",
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
"Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets."
]
},
{
@@ -438,7 +464,8 @@
},
"outputs": [],
"source": [
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
@@ -449,8 +476,9 @@
},
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
" BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
@@ -470,7 +498,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
]
},
{
@@ -490,7 +518,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gsutil ls -al $BUCKET_URI"
]
},
{
@@ -499,9 +527,6 @@
"id": "setup_vars"
},
"source": [
"### Set up variables\n",
"\n",
"Next, set up some variables used throughout the tutorial.\n",
"### Import libraries and define constants"
]
},
@@ -513,7 +538,11 @@
},
"outputs": [],
"source": [
"import google.cloud.aiplatform as aiplatform"
"import json\n",
"import os\n",
"\n",
"import google.cloud.aiplatform as aiplatform\n",
"from google.cloud import storage"
]
},
{
@@ -522,9 +551,9 @@
"id": "init_aip:mbsdk"
},
"source": [
"## Initialize Vertex SDK for Python\n",
"## Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex SDK for Python for your project and corresponding bucket."
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
]
},
{
@@ -535,7 +564,7 @@
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
]
},
{
@@ -546,15 +575,8 @@
"source": [
"# Tutorial\n",
"\n",
"Now you are ready to start creating your own AutoML video action recognition model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "import_file:u_dataset,csv"
},
"source": [
"Now you are ready to start creating your own AutoML video action recognition model.\n",
"\n",
"#### Location of Cloud Storage training data.\n",
"\n",
"Now set the variable `IMPORT_FILES` to the location of the CSV index files in Cloud Storage."
@@ -629,7 +651,7 @@
"outputs": [],
"source": [
"dataset = aiplatform.VideoDataset.create(\n",
" display_name=\"Golf Swings\" + \"_\" + TIMESTAMP,\n",
" display_name=\"Golf Swings\" + \"_\" + UUID,\n",
" gcs_source=IMPORT_FILES,\n",
" import_schema_uri=aiplatform.schema.dataset.ioformat.video.action_recognition,\n",
")\n",
@@ -645,17 +667,19 @@
"source": [
"### Create and run training pipeline\n",
"\n",
"To train an AutoML model, you perform two steps: 1) create a training pipeline, and 2) run the pipeline.\n",
"To train an AutoML model, you perform two steps: \n",
"1. create a training pipeline.\n",
"2. run the pipeline.\n",
"\n",
"#### Create training pipeline\n",
"#### Create the training pipeline\n",
"\n",
"An AutoML training pipeline is created with the `AutoMLVideoTrainingJob` class, with the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `TrainingJob` resource.\n",
"- `prediction_type`: The type task to train the model for.\n",
" - `classification`: A video classification model.\n",
" - `object_tracking`: A video object tracking model.\n",
" - `action_recognition`: A video action recognition model."
" - `classification`: A video classification model.\n",
" - `object_tracking`: A video object tracking model.\n",
" - `action_recognition`: A video action recognition model."
]
},
{
@@ -667,7 +691,7 @@
"outputs": [],
"source": [
"job = aiplatform.AutoMLVideoTrainingJob(\n",
" display_name=\"golf_\" + TIMESTAMP,\n",
" display_name=\"golf_\" + UUID,\n",
" prediction_type=\"action_recognition\",\n",
")\n",
"\n",
@@ -704,7 +728,7 @@
"source": [
"model = job.run(\n",
" dataset=dataset,\n",
" model_display_name=\"golf_\" + TIMESTAMP,\n",
" model_display_name=\"golf_\" + UUID,\n",
" training_fraction_split=0.8,\n",
" test_fraction_split=0.2,\n",
")"
@@ -717,9 +741,7 @@
},
"source": [
"## Review model evaluation scores\n",
"After your model has finished training, you can review the evaluation scores for it.\n",
"\n",
"First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project."
"After your model has finished training, you can review the evaluation scores for it.\n"
]
},
{
@@ -730,18 +752,9 @@
},
"outputs": [],
"source": [
"# Get model resource ID\n",
"models = aiplatform.Model.list(filter=\"display_name=golf_\" + TIMESTAMP)\n",
"# Get evaluations\n",
"model_evaluations = model.list_model_evaluations()\n",
"\n",
"# Get a reference to the Model Service client\n",
"client_options = {\"api_endpoint\": f\"{REGION}-aiplatform.googleapis.com\"}\n",
"model_service_client = aiplatform.gapic.ModelServiceClient(\n",
" client_options=client_options\n",
")\n",
"\n",
"model_evaluations = model_service_client.list_model_evaluations(\n",
" parent=models[0].resource_name\n",
")\n",
"model_evaluation = list(model_evaluations)[0]\n",
"print(model_evaluation)"
]
@@ -754,18 +767,11 @@
"source": [
"## Send a batch prediction request\n",
"\n",
"Send a batch prediction to your deployed model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "get_test_items:batch_prediction"
},
"source": [
"Send a batch prediction request to your registered model.\n",
"\n",
"### Get test item(s)\n",
"\n",
"Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model as we just want to demonstrate how to make a prediction."
"Now send a batch prediction request to your Vertex AI model. You use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model as we just want to demonstrate how to make a prediction."
]
},
{
@@ -799,7 +805,7 @@
"source": [
"### Make a batch input file\n",
"\n",
"Now make a batch input file, which you store in your local Cloud Storage bucket. The batch input file can be either CSV or JSONL. You will use JSONL in this tutorial. For JSONL file, you make one dictionary entry per line for each video. The dictionary contains the key/value pairs:\n",
"Now make a batch input file, which you store in your local Cloud Storage bucket. The batch input file can be either CSV or JSONL. You use JSONL in this tutorial. For JSONL file, you make one dictionary entry per line for each video. The dictionary contains the key/value pairs:\n",
"\n",
"- `content`: The Cloud Storage path to the video.\n",
"- `mimeType`: The content type. In our example, it is a `avi` file.\n",
@@ -815,12 +821,8 @@
},
"outputs": [],
"source": [
"import json\n",
"\n",
"from google.cloud import storage\n",
"\n",
"test_filename = \"test.jsonl\"\n",
"gcs_input_uri = BUCKET_NAME + \"/\" + test_filename\n",
"gcs_input_uri = BUCKET_URI + \"/\" + test_filename\n",
"\n",
"# Configure the test-data\n",
"data_1 = {\n",
@@ -837,7 +839,7 @@
"}\n",
"\n",
"# Upload the test-data to Cloud storage bucket\n",
"bucket = storage.Client(project=PROJECT_ID).bucket(BUCKET_NAME.replace(\"gs://\", \"\"))\n",
"bucket = storage.Client(project=PROJECT_ID).bucket(BUCKET_URI.replace(\"gs://\", \"\"))\n",
"blob = bucket.blob(blob_name=test_filename)\n",
"data = json.dumps(data_1) + \"\\n\" + json.dumps(data_2) + \"\\n\"\n",
"blob.upload_from_string(data)\n",
@@ -855,7 +857,7 @@
"source": [
"### Make the batch prediction request\n",
"\n",
"Now that your Model resource is trained, you can make a batch prediction by invoking the batch_predict() method, with the following parameters:\n",
"Now that your Vertex AI Model resource is trained, you can make a batch prediction by invoking the batch_predict() method, with the following parameters:\n",
"\n",
"- `job_display_name`: The human readable name for the batch prediction job.\n",
"- `gcs_source`: A list of one or more batch request input files.\n",
@@ -872,9 +874,9 @@
"outputs": [],
"source": [
"batch_predict_job = model.batch_predict(\n",
" job_display_name=\"golf_\" + TIMESTAMP,\n",
" job_display_name=\"golf_\" + UUID,\n",
" gcs_source=gcs_input_uri,\n",
" gcs_destination_prefix=BUCKET_NAME,\n",
" gcs_destination_prefix=BUCKET_URI,\n",
" sync=False,\n",
")\n",
"\n",
@@ -941,7 +943,7 @@
"\n",
"for prediction_result in prediction_results:\n",
" gfile_name = f\"gs://{bp_iter_outputs.bucket.name}/{prediction_result}\".replace(\n",
" BUCKET_NAME + \"/\", \"\"\n",
" BUCKET_URI + \"/\", \"\"\n",
" )\n",
" data = bucket.get_blob(gfile_name).download_as_string()\n",
" data = json.loads(data)\n",
@@ -988,9 +990,10 @@
"# Delete the batch prediction job using the Vertex batch prediction object\n",
"batch_predict_job.delete()\n",
"\n",
"# Delete the Cloud Storage bucket\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_NAME"
"# Delete Cloud Storage objects\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil -m rm -r $BUCKET_URI"
]
}
],