refactor(egen): Refactored the code as per template guidelines (#3015)

* refactored code according to template notebook

* refactored code according to template notebook
This commit is contained in:
Rohith Alla
2024-06-03 23:46:29 +00:00
committed by GitHub
parent 78d2d46f80
commit c240cf6a34
@@ -26,22 +26,45 @@
{
"cell_type": "markdown",
"metadata": {
"id": "vBSXW09S05My"
"id": "103lT16d4tLE"
},
"source": [
"# Vertex AI Feature Store Feature View Service Agents Tutorial"
"# Vertex AI Feature Store Feature View Service Agents Tutorial\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Ffeature_store%2Fvertex_ai_feature_store_feature_view_service_agents.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "103lT16d4tLE"
"id": "becda595b9b8"
},
"source": [
"<table align=\"left\">\n",
"<a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\\\"><img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\">Run in Colab\n",
"<a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\\\"><img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\\\" alt=\"GitHub logo\">View on GitHub\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/feature_store/vertex_ai_feature_store_feature_view_service_agents.ipynb\\\"><img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\""
"## Overview\n",
"In this tutorial, you learn how to enable feature view Service Agents and grant each feature view access to the specific source data that is used.\n",
"\n",
"Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/latest/overview)."
]
},
{
@@ -50,12 +73,7 @@
"id": "E5wnTSHgyuFr"
},
"source": [
"## Overview\n",
"In this tutorial, you learn how to enable feature view Service Agents and grant each feature view access to the specific source data that is used.\n",
"\n",
"Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/latest/overview).\n",
"\n",
"## Objective\n",
"### Objective\n",
"In this tutorial, you learn how to use a dedicated service agent for a feature view in Vertex AI Feature Store. Using a dedicated service agent for a feature view, you grant access to extract feature data from BigQuery in an end-to-end feature serving workflow.\n",
"\n",
"This tutorial uses the following Google Cloud services and resources:\n",
@@ -69,7 +87,7 @@
"## Note\n",
"This is a Preview release. By using the feature, you acknowledge that you're aware of the open issues and that this preview is provided “as is” under the pre-GA terms of service.\n",
"\n",
"## Costs\n",
"### Costs\n",
"This tutorial uses billable components of Google Cloud:\n",
"* `Vertex AI`\n",
"* `BigQuery`\n",
@@ -83,9 +101,16 @@
"id": "-HGNMzgHFy34"
},
"source": [
"## Installation\n",
"\n",
"Install the following packages required to execute this notebook."
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-HGNMzgHFy34"
},
"source": [
"### Install Vertex AI SDK and other required packages"
]
},
{
@@ -108,7 +133,9 @@
"id": "UGJ_-HL1_P7w"
},
"source": [
"## Colab only: Uncomment the following cell to restart the kernel."
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
@@ -119,32 +146,53 @@
},
"outputs": [],
"source": [
"# # Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TZP9eoNCAAi4"
"id": "e767418763cd"
},
"source": [
"## Before you begin\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "870814a62e87"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"### Set up your Google Cloud project\n",
"Authenticate your environment on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c97be6a73155"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
" from google.colab import auth\n",
"\n",
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
" auth.authenticate_user()"
]
},
{
@@ -153,12 +201,9 @@
"id": "7zkyTnWjACyk"
},
"source": [
"#### Set your project ID\n",
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -170,116 +215,12 @@
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vbc1nfqhxvrS"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations). Note that the new API is currently only available in the following regions:\n",
"* `us-central1`\n",
"* `asia-southeast1`\n",
"* `europe-west1`\n",
"* `europe-west2`\n",
"* `europe-west3`\n",
"* `europe-west4`\n",
"* `us-east1`\n",
"* `us-east4`\n",
"* `us-west1`\n",
"* `us-west2`\n",
"* `us-west3`\n",
"* `europe-west8`\n",
"* `asia-northeast2`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7Ywl-59Sxy4h"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1Z_1vxrS4quc"
},
"source": [
"### Authenticate your Google Cloud account\n",
"from google.cloud import aiplatform\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xYej1E-9Acgz"
},
"source": [
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_ii5t0qTAhDu"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JQ7TXbsUAgF7"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M0VJ-rsK4tu8"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gyUOLvGJAqYo"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PsAbCyVSAs62"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
@@ -299,7 +240,6 @@
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"from google.cloud.aiplatform_v1beta1 import (\n",
" FeatureOnlineStoreAdminServiceClient, FeatureOnlineStoreServiceClient,\n",
" FeatureRegistryServiceClient)\n",
@@ -328,9 +268,9 @@
"id": "S4nZTv4sx_wS"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"### Initialize AI Platform Feature Store Clients\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project."
"Initialize the AI platform Feature Store Clients for your project."
]
},
{
@@ -341,9 +281,7 @@
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)\n",
"\n",
"API_ENDPOINT = f\"{REGION}-aiplatform.googleapis.com\"\n",
"API_ENDPOINT = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
"\n",
"admin_client = FeatureOnlineStoreAdminServiceClient(\n",
" client_options={\"api_endpoint\": API_ENDPOINT}\n",
@@ -373,10 +311,10 @@
},
"outputs": [],
"source": [
"DATASET_ID = \"test_data\"+\"_\"+REGION.replace('-', '_') # @param {type:\"string\"}\n",
"DATASET_ID = \"test_data\"+\"_\"+LOCATION.replace('-', '_') # @param {type:\"string\"}\n",
"TABLE_ID = \"tableA\" # @param {type:\"string\"}\n",
"\n",
"!bq mk --dataset_id={DATASET_ID} --location={REGION}\n",
"!bq mk --dataset_id={DATASET_ID} --location={LOCATION}\n",
"!bq query --nouse_legacy_sql \\\n",
"\"CREATE TABLE {DATASET_ID}.{TABLE_ID} AS (\" \\\n",
"\"SELECT * FROM UNNEST(ARRAY<STRUCT<entity_id STRING, feature_timestamp TIMESTAMP, feature1 INT64, feature2 INT64>>[\" \\\n",
@@ -391,7 +329,7 @@
"id": "85tGwbXKkKCo"
},
"source": [
"## Create a Feature Group\n",
"### Create a Feature Group\n",
"First, create a feature group."
]
},
@@ -429,7 +367,7 @@
"\n",
"create_group_lro = registry_client.create_feature_group(\n",
" feature_registry_service_pb2.CreateFeatureGroupRequest(\n",
" parent=f\"projects/{PROJECT_ID}/locations/{REGION}\",\n",
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}\",\n",
" feature_group_id=FEATURE_GROUP_ID,\n",
" feature_group=feature_group_config,\n",
" )\n",
@@ -442,7 +380,7 @@
" create_feature_lros.append(\n",
" registry_client.create_feature(\n",
" featurestore_service_pb2.CreateFeatureRequest(\n",
" parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureGroups/{FEATURE_GROUP_ID}\",\n",
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureGroups/{FEATURE_GROUP_ID}\",\n",
" feature_id=id,\n",
" feature=feature_pb2.Feature(),\n",
" )\n",
@@ -471,7 +409,7 @@
"source": [
"# Verify feature group is created.\n",
"registry_client.get_feature_group(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
")"
]
},
@@ -494,7 +432,7 @@
"source": [
"# Use list to verify the features are created.\n",
"registry_client.list_features(\n",
" parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
")"
]
},
@@ -538,7 +476,7 @@
"\n",
"create_store_lro = admin_client.create_feature_online_store(\n",
" feature_online_store_admin_service_pb2.CreateFeatureOnlineStoreRequest(\n",
" parent=f\"projects/{PROJECT_ID}/locations/{REGION}\",\n",
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}\",\n",
" feature_online_store_id=FEATURE_ONLINE_STORE_ID,\n",
" feature_online_store=online_store_config,\n",
" )\n",
@@ -568,7 +506,7 @@
"source": [
"# Use list to verify the store is created.\n",
"admin_client.get_feature_online_store(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n",
")"
]
},
@@ -616,7 +554,7 @@
"sync_config = feature_view_pb2.FeatureView.SyncConfig(cron=CRON_SCHEDULE)\n",
"\n",
"create_view_lro = admin_client.create_feature_view(\n",
" parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\",\n",
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\",\n",
" feature_view_id=\"test_fv\",\n",
" feature_view=feature_view_pb2.FeatureView(\n",
" feature_registry_source=feature_registry_source,\n",
@@ -656,7 +594,7 @@
"source": [
"# Use list to verify the store is created.\n",
"admin_client.get_feature_view(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
")"
]
},
@@ -687,7 +625,7 @@
"\n",
"# Get feature view\n",
"feature_view = admin_client.get_feature_view(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
")\n",
"SERVICE_ACCOUNT = feature_view.service_account_email"
]
@@ -747,7 +685,7 @@
"outputs": [],
"source": [
"sync_response = admin_client.sync_feature_view(\n",
" feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
" feature_view=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
")\n",
"\n",
"# Wait for sync completion\n",
@@ -827,7 +765,7 @@
"source": [
"data_client.fetch_feature_values(\n",
" request=feature_online_store_service_pb2.FetchFeatureValuesRequest(\n",
" feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\",\n",
" feature_view=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\",\n",
" data_key=feature_online_store_service_pb2.FeatureViewDataKey(key=\"test\"),\n",
" )\n",
")"
@@ -839,16 +777,12 @@
"id": "6J8bU-GmrV5i"
},
"source": [
"# Clean up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sXNEYNZiBcpj"
},
"source": [
"To clean up all the Google Cloud resources used in this project, delete the individual resources you created in this tutorial."
"## Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the cluster you created in this tutorial."
]
},
{
@@ -861,23 +795,23 @@
"source": [
"# Delete feature view\n",
"admin_client.delete_feature_view(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n",
")\n",
"\n",
"# Delete online store\n",
"admin_client.delete_feature_online_store(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n",
")\n",
"\n",
"# Delete features\n",
"for feature_id in FEATURE_IDS:\n",
" registry_client.delete_feature(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureGroups/{FEATURE_GROUP_ID}/features/{feature_id}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureGroups/{FEATURE_GROUP_ID}/features/{feature_id}\"\n",
" )\n",
"\n",
"# Delete feature group\n",
"registry_client.delete_feature_group(\n",
" name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/featureGroups/{FEATURE_GROUP_ID}\"\n",
")\n",
"\n",
"# Delete test data\n",