From 7d406848ea5fbb540c490f35c3d7abd7e42e4890 Mon Sep 17 00:00:00 2001 From: Eric Dong Date: Tue, 3 Oct 2023 21:20:44 -0400 Subject: [PATCH] feat: feature store 2.0 tutorial (goku) (#2281) * feat: feature store 2.0 tutorial (goku) * Complying with the template * Complying with the template 2 * clarify the notebook description * clarify the notebook description 2 * address review comments and rename file * add known issue and change install package * wait for the sync job to complete, and fix resource conflicts * address review comments * address review comments 2 * add a delay for the endpoint to start properly --- ...val_bigquery_data_with_feature_store.ipynb | 1193 +++++++++++++++++ 1 file changed, 1193 insertions(+) create mode 100644 notebooks/official/feature_store/online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb diff --git a/notebooks/official/feature_store/online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb b/notebooks/official/feature_store/online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb new file mode 100644 index 000000000..d7528b3e1 --- /dev/null +++ b/notebooks/official/feature_store/online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb @@ -0,0 +1,1193 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8EcdxqUnftBL" + }, + "outputs": [], + "source": [ + "# Copyright 2023 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rbTKH4_6f1ux" + }, + "source": [ + "## Online feature serving and vector retrieval BigQuery data with Vertex AI Feature Store\n", + "\n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + " \n", + " \n", + " \"Vertex\n", + " Open in Vertex AI Workbench\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WWNCLbZZ6MLi" + }, + "source": [ + "## Overview\n", + "\n", + "This tutorial demonstrates how to use `Vertex AI Feature Store` for online serving and vector retrieval of feature values in `BigQuery`.\n", + "\n", + "Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/overview)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CBeo3dIqJVDd" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you learn how to create and use an online feature store instance to host and serve data in `BigQuery` with `Vertex AI Feature Store` in an end to end workflow of features serving and vector retrieval user journey.\n", + "\n", + "This tutorial uses the following Google Cloud ML services and resources:\n", + "\n", + "- `Vertex AI Feature Store`\n", + "\n", + "The steps performed include:\n", + "\n", + "- Provision an online feature store instance to host and serve data.\n", + "- Create an online feature store instance to serve a `BigQuery` table.\n", + "- Use the online server to search nearest neighbors." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "H1t0xA5XB_3n" + }, + "source": [ + "### Known Issues\n", + "This public Preview release has the following known issues:\n", + "\n", + "- While running, the data sync pipeline creates temporary tables and datasets in your Google Cloud project. These tables and datasets can be viewed by all users who have `bigquery.tables.getData` permission for the project.\n", + "\n", + "The development team is actively working on resolutions for these issues. By using the feature, you acknowledge that you're aware of the open issues and that this preview is provided “as is” under the pre-GA terms of service." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SbCa7Pcpqgaz" + }, + "source": [ + "### Dataset\n", + "\n", + "This tutorial uses the [Google Patents Public Data](https://pantheon.corp.google.com/marketplace/product/google_patents_public_datasets/google-patents-public-data) dataset from the `BigQuery` public datasets.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nrggkyCUrhZM" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* `Vertex AI`\n", + "* `BigQuery`\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and\n", + "[BigQuery pricing](https://cloud.google.com/bigquery/pricing)\n", + "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yTJiDCrYsOmT" + }, + "source": [ + "## Installation\n", + "\n", + "Install the following packages required to execute this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GAqbE5Z2sTVM" + }, + "outputs": [], + "source": [ + "# Install the packages\n", + "! pip3 install --upgrade --quiet google-cloud-aiplatform\\\n", + " google-cloud-bigquery\\\n", + " db-dtypes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "np60_uuCs7X5" + }, + "source": [ + "### Colab only: Uncomment the following cell to restart the kernel." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_u0aEgaSs-3v" + }, + "outputs": [], + "source": [ + "# # Automatically restart kernel after installs so that your environment can access the new packages\n", + "# import IPython\n", + "\n", + "# app = IPython.Application.instance()\n", + "# app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mdqw6ADTtJRI" + }, + "source": [ + "## Before you begin\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", + "\n", + "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bFANidV0tPbo" + }, + "source": [ + "#### Set your project ID\n", + "\n", + "**If you don't know your project ID**, try the following:\n", + "* Run `gcloud config list`.\n", + "* Run `gcloud projects list`.\n", + "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "t5cutPRQtQ7m" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "\n", + "# Set the project id\n", + "! gcloud config set project {PROJECT_ID}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lfY8yWnbtZ0K" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations).Note that the new API is currently only available in the following regions:\n", + "* `us-centra1`\n", + "* `us-east1`\n", + "* `us-west1`\n", + "* `europe-west4`\n", + "* `asia-southeast1`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "G6iMMALZthFM" + }, + "outputs": [], + "source": [ + "REGION = \"us-east1\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ni5jx6RGtzG3" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UzsMphY2t4-v" + }, + "source": [ + "**1. Vertex AI Workbench**\n", + "* Do nothing as you are already authenticated." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R_OnJm_Yt8bw" + }, + "source": [ + "**2. Local JupyterLab instance, uncomment and run:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "x4ybIfzhuAOc" + }, + "outputs": [], + "source": [ + "# ! gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gmnRqX6BuBnx" + }, + "source": [ + "**3. Colab, uncomment and run:**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "uN9JoC1buE9P" + }, + "outputs": [], + "source": [ + "# from google.colab import auth\n", + "# auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YNAvMVJjuH5b" + }, + "source": [ + "**4. Service account or other**\n", + "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OMCl0avIusKl" + }, + "source": [ + "### Import libraries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ifbIQuN1uz2r" + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform, bigquery\n", + "from google.cloud.aiplatform_v1beta1 import (\n", + " FeatureOnlineStoreAdminServiceClient, FeatureOnlineStoreServiceClient)\n", + "from google.cloud.aiplatform_v1beta1.types import NearestNeighborQuery\n", + "from google.cloud.aiplatform_v1beta1.types import \\\n", + " feature_online_store as feature_online_store_pb2\n", + "from google.cloud.aiplatform_v1beta1.types import \\\n", + " feature_online_store_admin_service as \\\n", + " feature_online_store_admin_service_pb2\n", + "from google.cloud.aiplatform_v1beta1.types import \\\n", + " feature_online_store_service as feature_online_store_service_pb2\n", + "from google.cloud.aiplatform_v1beta1.types import \\\n", + " feature_view as feature_view_pb2" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6ooJNSOvu6Q5" + }, + "source": [ + "### Initialize Vertex AI SDK for Python\n", + "\n", + "Initialize the Vertex AI SDK for Python for your project." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mLTm3pquu9ar" + }, + "outputs": [], + "source": [ + "aiplatform.init(project=PROJECT_ID, location=REGION)\n", + "\n", + "API_ENDPOINT = f\"{REGION}-aiplatform.googleapis.com\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "szHWlytYTA0I" + }, + "source": [ + "## Set up data source in BigQuery\n", + "\n", + "### Requirements\n", + "The data source has to be a BigQuery table or a BigQuery view, with the following requirements on columns:\n", + "1. [*Required*] One entity id column, type: string\n", + "2. [*Required*] One embedding column, type: double array\n", + "3. [*Optional*] One or more filtering columns, type: string or string array\n", + "4. [*Optional*] One crowding column, type: integer. Crowding ensures that results are diverse by returning at most k' < k neighbors with any single crowding attribute out of k total neighbors" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "E2i4ZmnrT1lf" + }, + "source": [ + "### Test data source\n", + "\n", + "Select a subset and exclude the repeated records type of columns that are not compatible with Feature Store from the `patents-public-data.google_patents_research.publications_202304` table:\n", + "\n", + "Create a small dataset (<=100MB) for demo purposes, you can use the full dataset if needed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IYD5pBRL2fvX" + }, + "outputs": [], + "source": [ + "FEATURE_EXTRACT_QUERY_FULL = \"\"\"\n", + "SELECT publication_number, embedding_v1 as embedding, url, country, publication_description,\n", + "cpc_low, cpc_inventive_low, top_terms, title, CAST(title_translated as INT) as title_translated,\n", + "abstract, CAST(abstract_translated as INT) as abstract_translated,\n", + "cited_by[safe_offset(0)].filing_date as cited_by_filing_date,\n", + "similar[safe_offset(0)].filing_date as similar_filing_date\n", + "FROM `patents-public-data.google_patents_research.publications_202304`\n", + "\"\"\"\n", + "FEATURE_EXTRACT_QUERY_SMALL = f\"{FEATURE_EXTRACT_QUERY_FULL} WHERE cited_by[safe_offset(0)].filing_date is not NULL LIMIT 1000\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZVec1FjhpwP6" + }, + "source": [ + "This data source has some filtering columns (e.g. country) and crowding columns (e.g. cited_by_filing_date). Below is the schema of publications data table we use in this guide:\n", + "\n", + "|Column name | Type | Mode |\n", + "|------------|--------|----------|\n", + "|publication_number\t| STRING |\tNULLABLE\n", + "|embedding |\tFLOAT |\tREPEATED\n", + "|url |\tSTRING\t| NULLABLE\n", + "|country|\tSTRING|\tNULLABLE|\n", + "|publication_description|\tSTRING|\tNULLABLE|\n", + "|cpc_low|\tSTRING|\tREPEATED|\n", + "|cpc_inventive_low|\tSTRING|\tREPEATED|\n", + "|top_terms|\tSTRING|\tREPEATED|\n", + "|title|\tSTRING|\tNULLABLE|\n", + "|title_translated|\tINTEGER|\tNULLABLE|\n", + "|abstract|\tSTRING|\tNULLABLE|\n", + "|abstract_translated|\tINTEGER|\tNULLABLE|\n", + "|cited_by_filing_date|\tINTEGER|\tNULLABLE|\n", + "|similar_filing_date|\tINTEGER|\tNULLABLE|" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tZIsNo40p2Ns" + }, + "source": [ + "View the retrieved data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Qhp2A2wCpwrQ" + }, + "outputs": [], + "source": [ + "bq_client = bigquery.Client(project=PROJECT_ID)\n", + "\n", + "product_data = bq_client.query(FEATURE_EXTRACT_QUERY_SMALL).result().to_dataframe()\n", + "\n", + "print(product_data.shape)\n", + "product_data.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7CspnYk2GyDV" + }, + "source": [ + "### Create BigQuery dataset\n", + "\n", + "Create a BigQuery dataset to hold the BigQuery table for the tutorial. Since the source data for this tutorial is located in the `US` region, the dataset must also be located in the `US` region. If you use your own data and dataset, you can also use that dataset to create the BigQuery table.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0-2pb-NUGzMI" + }, + "outputs": [], + "source": [ + "# First, create a dataset if it does not already exist. The source data for this demo is located in the US region, so the dataset must also be located in the US region.\n", + "\n", + "BQ_DATASET_ID = \"featurestore_demo\" # @param {type:\"string\"}\n", + "dataset = bigquery.Dataset(f\"{PROJECT_ID}.{BQ_DATASET_ID}\")\n", + "dataset.location = \"US\"\n", + "dataset = bq_client.create_dataset(\n", + " dataset, exists_ok=True, timeout=30\n", + ") # Make an API request.\n", + "\n", + "# Confirm dataset created.\n", + "print(f\"Created dataset {dataset}.{BQ_DATASET_ID}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1KSrpN_LG1ia" + }, + "source": [ + "#### Create a BigQuery table" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "JXP2ZaFAG18W" + }, + "outputs": [], + "source": [ + "BQ_TABLE_ID = \"publications_202304_small\" # @param {type:\"string\"}\n", + "BQ_TABLE_ID_FQN = f\"{PROJECT_ID}.{BQ_DATASET_ID}.{BQ_TABLE_ID}\"\n", + "job_config = bigquery.QueryJobConfig(destination=BQ_TABLE_ID_FQN)\n", + "query_job = bq_client.query(FEATURE_EXTRACT_QUERY_SMALL, job_config=job_config)\n", + "\n", + "try:\n", + " query_job.result()\n", + "except Exception as e:\n", + " # Table already exists\n", + " print(\"Error: \", e.message)\n", + "\n", + "print(f\"Created table: {BQ_TABLE_ID_FQN}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nw5tnym8G3wX" + }, + "outputs": [], + "source": [ + "DATA_SOURCE = f\"bq://{BQ_TABLE_ID_FQN}\"\n", + "print(f\"Data source is: {DATA_SOURCE}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AYzwb095DJTl" + }, + "source": [ + "## Set up and start online serving\n", + "\n", + "To serve embedding data in feature store, you need to do the following:\n", + "\n", + "1. Create an online store cluster to host the data.\n", + "2. Define the data (FeatureView) to be served by the newly-created instance." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KMdXOJB7EM47" + }, + "source": [ + "### Initialize Admin Service Client\n", + "\n", + "Load the Feature Store SDK." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KNJF37XGEOi6" + }, + "outputs": [], + "source": [ + "admin_client = FeatureOnlineStoreAdminServiceClient(\n", + " client_options={\"api_endpoint\": API_ENDPOINT}\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "43CSdWFTTn-_" + }, + "source": [ + "### Create Feature Online Store\n", + "\n", + "Create a feature online store with embedding management enabled." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "uoCExlfzfjNE" + }, + "outputs": [], + "source": [ + "FEATURE_ONLINE_STORE_ID = \"my_feature_online_store_unique\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "s4cS3i3zfld6" + }, + "outputs": [], + "source": [ + "online_store_config = feature_online_store_pb2.FeatureOnlineStore(\n", + " bigtable=feature_online_store_pb2.FeatureOnlineStore.Bigtable(\n", + " auto_scaling=feature_online_store_pb2.FeatureOnlineStore.Bigtable.AutoScaling(\n", + " min_node_count=1, max_node_count=3, cpu_utilization_target=50\n", + " )\n", + " ),\n", + " embedding_management=feature_online_store_pb2.FeatureOnlineStore.EmbeddingManagement(\n", + " enabled=True\n", + " ),\n", + ")\n", + "\n", + "create_store_lro = admin_client.create_feature_online_store(\n", + " feature_online_store_admin_service_pb2.CreateFeatureOnlineStoreRequest(\n", + " parent=f\"projects/{PROJECT_ID}/locations/{REGION}\",\n", + " feature_online_store_id=FEATURE_ONLINE_STORE_ID,\n", + " feature_online_store=online_store_config,\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wzExizycIaHH" + }, + "source": [ + "### Verify online store instance creation\n", + "\n", + "After the long-running operation (LRO) is complete, show the result.\n", + "\n", + "> **Note:** This operation might take up to 10 minutes to complete." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3Ql5LrIUO0kJ" + }, + "outputs": [], + "source": [ + "# Wait for the LRO to finish and get the LRO result.\n", + "print(create_store_lro.result())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ldCMH-IOOx-G" + }, + "source": [ + "#### Verify that the `FeatureOnlineStore` instance is created by getting the online stores instance" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "D3dZEtXKIqXT" + }, + "outputs": [], + "source": [ + "# Use get to verify the store is created.\n", + "admin_client.get_feature_online_store(\n", + " name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2vIjCzCwIGMF" + }, + "source": [ + "#### List all online stores for the location" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rk8JseF4ICur" + }, + "outputs": [], + "source": [ + "# Use list to verify the store is created.\n", + "admin_client.list_feature_online_stores(\n", + " parent=f\"projects/{PROJECT_ID}/locations/{REGION}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7QZnKMo41ud5" + }, + "source": [ + "### Create feature view instance\n", + "\n", + "After creating a `FeatureOnlineStore` instance, you define the features to serve with it. To do this, create a `FeatureView` instance, which specifies the following:\n", + "\n", + "* A data source (BigQuery table or view URI or FeatureGroup/features ) synced to the `FeatureOnlineStore` instance for serving.\n", + "* The cron schedule to run the sync pipeline.\n", + "\n", + "Within feature view creation, a sync job will be scheduled, either started immediately or following the cron schedule. In the sync job, data is exported to Cloud Bigtable, index is built and deployed to GKE cluster." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y9EfVmeu5KBu" + }, + "outputs": [], + "source": [ + "FEATURE_VIEW_ID = \"feature_view_publications\" # @param {type: \"string\"}\n", + "# A schedule will be created based on cron setting.\n", + "# If cron is empty, an immediate schedule job will be started.\n", + "CRON_SCHEDULE = \"TZ=America/Los_Angeles 00 13 11 8 *\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8VQ4P2om6SNQ" + }, + "outputs": [], + "source": [ + "# Vector search configs\n", + "DIMENSIONS = 64 # @param {type: \"number\"}\n", + "EMBEDDING_COLUMN = \"embedding\" # @param {type: \"string\"}\n", + "# Optional\n", + "LEAF_NODE_EMBEDDING_COUNT = 10000 # @param {type: \"number\"}\n", + "# Optional\n", + "CROWDING_COLUMN = \"cited_by_filing_date\" # @param {type: \"string\"}\n", + "# Optional\n", + "FILTER_COLUMNS = [\"country\"] # @param" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AKdSax3N1ypm" + }, + "outputs": [], + "source": [ + "big_query_source = feature_view_pb2.FeatureView.BigQuerySource(\n", + " uri=DATA_SOURCE, entity_id_columns=[\"publication_number\"]\n", + ")\n", + "\n", + "sync_config = feature_view_pb2.FeatureView.SyncConfig(cron=CRON_SCHEDULE)\n", + "\n", + "vector_search_config = feature_view_pb2.FeatureView.VectorSearchConfig(\n", + " embedding_column=EMBEDDING_COLUMN,\n", + " filter_columns=FILTER_COLUMNS,\n", + " crowding_column=CROWDING_COLUMN,\n", + " embedding_dimension=DIMENSIONS,\n", + " tree_ah_config=feature_view_pb2.FeatureView.VectorSearchConfig.TreeAHConfig(),\n", + ")\n", + "\n", + "print(f\"vector_search_config: {vector_search_config}\")\n", + "\n", + "create_view_lro = admin_client.create_feature_view(\n", + " feature_online_store_admin_service_pb2.CreateFeatureViewRequest(\n", + " parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\",\n", + " feature_view_id=FEATURE_VIEW_ID,\n", + " feature_view=feature_view_pb2.FeatureView(\n", + " big_query_source=big_query_source,\n", + " sync_config=sync_config,\n", + " vector_search_config=vector_search_config,\n", + " ),\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vDhR7WOTOthp" + }, + "source": [ + " Wait for LRO to complete and show result" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "uV5rT5UKOqzv" + }, + "outputs": [], + "source": [ + "print(create_view_lro.result())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mmYQZroEO7dp" + }, + "source": [ + "### Verify feature view instance creation\n", + "\n", + "Verify that the FeatureView instance is created by gettting the feature view." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PaqTIYuwPFC7" + }, + "outputs": [], + "source": [ + "admin_client.get_feature_view(\n", + " name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sqpu4nHAO4pW" + }, + "source": [ + "Verify that the FeatureView instance is created by listing all the feature views within the online store." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "O_tv0Z3BPHW4" + }, + "outputs": [], + "source": [ + "admin_client.list_feature_views(\n", + " parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KZglLt0iuVs0" + }, + "source": [ + "### Feature view syncs\n", + "\n", + "The sync pipeline executes according to the schedule specified in the `FeatureView` instance.\n", + "\n", + "To skip the wait and execute the sync pipeline immediately, start the sync manually." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "G-j5B6InuNfW" + }, + "outputs": [], + "source": [ + "sync_response = admin_client.sync_feature_view(\n", + " feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GqyYqYplMUjZ" + }, + "source": [ + "The `sync_response` contains the ID of the sync job.\n", + "\n", + "#### Use `get_feature_view_sync` to check the status of the job" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_kl8ZBAR2o0b" + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "while True:\n", + " feature_view_sync = admin_client.get_feature_view_sync(\n", + " name=sync_response.feature_view_sync\n", + " )\n", + " if feature_view_sync.run_time.end_time.seconds > 0:\n", + " status = \"Succeed\" if feature_view_sync.final_status.code == 0 else \"Failed\"\n", + " print(f\"Sync {status} for {feature_view_sync.name}.\")\n", + " # wait a little more for the job to properly shutdown\n", + " time.sleep(30)\n", + " break\n", + " else:\n", + " print(\"Sync ongoing, waiting for 30 seconds.\")\n", + " time.sleep(30)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-3XHiHnAIJGj" + }, + "source": [ + "#### Use `list_feature_view_syncs` to view all your syncs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3CU67MyLIOAk" + }, + "outputs": [], + "source": [ + "admin_client.list_feature_view_syncs(\n", + " parent=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eUBuFxXBbYOV" + }, + "source": [ + "### Start online serving\n", + "\n", + "After the data sync is complete, use the `FetchFeatureValuesRequest` and `SearchNearestEntities` APIs to retrieve the data.\n", + "\n", + "Get public endpoint domain name." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ssz79-yvgc1P" + }, + "outputs": [], + "source": [ + "# Use get to verify the store is created.\n", + "featore_online_store_instance = admin_client.get_feature_online_store(\n", + " name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\"\n", + ")\n", + "PUBLIC_ENDPOINT = (\n", + " featore_online_store_instance.dedicated_serving_endpoint.public_endpoint_domain_name\n", + ")\n", + "\n", + "print(f\"PUBLIC_ENDPOINT for online serving: {PUBLIC_ENDPOINT}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tpyAVYIkUX0N" + }, + "source": [ + "#### Initialize the data client" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PKJKZx36TT8d" + }, + "outputs": [], + "source": [ + "data_client = FeatureOnlineStoreServiceClient(\n", + " client_options={\"api_endpoint\": PUBLIC_ENDPOINT}\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jd7gHqZ8blJ-" + }, + "source": [ + "#### Set `NearestNeighborQuery.StringFilter`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y4erlJr3_1Du" + }, + "outputs": [], + "source": [ + "results_df = (\n", + " bq_client.query(f\"select publication_number from {BQ_TABLE_ID_FQN} limit 1\")\n", + " .result()\n", + " .to_dataframe()\n", + ")\n", + "ENTITY_ID = results_df.loc[0, \"publication_number\"]\n", + "print(f\"Sample publication number: {ENTITY_ID}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7sYPMibU3YwR" + }, + "outputs": [], + "source": [ + "country_filter = NearestNeighborQuery.StringFilter(\n", + " name=\"country\",\n", + " allow_tokens=[\"WIPO (PCT)\"], # try different allow tokens\n", + " deny_tokens=[\"United States\"], # try different deny tokens\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "v5gYpORiBW1n" + }, + "source": [ + "#### Search with `ENTITY_ID`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NWhrk3pqSU6h" + }, + "outputs": [], + "source": [ + "data_client.search_nearest_entities(\n", + " request=feature_online_store_service_pb2.SearchNearestEntitiesRequest(\n", + " feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\",\n", + " query=NearestNeighborQuery(\n", + " entity_id=ENTITY_ID, neighbor_count=5, string_filters=[country_filter]\n", + " ),\n", + " return_full_entity=True, # returning entities with metadata\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OQEpt08GBX-b" + }, + "source": [ + "#### Search with `Embedding`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "JWcOT26krxuj" + }, + "outputs": [], + "source": [ + "EMBEDDINGS = [1] * DIMENSIONS" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "e3DZl0xRAusK" + }, + "outputs": [], + "source": [ + "data_client.search_nearest_entities(\n", + " request=feature_online_store_service_pb2.SearchNearestEntitiesRequest(\n", + " feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\",\n", + " query=NearestNeighborQuery(\n", + " embedding=NearestNeighborQuery.Embedding(value=EMBEDDINGS),\n", + " neighbor_count=10,\n", + " string_filters=[country_filter],\n", + " ),\n", + " return_full_entity=True, # returning entities with metadata\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MKALOxbsZfce" + }, + "source": [ + "#### Use the `FetchFeatureValues` API to retrieve the full data without search\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_7op0efLdDFs" + }, + "outputs": [], + "source": [ + "data_client.fetch_feature_values(\n", + " request=feature_online_store_service_pb2.FetchFeatureValuesRequest(\n", + " feature_view=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\",\n", + " id=ENTITY_ID,\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OR1ve2OyyiKq" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Um27J_vvvzGc" + }, + "outputs": [], + "source": [ + "# Delete Feature View\n", + "admin_client.delete_feature_view(\n", + " name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}/featureViews/{FEATURE_VIEW_ID}\"\n", + ")\n", + "\n", + "# Delete Feature Online Store\n", + "admin_client.delete_feature_online_store(\n", + " name=f\"projects/{PROJECT_ID}/locations/{REGION}/featureOnlineStores/{FEATURE_ONLINE_STORE_ID}\",\n", + " force=True,\n", + ")" + ] + } + ], + "metadata": { + "colab": { + "name": "online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}