diff --git a/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb b/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb new file mode 100644 index 000000000..069fed266 --- /dev/null +++ b/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb @@ -0,0 +1,598 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e" + }, + "source": [ + "## Fetch historical feature values\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Open in Colab\n", + " \n", + " 		\n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Workbench\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "In this tutorial, you will learn how to use the Vertex AI SDK for Python to retrieve historical values from the feature data source in BigQuery.\n", + "\n", + "This tutorial uses the following Google Cloud ML services and resources:\n", + "\n", + "* Vertex AI Feature Store\n", + "* BigQuery\n", + "\n", + "The steps performed include the following:\n", + "\n", + "* Setup BigQuery data\n", + "* Setup Feature Registry\n", + "* Fetch historical feature values from feature data source in BigQuery\n", + "* Clean up" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "19cf444ebb99" + }, + "source": [ + "### Objective" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "61RBz8LLbxCR" + }, + "source": [ + "## Get started" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "No17Cw5hgx12" + }, + "source": [ + "### Install Vertex AI SDK for Python and other required packages\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tFy3H3aPgx12" + }, + "outputs": [], + "source": [ + "! pip3 install --upgrade --quiet google-cloud-aiplatform bigframes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R5Xep4W9lq-Z" + }, + "source": [ + "### Restart runtime (Colab only)\n", + "\n", + "To use the newly installed packages, you must restart the runtime on Google Colab." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XRvKdaPDTznN" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SbmM4z7FOBpM" + }, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dmWOrTJ3gx13" + }, + "source": [ + "### Authenticate your notebook environment (Colab only)\n", + "\n", + "Authenticate your environment on Google Colab.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NyKGtVQjgx13" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DF4l8DTdWgPY" + }, + "source": [ + "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n", + "\n", + "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Nqwi-5ufWp_B" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}\n", + "\n", + "\n", + "import vertexai\n", + "\n", + "vertexai.init(project=PROJECT_ID, location=LOCATION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "33067053f38b" + }, + "source": [ + "### Imports and IDs\n", + "\n", + "Import the packages required to use the`fetch_historical_feature_values()`\n", + "function in the Vertex AI SDK for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c8abe818393b" + }, + "outputs": [], + "source": [ + "import bigframes\n", + "import bigframes.pandas\n", + "import pandas as pd\n", + "from google.cloud import bigquery\n", + "from vertexai.resources.preview.feature_store import (Feature, FeatureGroup,\n", + " offline_store)\n", + "from vertexai.resources.preview.feature_store import utils as fs_utils" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4d0295f5d524" + }, + "source": [ + "The following variables set BigQuery and Feature Group resources that will be\n", + "used or created. If you'd like to use your own data source (CSV), please adjust\n", + "`DATA_SOURCE`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ac036ecfbc32" + }, + "outputs": [], + "source": [ + "BQ_DATASET_ID = \"fhfv_dataset_unique\" # @param {type:\"string\"}\n", + "BQ_TABLE_ID = \"fhfv_table_unique\" # @param {type:\"string\"}\n", + "BQ_TABLE_URI = f\"{PROJECT_ID}.{BQ_DATASET_ID}.{BQ_TABLE_ID}\"\n", + "\n", + "FEATURE_GROUP_ID = \"fhfv_fg_unique\" # @param {type:\"string\"}\n", + "\n", + "DATA_SOURCE = \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cd580a0679ce" + }, + "source": [ + "## Create BigQuery table containing feature data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1e2c688b844b" + }, + "source": [ + "First we'll use BigQuery DataFrames to load in our CSV data source. Then we'll\n", + "rename the `timestamp` column to `feature_timestamp` to support usage as a\n", + "BigQuery source in Feature Registry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1ff4481243a8" + }, + "outputs": [], + "source": [ + "session = bigframes.connect(\n", + " bigframes.BigQueryOptions(\n", + " project=PROJECT_ID,\n", + " location=LOCATION,\n", + " )\n", + ")\n", + "df = session.read_csv(DATA_SOURCE)\n", + "df[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"], utc=True)\n", + "df = df.rename(columns={\"timestamp\": \"feature_timestamp\"})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "de5008f71567" + }, + "source": [ + "Let's preview the data we'll write to the table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "38b448c47657" + }, + "outputs": [], + "source": [ + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "967ec2a0193f" + }, + "source": [ + "And finally we'll write the DataFrame to the target BigQuery table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4c11b88ab55d" + }, + "outputs": [], + "source": [ + "df.to_gbq(BQ_TABLE_URI, if_exists=\"replace\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "818a36b9da86" + }, + "source": [ + "## Create feature registry resources" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2fd96d0d8628" + }, + "source": [ + "Create a feature group backed by the BigQuery table created above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1f915ddd4669" + }, + "outputs": [], + "source": [ + "fg: FeatureGroup = FeatureGroup.create(\n", + " f\"{FEATURE_GROUP_ID}\",\n", + " fs_utils.FeatureGroupBigQuerySource(\n", + " uri=f\"bq://{BQ_TABLE_URI}\", entity_id_columns=[\"users\"]\n", + " ),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ebb9179572f7" + }, + "source": [ + "Create the `movies` feature which corresponds to the `movies` column in the\n", + "recently created BigQuery table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0dba1c02883c" + }, + "outputs": [], + "source": [ + "movies_feature: Feature = fg.create_feature(\"movies\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c71964219962" + }, + "source": [ + "## Fetch historical feature values" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8eb9b5257006" + }, + "source": [ + "### Fetch historical feature values for an entity" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "31edf36830b7" + }, + "source": [ + "The following will fetch historical feature values for the same entity (`alice`)\n", + "at two different timestamps. We expect the values of the `movies` feature at\n", + "each of those timestamps." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8fb9774db748" + }, + "outputs": [], + "source": [ + "entity_df = pd.DataFrame(\n", + " data={\n", + " \"users\": [\"alice\", \"alice\"],\n", + " \"timestamp\": [\n", + " pd.Timestamp(\"2021-09-14T09:36\"),\n", + " pd.Timestamp(\"2023-12-12T13:13\"),\n", + " ],\n", + " },\n", + ")\n", + "\n", + "offline_store.fetch_historical_feature_values(\n", + " entity_df=entity_df,\n", + " features=[movies_feature],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f9ecae3005df" + }, + "source": [ + "### Fetch with multiple entities" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f463b68f2f4f" + }, + "source": [ + "The following will fetch historical feature values for two different entities\n", + "at different timestamps. We expect the values of the `movies` feature for each\n", + "entity at it's corresponding timestamp." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "80ab288afd22" + }, + "outputs": [], + "source": [ + "entity_df = pd.DataFrame(\n", + " data={\n", + " \"users\": [\"alice\", \"bob\"],\n", + " \"timestamp\": [\n", + " pd.Timestamp(\"2021-09-14T09:36\"),\n", + " pd.Timestamp(\"2023-12-12T13:13\"),\n", + " ],\n", + " },\n", + ")\n", + "\n", + "offline_store.fetch_historical_feature_values(\n", + " entity_df=entity_df,\n", + " features=[movies_feature],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2a4e033321ad" + }, + "source": [ + "## Cleaning up" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ffd3dc65a25f" + }, + "source": [ + "### Delete feature and feature group" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7517048d8510" + }, + "outputs": [], + "source": [ + "movies_feature.delete()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8488287340ca" + }, + "outputs": [], + "source": [ + "fg.delete()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "753e85f60d06" + }, + "source": [ + "### Delete BigQuery dataset and table" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "99e984fe9c53" + }, + "outputs": [], + "source": [ + "client = bigquery.Client()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6cc5fdf51c9e" + }, + "outputs": [], + "source": [ + "client.delete_table(f\"{BQ_TABLE_URI}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4bac93a9ffcb" + }, + "outputs": [], + "source": [ + "client.delete_dataset(f\"{PROJECT_ID}.{BQ_DATASET_ID}\")" + ] + } + ], + "metadata": { + "colab": { + "name": "offline_feature_serving_from_bigquery_with_feature_registry.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}