mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
feat: add offline feature serving notebook (#3184)
This commit is contained in:
Manu Kumar
+598
@@ -0,0 +1,598 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2024 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "JAPoU8Sm5E6e"
|
||||
},
|
||||
"source": [
|
||||
"## Fetch historical feature values\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Ffeature_store%2Foffline_feature_serving_from_bigquery_with_feature_registry.ipynb\">\n",
|
||||
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/offline_feature_serving_from_bigquery_with_feature_registry.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"In this tutorial, you will learn how to use the Vertex AI SDK for Python to retrieve historical values from the feature data source in BigQuery.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services and resources:\n",
|
||||
"\n",
|
||||
"* Vertex AI Feature Store\n",
|
||||
"* BigQuery\n",
|
||||
"\n",
|
||||
"The steps performed include the following:\n",
|
||||
"\n",
|
||||
"* Setup BigQuery data\n",
|
||||
"* Setup Feature Registry\n",
|
||||
"* Fetch historical feature values from feature data source in BigQuery\n",
|
||||
"* Clean up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "19cf444ebb99"
|
||||
},
|
||||
"source": [
|
||||
"### Objective"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "61RBz8LLbxCR"
|
||||
},
|
||||
"source": [
|
||||
"## Get started"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "No17Cw5hgx12"
|
||||
},
|
||||
"source": [
|
||||
"### Install Vertex AI SDK for Python and other required packages\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "tFy3H3aPgx12"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade --quiet google-cloud-aiplatform bigframes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "R5Xep4W9lq-Z"
|
||||
},
|
||||
"source": [
|
||||
"### Restart runtime (Colab only)\n",
|
||||
"\n",
|
||||
"To use the newly installed packages, you must restart the runtime on Google Colab."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "XRvKdaPDTznN"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "SbmM4z7FOBpM"
|
||||
},
|
||||
"source": [
|
||||
"<div class=\"alert alert-block alert-warning\">\n",
|
||||
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
||||
"</div>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dmWOrTJ3gx13"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your notebook environment (Colab only)\n",
|
||||
"\n",
|
||||
"Authenticate your environment on Google Colab.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "NyKGtVQjgx13"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" from google.colab import auth\n",
|
||||
"\n",
|
||||
" auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "DF4l8DTdWgPY"
|
||||
},
|
||||
"source": [
|
||||
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Nqwi-5ufWp_B"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"import vertexai\n",
|
||||
"\n",
|
||||
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "33067053f38b"
|
||||
},
|
||||
"source": [
|
||||
"### Imports and IDs\n",
|
||||
"\n",
|
||||
"Import the packages required to use the`fetch_historical_feature_values()`\n",
|
||||
"function in the Vertex AI SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c8abe818393b"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import bigframes\n",
|
||||
"import bigframes.pandas\n",
|
||||
"import pandas as pd\n",
|
||||
"from google.cloud import bigquery\n",
|
||||
"from vertexai.resources.preview.feature_store import (Feature, FeatureGroup,\n",
|
||||
" offline_store)\n",
|
||||
"from vertexai.resources.preview.feature_store import utils as fs_utils"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "4d0295f5d524"
|
||||
},
|
||||
"source": [
|
||||
"The following variables set BigQuery and Feature Group resources that will be\n",
|
||||
"used or created. If you'd like to use your own data source (CSV), please adjust\n",
|
||||
"`DATA_SOURCE`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ac036ecfbc32"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BQ_DATASET_ID = \"fhfv_dataset_unique\" # @param {type:\"string\"}\n",
|
||||
"BQ_TABLE_ID = \"fhfv_table_unique\" # @param {type:\"string\"}\n",
|
||||
"BQ_TABLE_URI = f\"{PROJECT_ID}.{BQ_DATASET_ID}.{BQ_TABLE_ID}\"\n",
|
||||
"\n",
|
||||
"FEATURE_GROUP_ID = \"fhfv_fg_unique\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"DATA_SOURCE = \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cd580a0679ce"
|
||||
},
|
||||
"source": [
|
||||
"## Create BigQuery table containing feature data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1e2c688b844b"
|
||||
},
|
||||
"source": [
|
||||
"First we'll use BigQuery DataFrames to load in our CSV data source. Then we'll\n",
|
||||
"rename the `timestamp` column to `feature_timestamp` to support usage as a\n",
|
||||
"BigQuery source in Feature Registry."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1ff4481243a8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"session = bigframes.connect(\n",
|
||||
" bigframes.BigQueryOptions(\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" location=LOCATION,\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"df = session.read_csv(DATA_SOURCE)\n",
|
||||
"df[\"timestamp\"] = pd.to_datetime(df[\"timestamp\"], utc=True)\n",
|
||||
"df = df.rename(columns={\"timestamp\": \"feature_timestamp\"})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "de5008f71567"
|
||||
},
|
||||
"source": [
|
||||
"Let's preview the data we'll write to the table."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "38b448c47657"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "967ec2a0193f"
|
||||
},
|
||||
"source": [
|
||||
"And finally we'll write the DataFrame to the target BigQuery table."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4c11b88ab55d"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df.to_gbq(BQ_TABLE_URI, if_exists=\"replace\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "818a36b9da86"
|
||||
},
|
||||
"source": [
|
||||
"## Create feature registry resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2fd96d0d8628"
|
||||
},
|
||||
"source": [
|
||||
"Create a feature group backed by the BigQuery table created above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1f915ddd4669"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fg: FeatureGroup = FeatureGroup.create(\n",
|
||||
" f\"{FEATURE_GROUP_ID}\",\n",
|
||||
" fs_utils.FeatureGroupBigQuerySource(\n",
|
||||
" uri=f\"bq://{BQ_TABLE_URI}\", entity_id_columns=[\"users\"]\n",
|
||||
" ),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ebb9179572f7"
|
||||
},
|
||||
"source": [
|
||||
"Create the `movies` feature which corresponds to the `movies` column in the\n",
|
||||
"recently created BigQuery table."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "0dba1c02883c"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"movies_feature: Feature = fg.create_feature(\"movies\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c71964219962"
|
||||
},
|
||||
"source": [
|
||||
"## Fetch historical feature values"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8eb9b5257006"
|
||||
},
|
||||
"source": [
|
||||
"### Fetch historical feature values for an entity"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "31edf36830b7"
|
||||
},
|
||||
"source": [
|
||||
"The following will fetch historical feature values for the same entity (`alice`)\n",
|
||||
"at two different timestamps. We expect the values of the `movies` feature at\n",
|
||||
"each of those timestamps."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "8fb9774db748"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"entity_df = pd.DataFrame(\n",
|
||||
" data={\n",
|
||||
" \"users\": [\"alice\", \"alice\"],\n",
|
||||
" \"timestamp\": [\n",
|
||||
" pd.Timestamp(\"2021-09-14T09:36\"),\n",
|
||||
" pd.Timestamp(\"2023-12-12T13:13\"),\n",
|
||||
" ],\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"offline_store.fetch_historical_feature_values(\n",
|
||||
" entity_df=entity_df,\n",
|
||||
" features=[movies_feature],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f9ecae3005df"
|
||||
},
|
||||
"source": [
|
||||
"### Fetch with multiple entities"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f463b68f2f4f"
|
||||
},
|
||||
"source": [
|
||||
"The following will fetch historical feature values for two different entities\n",
|
||||
"at different timestamps. We expect the values of the `movies` feature for each\n",
|
||||
"entity at it's corresponding timestamp."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "80ab288afd22"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"entity_df = pd.DataFrame(\n",
|
||||
" data={\n",
|
||||
" \"users\": [\"alice\", \"bob\"],\n",
|
||||
" \"timestamp\": [\n",
|
||||
" pd.Timestamp(\"2021-09-14T09:36\"),\n",
|
||||
" pd.Timestamp(\"2023-12-12T13:13\"),\n",
|
||||
" ],\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"offline_store.fetch_historical_feature_values(\n",
|
||||
" entity_df=entity_df,\n",
|
||||
" features=[movies_feature],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2a4e033321ad"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ffd3dc65a25f"
|
||||
},
|
||||
"source": [
|
||||
"### Delete feature and feature group"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7517048d8510"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"movies_feature.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "8488287340ca"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fg.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "753e85f60d06"
|
||||
},
|
||||
"source": [
|
||||
"### Delete BigQuery dataset and table"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "99e984fe9c53"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = bigquery.Client()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "6cc5fdf51c9e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.delete_table(f\"{BQ_TABLE_URI}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4bac93a9ffcb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.delete_dataset(f\"{PROJECT_ID}.{BQ_DATASET_ID}\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "offline_feature_serving_from_bigquery_with_feature_registry.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
Reference in New Issue
Block a user