xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: rebrand "Vertex AI Feature Store (Legacy)", miscellaneous editorial changes, and changing "ingest" to "import" globally. (#2369)

Browse Source
This commit is contained in:
sen-sam
2023-10-12 20:09:17 +00:00
committed by GitHub
parent 283c0a3727
commit 3d4bd5fdfc
3 changed files with 74 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/feature_store/feature_store_streaming_ingestion_sdk.ipynb
+6 -6
View File
@@ -85,7 +85,7 @@
"source": [ "source": [
"### Objective\n", "### Objective\n",
"\n", "\n",
"In this tutorial, you learn how to ingest features from a `Pandas DataFrame` into Vertex AI Feature Store (Legacy) using `write_feature_values` method from the Vertex AI SDK.\n", "In this tutorial, you learn how to import features from a `Pandas DataFrame` into Vertex AI Feature Store (Legacy) using `write_feature_values` method from the Vertex AI SDK.\n",
"\n", "\n",
"This tutorial uses the following Google Cloud ML services and resources:\n", "This tutorial uses the following Google Cloud ML services and resources:\n",
"\n", "\n",
@@ -212,7 +212,7 @@
"**If you don't know your project ID**, try the following:\n", "**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n", "* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n", "* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)" "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
] ]
}, },
{ {
@@ -501,7 +501,7 @@
"\n", "\n",
"You must create these resources before you can import data into Vertex AI Feature Store (Legacy).\n", "You must create these resources before you can import data into Vertex AI Feature Store (Legacy).\n",
"\n", "\n",
"Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore)" "Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore)."
] ]
}, },
{ {
@@ -630,7 +630,7 @@
"### Create features\n", "### Create features\n",
"A feature is a measurable property or attribute of an entity type. For example, `penguin` entity type has features such as `flipper_length_mm`, and `body_mass_g`. Features can be created within each entity type.\n", "A feature is a measurable property or attribute of an entity type. For example, `penguin` entity type has features such as `flipper_length_mm`, and `body_mass_g`. Features can be created within each entity type.\n",
"\n", "\n",
"When you create a feature, you specify its value type such as `DOUBLE`, and `STRING`. This value determines what value types you can ingest for a particular feature.\n", "When you create a feature, you specify its value type such as `DOUBLE`, and `STRING`. This value determines what value types you can import for a particular feature.\n",
"\n", "\n",
"Learn more about [Feature Value Types](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.featurestores.entityTypes.features)" "Learn more about [Feature Value Types](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.featurestores.entityTypes.features)"
] ]
@@ -697,9 +697,9 @@
"\n", "\n",
"* `instances`: Feature values to be written to the featurestore that can take the form of a list of `WriteFeatureValuesPayload` objects, a Python dict, or a pandas Dataframe.\n", "* `instances`: Feature values to be written to the featurestore that can take the form of a list of `WriteFeatureValuesPayload` objects, a Python dict, or a pandas Dataframe.\n",
"\n", "\n",
"This streaming ingestion feature has been introduced to the Vertex AI SDK under the **preview** namespace. Here, you pass the pandas `Dataframe` you created from penguins dataset as `instances` parameter.\n", "Streaming import has been introduced in the Vertex AI SDK under the **preview** namespace. Here, you pass the pandas `Dataframe` you created from penguins dataset as `instances` parameter.\n",
"\n", "\n",
"Learn more about [Streaming ingestion API](https://github.com/googleapis/python-aiplatform/blob/e6933503d2d3a0f8a8f7ef8c178ed50a69ac2268/google/cloud/aiplatform/preview/featurestore/entity_type.py#L36)" "Learn more about [Streaming import API](https://github.com/googleapis/python-aiplatform/blob/e6933503d2d3a0f8a8f7ef8c178ed50a69ac2268/google/cloud/aiplatform/preview/featurestore/entity_type.py#L36)"
] ]
}, },
{ {
notebooks/official/feature_store/sdk-feature-store-pandas.ipynb
+58 -58
View File
@@ -29,7 +29,7 @@
"id": "JAPoU8Sm5E6e" "id": "JAPoU8Sm5E6e"
}, },
"source": [ "source": [
"# Using Vertex AI Feature Store with Pandas Dataframe\n", "# Using Vertex AI Feature Store (Legacy) with Pandas Dataframe\n",
"\n", "\n",
"<table align=\"left\">\n", "<table align=\"left\">\n",
" <td>\n", " <td>\n",
@@ -73,7 +73,7 @@
"source": [ "source": [
"## Overview\n", "## Overview\n",
"\n", "\n",
"This notebook introduces Pandas support for Feature Store using Vertex AI SDK. For pre-requisites and introduction on Vertex AI SDK and Feature Store native support, please go through this [Colab notebook](https://colab.sandbox.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb). \n", "This notebook introduces Pandas support for Vertex AI Feature Store (Legacy) using the Vertex AI SDK. For pre-requisites and introduction on Vertex AI SDK and Vertex AI Feature Store (Legacy) native support, please go through this [Colab notebook](https://colab.sandbox.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb). \n",
"\n", "\n",
"Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore)." "Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore)."
] ]
@@ -86,20 +86,20 @@
"source": [ "source": [
"### Objective\n", "### Objective\n",
"\n", "\n",
"In this notebook, you learn how to use `Vertex AI Feature Store` with pandas Dataframe.\n", "In this notebook, you learn how to use `Vertex AI Feature Store (Legacy)` with pandas Dataframe.\n",
"\n", "\n",
"This tutorial uses the following Google Cloud ML services and resources:\n", "This tutorial uses the following Google Cloud ML services and resources:\n",
"\n", "\n",
"- Vertex AI Feature Store\n", "- Vertex AI Feature Store (Legacy)\n",
"\n", "\n",
"The steps performed include:\n", "The steps performed include:\n",
"\n", "\n",
"- Create Featurestore, entity types and features.\n", "- Create `Featurestore`, `EntityType`, and `Feature` resources.\n",
"- Ingest feature values from Pandas DataFrame into Feature Store's Entity types.\n", "- Import feature values from Pandas DataFrame into the entity type.\n",
"- Read Entity feature values from Online Feature Store into Pandas DataFrame.\n", "- Read entity feature values from the online feature store into Pandas DataFrame.\n",
"- Batch serve feature values from your Feature Store into Pandas DataFrame.\n", "- Batch serve feature values from your featurestore into Pandas DataFrame.\n",
"\n", "\n",
"You also learn how Vertex AI Feature Store can be useful in the below scenarios:\n", "You also learn how Vertex AI Feature Store (Legacy) is useful in the following scenarios:\n",
"\n", "\n",
"- Online serving with updated feature values.\n", "- Online serving with updated feature values.\n",
"- Point-in-time correctness to fetch feature values for training." "- Point-in-time correctness to fetch feature values for training."
@@ -113,7 +113,7 @@
"source": [ "source": [
"### Dataset\n", "### Dataset\n",
"\n", "\n",
"This tutorial is a part of the Feature Store tutorial notebooks. It uses a movie recommendation dataset as an example for demonstrating various functionalities of Feature Store. The original task is to train a model to predict if a user is going to watch a movie, and serve the model online." "This tutorial is a part of the Vertex AI Feature Store (Legacy) tutorial notebooks. It uses a movie recommendation dataset as an example for demonstrating various functionalities of Vertex AI Feature Store (Legacy). The original task is to train a model to predict if a user is going to watch a movie, and serve the model online."
] ]
}, },
{ {
@@ -413,15 +413,15 @@
"id": "buQBIv3ZL3A0" "id": "buQBIv3ZL3A0"
}, },
"source": [ "source": [
"## Create a Feature Store\n", "## Create a featurestore\n",
"\n", "\n",
"Vertex AI Feature Store serves you as a centralised and organised repository for your ML features. You can store, serve and monitor certain aspects of your features like their distributions and drift. Learn more about [Vertex AI Feature Store data model](https://cloud.google.com/vertex-ai/docs/featurestore/concepts), and [benefits of Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/overview#benefits).\n", "Vertex AI Feature Store (Legacy) serves as a centralised and organised repository for your ML features. You can store, serve and monitor certain aspects of your features like their distributions and drift. Learn more about the [Vertex AI Feature Store (Legacy) data model](https://cloud.google.com/vertex-ai/docs/featurestore/concepts), and [benefits of Vertex AI Feature Store (Legacy)](https://cloud.google.com/vertex-ai/docs/featurestore/overview#benefits).\n",
"\n", "\n",
"To begin this tutorial, you create a feature store using the Vertex AI SDK for Python. A feature store is a top-level container for your features and their values. For this, you use the [`Featurestore.create()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Featurestore) method which returns a LRO ([long-running operation](https://google.aip.dev/151)). A LRO starts an asynchronous job. LROs are returned for other API methods too, such as updating or deleting a featurestore. \n", "To begin this tutorial, you create a featurestore using the Vertex AI SDK for Python. A featurestore is a top-level container for your features and their values. For this, you use the [`Featurestore.create()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Featurestore) method which returns a LRO ([long-running operation](https://google.aip.dev/151)). A LRO starts an asynchronous job. LROs are returned for other API methods too, such as updating or deleting a featurestore. \n",
"\n", "\n",
"You pass the below parameters while creating the featurestore:\n", "You pass the below parameters while creating the featurestore:\n",
"\n", "\n",
"- `featurestore_id`: A unique name or id for your featurestore\n", "- `featurestore_id`: A unique name or id for your featurestore.\n",
"- `online_store_fixed_node_count`: Config for online serving resources. The number of nodes will not scale automatically but can be scaled manually by providing different values when updating." "- `online_store_fixed_node_count`: Config for online serving resources. The number of nodes will not scale automatically but can be scaled manually by providing different values when updating."
] ]
}, },
@@ -433,7 +433,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# set the id or name for the feature store\n", "# set the id or name for the featurestore\n",
"featurestore_id = \"movie_predictions_unique\" # @param {type:\"string\"}\n", "featurestore_id = \"movie_predictions_unique\" # @param {type:\"string\"}\n",
"\n", "\n",
"# Create featurestore\n", "# Create featurestore\n",
@@ -448,13 +448,13 @@
"id": "EpmJq75zXjmT" "id": "EpmJq75zXjmT"
}, },
"source": [ "source": [
"## Create Entity types\n", "## Create entity types\n",
"\n", "\n",
"Using Vertex AI Feature Store, you can create and manage feature stores, entity types, and features. An entity type is a collection of semantically related features. You define your own entity types, based on the concepts that are relevant to your use case. For example, a movie service might have the entity types movie and user, which group related features that correspond to movies or customers.\n", "Using Vertex AI Feature Store (Legacy), you can create and manage featurestores, entity types, and features. An entity type is a collection of semantically related features. You define your own entity types, based on the concepts that are relevant to your use case. For example, a movie service might have the entity types movie and user, which group related features that correspond to movies or customers.\n",
"\n", "\n",
"Learn more about [Entity types](https://cloud.google.com/vertex-ai/docs/featurestore/concepts#entity_type).\n", "Learn more about [entity types](https://cloud.google.com/vertex-ai/docs/featurestore/concepts#entity_type).\n",
"\n", "\n",
"Entity types are created within the Featurestore class. Below, you create the following entity types `users` and `movies` for the movie recommendation dataset.\n", "Entity types are created within the `Featurestore` class. Below, you create the following entity types `users` and `movies` for the movie recommendation dataset.\n",
"\n", "\n",
"You pass the following parameters while creating the entity types:\n", "You pass the following parameters while creating the entity types:\n",
"\n", "\n",
@@ -489,7 +489,7 @@
"id": "FJW4q-0jO2Xf" "id": "FJW4q-0jO2Xf"
}, },
"source": [ "source": [
"## Create Features\n", "## Create features\n",
"\n", "\n",
"A feature is a measurable property or attribute of an entity type. For example, the movie entity type has features such as average_rating and title that track various properties of movies. Features are associated with entity types. \n", "A feature is a measurable property or attribute of an entity type. For example, the movie entity type has features such as average_rating and title that track various properties of movies. Features are associated with entity types. \n",
"\n", "\n",
@@ -501,9 +501,9 @@
"\n", "\n",
"You provide the following parameters for creating features:\n", "You provide the following parameters for creating features:\n",
"\n", "\n",
"- `feature_id`: Resource name or an id for the Feature.\n", "- `feature_id`: Resource name or an id for the feature.\n",
"- `value_type`: Type of Feature value. One of BOOL, BOOL_ARRAY, DOUBLE, DOUBLE_ARRAY, INT64, INT64_ARRAY, STRING, STRING_ARRAY, BYTES.\n", "- `value_type`: Type of feature value. One of BOOL, BOOL_ARRAY, DOUBLE, DOUBLE_ARRAY, INT64, INT64_ARRAY, STRING, STRING_ARRAY, BYTES.\n",
"- `description`: Description of the Feature." "- `description`: Description of the feature."
] ]
}, },
{ {
@@ -584,13 +584,13 @@
"id": "K3n5XdK8Xjmw" "id": "K3n5XdK8Xjmw"
}, },
"source": [ "source": [
"## Ingest Feature values into Entity types from dataframes\n", "## Import feature values into entity types from dataframes\n",
"\n", "\n",
"A Feature Store captures feature values for a feature belonging to an entity type at a specific point in time. After ingesting your feature values to feature store, you can later `read` (online) or `batch serve` (offline) the feature values from the entity type. \n", "A featurestore captures feature values for a feature belonging to an entity type at a specific point in time. After importing your feature values to feature store, you can later `read` (online) or `batch serve` (offline) the feature values from the entity type. \n",
"\n", "\n",
"In this section, you learn how to ingest feature values from a [Pandas dataframe](https://pandas.pydata.org/) into an entity type. \n", "In this section, you learn how to import feature values from a [Pandas dataframe](https://pandas.pydata.org/) into an entity type. \n",
"\n", "\n",
"Learn more about [Feature values](https://cloud.google.com/vertex-ai/docs/featurestore/concepts#feature_value).\n", "Learn more about [feature values](https://cloud.google.com/vertex-ai/docs/featurestore/concepts#feature_value).\n",
"\n", "\n",
"Note: You can also import feature values from BigQuery or Google Cloud Storage.\n", "Note: You can also import feature values from BigQuery or Google Cloud Storage.\n",
"\n", "\n",
@@ -688,15 +688,15 @@
"id": "bgb0WGwX5OW6" "id": "bgb0WGwX5OW6"
}, },
"source": [ "source": [
"### Ingest Feature values into Entity types\n", "### Import feature values into entity types\n",
"\n", "\n",
"Load the feature values into `users` entity type from the dataframe.\n", "Load the feature values into `users` entity type from the dataframe.\n",
"\n", "\n",
"You provide the following parameters for ingesting the data:\n", "You provide the following parameters for importing the data:\n",
"\n", "\n",
"- `feature_ids`: List of ids of the Feature to import values of. The Features must exist in the target EntityType, or the request will fail.\n", "- `feature_ids`: List of ids of the Feature to import values of. The Features must exist in the target EntityType, or the request will fail.\n",
"- `feature_time`: The source column that holds the Feature timestamp for all Feature values in each entity. It can also be a single Feature timestamp for all entities being imported.\n", "- `feature_time`: The source column that holds the Feature timestamp for all Feature values in each entity. It can also be a single Feature timestamp for all entities being imported.\n",
"- `df_source`: Pandas DataFrame containing the source data for ingestion.\n", "- `df_source`: Pandas DataFrame containing the source data to import.\n",
"- `entity_id_field`: Source column that holds entity IDs.\n", "- `entity_id_field`: Source column that holds entity IDs.\n",
"\n", "\n",
"Learn more about the [`EntityType.ingest_from_df()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.EntityType#google_cloud_aiplatform_EntityType_ingest_from_df) method." "Learn more about the [`EntityType.ingest_from_df()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.EntityType#google_cloud_aiplatform_EntityType_ingest_from_df) method."
@@ -710,7 +710,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# ingest the data for users\n", "# Import the data for users\n",
"users_entity_type.ingest_from_df(\n", "users_entity_type.ingest_from_df(\n",
" feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n", " feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n",
" feature_time=\"update_time\",\n", " feature_time=\"update_time\",\n",
@@ -751,13 +751,13 @@
"id": "pIYLZwao5OW6" "id": "pIYLZwao5OW6"
}, },
"source": [ "source": [
"## Read Entity's feature values online from Feature Store\n", "## Read feature values online from an entity within a featurestore\n",
"\n", "\n",
"Feature Store allows online serving which lets you read feature values for small batches of entities. It is beneficial when you want to read values of selected features from an entity or multiple entities in an entity type.\n", "Vertex AI Feature Store (Legacy) includes online serving, which lets you read feature values for small batches of entities. It is beneficial when you want to read values of selected features from an entity or multiple entities in an entity type.\n",
"\n", "\n",
"Note: An entity is an instance of an entity type. For example, movie_01 and movie_02 are entities of the entity type movie.\n", "Note: An entity is an instance of an entity type. For example, movie_01 and movie_02 are entities of the entity type movie.\n",
"\n", "\n",
"Learn more about [Feature Store online serving](https://cloud.google.com/vertex-ai/docs/featurestore/serving-online).\n", "Learn more about [online serving in Vertex AI Feature Store (Legacy)](https://cloud.google.com/vertex-ai/docs/featurestore/serving-online).\n",
"\n", "\n",
"### Read feature values for users\n", "### Read feature values for users\n",
"\n", "\n",
@@ -814,13 +814,13 @@
"id": "AK2Glzkq5OW7" "id": "AK2Glzkq5OW7"
}, },
"source": [ "source": [
"## Batch serve feature values from Feature Store\n", "## Batch serve feature values from featurestore\n",
"\n", "\n",
"Feature Store can also serve the feature values in large batches for high-throughput. Batch serving is typically used for training a model or batch prediction. \n", "Vertex AI Feature Store (Legacy) can also serve the feature values in large batches for high throughput. Batch serving is typically used for training a model or batch prediction. \n",
"\n", "\n",
"Learn more about [Feature Store batch serving](https://cloud.google.com/vertex-ai/docs/featurestore/serving-batch#batch_serving_inputs).\n", "Learn more about [batch serving in Vertex AI Feature Store (Legacy)](https://cloud.google.com/vertex-ai/docs/featurestore/serving-batch#batch_serving_inputs).\n",
"\n", "\n",
"In this section, you learn how to prepare training examples by using the Feature Store's batch serve function.\n", "In this section, you learn how to prepare training examples by using the batch serve function in Vertex AI Feature Store (Legacy).\n",
"\n", "\n",
"### Load read instances from source file\n", "### Load read instances from source file\n",
"\n", "\n",
@@ -868,13 +868,13 @@
"id": "ao1dC5Pc5OW8" "id": "ao1dC5Pc5OW8"
}, },
"source": [ "source": [
"### Batch serve feature values from Feature Store\n", "### Batch serve feature values\n",
"\n", "\n",
"Serve the batch response to a dataframe using the `batch_serve_to_df` method by providing the following parameters:\n", "Serve the batch response to a dataframe using the `batch_serve_to_df` method by providing the following parameters:\n",
"\n", "\n",
"- `serving_feature_ids`: A user defined dictionary to define the entity_types and their features for batch serve/read. The keys of the dictionary are the serving entity_type ids and the values are lists of serving feature ids in each entity_type.\n", "- `serving_feature_ids`: A user defined dictionary to define the entity_types and their features for batch serve/read. The keys of the dictionary are the serving entity_type ids and the values are lists of serving feature ids in each entity_type.\n",
" \n", " \n",
"- `read_instances_df`: A pandas DataFrame containing the read instances. Each read instance should consist of exactly one read timestamp and one or more entity IDs identifying entities of the corresponding EntityTypes whose Features are requested. Each output instance contains Feature values of requested entities concatenated together as of the read time. \n", "- `read_instances_df`: A pandas DataFrame containing the read instances. Each read instance should consist of exactly one read timestamp and one or more entity IDs identifying entities of the corresponding entity types whose features are requested. Each output instance contains feature values of requested entities concatenated together as of the read time. \n",
"\n", "\n",
" An example read_instances_df may be:\n", " An example read_instances_df may be:\n",
"\n", "\n",
@@ -898,9 +898,9 @@
" ``` \n", " ``` \n",
" \n", " \n",
"\n", "\n",
"Note: Calling the `batch_serve_to_df` method automatically creates and deletes a temporary bigquery dataset in the same GCP project, which is used as the intermediary storage for batch serve feature values from featurestore to dataframe.\n", "Note: Calling the `batch_serve_to_df` method automatically creates and deletes a temporary bigquery dataset in the same GCP project, which is used as the intermediary storage for batch serve feature values from Vertex AI Feature Store (Legacy) to dataframe.\n",
"\n", "\n",
"Learn more about [Feature Store batch serving to a dataframe](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Featurestore#google_cloud_aiplatform_Featurestore_batch_serve_to_df). " "Learn more about [batch serving from Vertex AI Feature Store (Legacy) to a dataframe](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Featurestore#google_cloud_aiplatform_Featurestore_batch_serve_to_df). "
] ]
}, },
{ {
@@ -931,11 +931,11 @@
"source": [ "source": [
"## Read the latest feature values\n", "## Read the latest feature values\n",
"\n", "\n",
"In Feature Store, you access the latest or the last available feature values unless a specific time is provided. \n", "In Vertex AI Feature Store (Legacy), you access the latest or the last available feature values unless a specific time is provided. \n",
"\n", "\n",
"Now, you test this feature by ingesting new data to the entity types and reading it from the Feature Store.\n", "Now, you test this feature by importing new data to the entity types and reading it from the featurestore.\n",
"\n", "\n",
"### Ingest updated feature values\n", "### Import updated feature values\n",
"\n", "\n",
"Now, you update the feature values by running the following cell. \n", "Now, you update the feature values by running the following cell. \n",
"\n", "\n",
@@ -956,7 +956,7 @@
" columns=[\"movie_id\", \"average_rating\"],\n", " columns=[\"movie_id\", \"average_rating\"],\n",
")\n", ")\n",
"\n", "\n",
"# Ingest the new data from the dataframe\n", "# Import the new data from the dataframe\n",
"movies_entity_type.ingest_from_df(\n", "movies_entity_type.ingest_from_df(\n",
" feature_ids=[\"average_rating\"],\n", " feature_ids=[\"average_rating\"],\n",
" feature_time=datetime.datetime.now(), # provide the current timestamp\n", " feature_time=datetime.datetime.now(), # provide the current timestamp\n",
@@ -973,7 +973,7 @@
"source": [ "source": [
"### Fetch the latest feature values\n", "### Fetch the latest feature values\n",
"\n", "\n",
"Reading from the entity type gives you the updated feature values from the latest ingestion.\n", "Reading from the entity type gives you the updated feature values from the latest import.\n",
"\n", "\n",
"Running the below cell should fetch the latest values for all the requested entities including `movie_03` and `movie_04` which you added in the last cell." "Running the below cell should fetch the latest values for all the requested entities including `movie_03` and `movie_04` which you added in the last cell."
] ]
@@ -1003,10 +1003,10 @@
"source": [ "source": [
"## Point-in-time correctness\n", "## Point-in-time correctness\n",
"\n", "\n",
"Vertex AI Feature Store captures feature values for a feature at a [specific point in time](https://cloud.google.com/vertex-ai/docs/featurestore/serving-batch#example_point-in-time_lookup). In case there are missing values in your past data, you can backfill them using batch serving.\n", "Vertex AI Feature Store (Legacy) captures feature values for a feature at a [specific point in time](https://cloud.google.com/vertex-ai/docs/featurestore/serving-batch#example_point-in-time_lookup). In case there are missing values in your past data, you can backfill them using batch serving.\n",
"\n", "\n",
"### Check missing data\n", "### Check missing data\n",
"Recall that response from the batch serve from last ingestion has some missing data in it." "Recall that response from the batch serve from last import has some missing data in it."
] ]
}, },
{ {
@@ -1029,9 +1029,9 @@
"source": [ "source": [
"### Backfill / correct point-in-time data\n", "### Backfill / correct point-in-time data\n",
"\n", "\n",
"Impute the missing data based on thetimestamps.\n", "Impute the missing data based on the timestamps.\n",
"\n", "\n",
"Note: The timestamp field should must use the RFC 3339 format(e.g. 2012-07-30T10:43:17.123Z) or should be compatible with the Timestamp datatype when loaded to BigQuery. This is because Feature Stores loads to a temporary BigQuery table as an intermediate step when batch serving. Learn more about [loading data to BigQuery from dataframe](https://cloud.google.com/bigquery/docs/samples/bigquery-load-table-dataframe)." "Note: The timestamp field should must use the RFC 3339 format(e.g. 2012-07-30T10:43:17.123Z) or should be compatible with the Timestamp datatype when loaded to BigQuery. This is because Vertex AI Feature Store (Legacy) loads to a temporary BigQuery table as an intermediate step when batch serving. Learn more about [loading data to BigQuery from dataframe](https://cloud.google.com/bigquery/docs/samples/bigquery-load-table-dataframe)."
] ]
}, },
{ {
@@ -1078,9 +1078,9 @@
"id": "WXb4JUhu5OW-" "id": "WXb4JUhu5OW-"
}, },
"source": [ "source": [
"### Ingest the backfilled / corrected data\n", "### Import the backfilled / corrected data\n",
"\n", "\n",
"Ingest the imputed point-in-time data from dataframe to the entity types in feature store." "Import the imputed point-in-time data from dataframe to the entity types in featurestore."
] ]
}, },
{ {
@@ -1091,7 +1091,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Ingest the users data\n", "# Import the users data\n",
"users_entity_type.ingest_from_df(\n", "users_entity_type.ingest_from_df(\n",
" feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n", " feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n",
" feature_time=\"update_time\",\n", " feature_time=\"update_time\",\n",
@@ -1108,7 +1108,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Ingest the users data\n", "# Import the users data\n",
"movies_entity_type.ingest_from_df(\n", "movies_entity_type.ingest_from_df(\n",
" feature_ids=[\"average_rating\", \"title\", \"genres\"],\n", " feature_ids=[\"average_rating\", \"title\", \"genres\"],\n",
" feature_time=\"update_time\",\n", " feature_time=\"update_time\",\n",
@@ -1124,7 +1124,7 @@
}, },
"source": [ "source": [
"### Fetch the latest data\n", "### Fetch the latest data\n",
"Batch serve the ingested backfilled data to a dataframe to ensure the feature store is updated. " "Batch serve the imported backfilled data to a dataframe to ensure the featurestore is updated. "
] ]
}, },
{ {
@@ -1160,7 +1160,7 @@
"\n", "\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n", "Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n", "\n",
"- Vertex AI Feature Store\n", "- Vertex AI Feature Store (Legacy)\n",
"- Cloud Storage bucket (set `delete_bucket` to True)" "- Cloud Storage bucket (set `delete_bucket` to True)"
] ]
}, },
@@ -1172,7 +1172,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Delete the feature store\n", "# Delete the featurestore\n",
"movie_predictions_feature_store.delete(force=True)\n", "movie_predictions_feature_store.delete(force=True)\n",
"\n", "\n",
"# remove the local users and movies avro files\n", "# remove the local users and movies avro files\n",
notebooks/official/feature_store/sdk-feature-store.ipynb
+10 -11
View File
@@ -79,7 +79,7 @@
"\n", "\n",
"This tutorial uses the following Google Cloud ML services:\n", "This tutorial uses the following Google Cloud ML services:\n",
"\n", "\n",
"- `Vertex AI Feature Store`\n", "- `Vertex AI Feature Store (Legacy)`\n",
"\n", "\n",
"The steps performed include:\n", "The steps performed include:\n",
"\n", "\n",
@@ -87,7 +87,7 @@
"- Import feature data into the `Featurestore` resource.\n", "- Import feature data into the `Featurestore` resource.\n",
"- Serve online prediction requests using the imported features.\n", "- Serve online prediction requests using the imported features.\n",
"- Access imported features in offline jobs, such as training jobs.\n", "- Access imported features in offline jobs, such as training jobs.\n",
"- Use streaming ingestion to ingest small amount of data." "- Use streaming import to import small amounts of data."
] ]
}, },
{ {
@@ -752,7 +752,7 @@
"source": [ "source": [
"## Import feature values\n", "## Import feature values\n",
"\n", "\n",
"You need to import feature values before you can use them for online or offline serving. In this step, you learn how to import feature values by ingesting the values from GCS (Google Cloud Storage). You can also import feature values from BigQuery or a pandas dataFrame.\n" "You need to import feature values before you can use them for online or offline serving. In this step, you learn how to import feature values from Cloud Storage. You can also import feature values from BigQuery or a pandas dataFrame.\n"
] ]
}, },
{ {
@@ -1051,7 +1051,7 @@
"* Labels: the ground-truth data recorded that user X has watched movie Y.\n", "* Labels: the ground-truth data recorded that user X has watched movie Y.\n",
"\n", "\n",
"\n", "\n",
"To be more specific, the ground-truth observation is described in Table 1 and the desired training dataset is described in Table 2. Each row in Table 2 is a result of joining the imported feature values from Vertex AI Feature Store according to the entity IDs and timestamps in Table 1. In this example, the `age`, `gender` and `liked_genres` features from `users` and\n", "To be more specific, the ground-truth observation is described in Table 1 and the desired training dataset is described in Table 2. Each row in Table 2 is a result of joining the imported feature values from Vertex AI Feature Store (Legacy) according to the entity IDs and timestamps in Table 1. In this example, the `age`, `gender` and `liked_genres` features from `users` and\n",
"the `titles`, `genres` and `average_rating` features from `movies` are chosen to train the model. Note that only positive examples are shown in these 2 tables, that is, you can imagine there is a label column whose values are all `True`.\n", "the `titles`, `genres` and `average_rating` features from `movies` are chosen to train the model. Note that only positive examples are shown in these 2 tables, that is, you can imagine there is a label column whose values are all `True`.\n",
"\n", "\n",
"[`batch_serve_to_bq`](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/featurestore.py#L770) takes Table 1 as\n", "[`batch_serve_to_bq`](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/featurestore.py#L770) takes Table 1 as\n",
@@ -1216,11 +1216,9 @@
"id": "7190f3c8b625" "id": "7190f3c8b625"
}, },
"source": [ "source": [
"## Streaming ingestion\n", "## Streaming import\n",
"\n", "\n",
"Streaming ingestion is currently public preview.\n", "Streaming import lets you make real-time updates to feature values. While batch import is suitable for importing a large volume of data with high latency, streaming import is suitable for importing small amount of data with low latency. The written data becomes available to read using batch export and online serving."
"\n",
"Streaming ingestion lets you make real-time updates to feature values. While batch import is suitable for importing a large volume of data with high latency, streaming ingestion is suitable for ingesting small amount of data with low latency. The written data becomes available to read using batch export and online serving."
] ]
}, },
{ {
@@ -1231,7 +1229,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Since streaming ingestion is public preview, the feature is available in aiplatform_v1beta1.\n", "# Since streaming import is public preview, the feature is available in aiplatform_v1beta1.\n",
"from google.cloud.aiplatform_v1beta1 import (\n", "from google.cloud.aiplatform_v1beta1 import (\n",
" FeaturestoreOnlineServingServiceClient, FeaturestoreServiceClient)\n", " FeaturestoreOnlineServingServiceClient, FeaturestoreServiceClient)\n",
"from google.cloud.aiplatform_v1beta1.types import \\\n", "from google.cloud.aiplatform_v1beta1.types import \\\n",
@@ -1254,7 +1252,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Call `write_feature_values` to ingest data to `users` entity type.\n", "# Call `write_feature_values` to import data to the `users` entity type.\n",
"data_client.write_feature_values(\n", "data_client.write_feature_values(\n",
" entity_type=admin_client.entity_type_path(\n", " entity_type=admin_client.entity_type_path(\n",
" PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n", " PROJECT_ID, REGION, FEATURESTORE_ID, \"users\"\n",
@@ -1297,7 +1295,8 @@
}, },
"source": [ "source": [
"Upon successful completion, the `write_feature_values` API returns an empty response.\n", "Upon successful completion, the `write_feature_values` API returns an empty response.\n",
"Similarly, ingest data to the `movies` entity type." "\n",
"Similarly, import data to the `movies` entity type."
] ]
}, },
{ {