diff --git a/notebooks/community/CODEOWNERS b/notebooks/community/CODEOWNERS index 4b678b90e..2579b8beb 100644 --- a/notebooks/community/CODEOWNERS +++ b/notebooks/community/CODEOWNERS @@ -10,11 +10,12 @@ /structured_data/rapid_prototyping_* @rafael-carvalho /managed_notebooks/ @notebooks-team -/sdk/SDK_FBProphet_Forecasting_Online.ipynb @brianchunkang +/sdk/SDK_FBProphet_Forecasting_Online.ipynb @brianchunkang /pipelines/google_cloud_pipeline_components_TPU_model_train_upload_deploy.ipynb @brianchunkang /sdk/SDK_AutoML_Forecasting_Model_Training_Example.ipynb @thehardikv /sdk/sdk_automl_forecasting_evaluating_a_model.ipynb @thehardikv -/matching_engine @yinghsienwu +/matching_engine/sdk_matching_engine_for_indexing.ipynb @ivanmkc +/matching_engine/matching_engine_for_indexing.ipynb @yinghsienwu /neo4j @benofben @htappen /sdk/pytorch_lightning_custom_container_training.ipynb @brianchunkang /tensorboard @yfang1 @wattli diff --git a/notebooks/community/matching_engine/sdk_matching_engine_for_indexing.ipynb b/notebooks/community/matching_engine/sdk_matching_engine_for_indexing.ipynb new file mode 100644 index 000000000..ecd5f3905 --- /dev/null +++ b/notebooks/community/matching_engine/sdk_matching_engine_for_indexing.ipynb @@ -0,0 +1,1485 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2022 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e" + }, + "source": [ + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Vertex\n", + " Run in Vertex Workbench\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "This example demonstrates how to use the GCP ANN Service. It is a high scale, low latency solution, to find similar vectors (or more specifically \"embeddings\") for a large corpus. Moreover, it is a fully managed offering, further reducing operational overhead. It is built upon [Approximate Nearest Neighbor (ANN) technology](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) developed by Google Research.\n", + "\n", + "### Dataset\n", + "\n", + "The dataset used for this tutorial is the [GloVe dataset](https://nlp.stanford.edu/projects/glove/).\n", + "\n", + "\"GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.\"\n", + "\n", + "### Objective\n", + "\n", + "In this notebook, you will learn how to create Approximate Nearest Neighbor (ANN) Index, query against indexes, and validate the performance of the index. \n", + "\n", + "The steps performed include:\n", + "\n", + "* Create ANN Index and Brute Force Index\n", + "* Create an IndexEndpoint with VPC Network\n", + "* Deploy ANN Index and Brute Force Index\n", + "* Perform online query\n", + "* Compute recall\n", + "\n", + "\n", + "### Costs \n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "S5zc4kbEiYCm" + }, + "source": [ + "## Before you begin\n", + "\n", + "* **Prepare a VPC network**. To reduce any network overhead that might lead to unnecessary increase in overhead latency, it is best to call the ANN endpoints from your VPC via a direct [VPC Peering](https://cloud.google.com/vertex-ai/docs/general/vpc-peering) connection. The following section describes how to setup a VPC Peering connection if you don't have one. This is a one-time initial setup task. You can also reuse existing VPC network and skip this section.\n", + "* **WARNING:** The match service gRPC API (to create online queries against your deployed index) has to be executed in a Google Cloud Notebook instance that is created with the following requirements:\n", + " * **In the same region as where your ANN service is deployed** (for example, if you set `REGION = \"us-central1\"` as same as the tutorial, the notebook instance has to be in `us-central1`).\n", + " * **Make sure you select the VPC network you created for ANN service** (instead of using the \"default\" one). That is, you will have to create the VPC network below and then create a new notebook instance that uses that VPC. \n", + " * If you run it in the colab or a Google Cloud Notebook instance in a different VPC network or region, the gRPC API will fail to peer the network (InactiveRPCError)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KDH8CgQiSxhv" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"\" # @param {type:\"string\"}\n", + "\n", + "NETWORK_NAME = \"ucaip-haystack-vpc-network\" # @param {type:\"string\"}\n", + "\n", + "PEERING_RANGE_NAME = \"ucaip-haystack-range\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lW2LneA5mmmP" + }, + "outputs": [], + "source": [ + "# Create a VPC network\n", + "! gcloud compute networks create {NETWORK_NAME} --bgp-routing-mode=regional --subnet-mode=auto --project={PROJECT_ID}\n", + "\n", + "# Add necessary firewall rules\n", + "! gcloud compute firewall-rules create {NETWORK_NAME}-allow-icmp --network {NETWORK_NAME} --priority 65534 --project {PROJECT_ID} --allow icmp\n", + "\n", + "! gcloud compute firewall-rules create {NETWORK_NAME}-allow-internal --network {NETWORK_NAME} --priority 65534 --project {PROJECT_ID} --allow all --source-ranges 10.128.0.0/9\n", + "\n", + "! gcloud compute firewall-rules create {NETWORK_NAME}-allow-rdp --network {NETWORK_NAME} --priority 65534 --project {PROJECT_ID} --allow tcp:3389\n", + "\n", + "! gcloud compute firewall-rules create {NETWORK_NAME}-allow-ssh --network {NETWORK_NAME} --priority 65534 --project {PROJECT_ID} --allow tcp:22\n", + "\n", + "# Reserve IP range\n", + "! gcloud compute addresses create {PEERING_RANGE_NAME} --global --prefix-length=16 --network={NETWORK_NAME} --purpose=VPC_PEERING --project={PROJECT_ID} --description=\"peering range for uCAIP Haystack.\"\n", + "\n", + "# Set up peering with service networking\n", + "! gcloud services vpc-peerings connect --service=servicenetworking.googleapis.com --network={NETWORK_NAME} --ranges={PEERING_RANGE_NAME} --project={PROJECT_ID}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "d3uj8x73nDX_" + }, + "source": [ + "* Authentication: `$ gcloud auth login` rerun this in Google Cloud Notebook terminal when you are logged out and need the credential again." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i7EUnXsZhAGF" + }, + "source": [ + "### Installation\n", + "\n", + "Download and install the latest (preview) version of the Vertex SDK for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wyy5Lbnzg5fi" + }, + "outputs": [], + "source": [ + "! pip install -U git+https://github.com/ivanmkc/python-aiplatform.git@imkc--matching-engine" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "irSMQn6gZ19l" + }, + "source": [ + "Install the `h5py` to prepare sample dataset, and the `grpcio-tools` for querying against the index. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-h5sqwOEZ5Yq" + }, + "outputs": [], + "source": [ + "! pip install -U grpcio-tools\n", + "! pip install -U h5py" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hhq5zEbGg0XX" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "After you install the additional packages, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EzrelQZ22IZj" + }, + "outputs": [], + "source": [ + "# Automatically restart kernel after installs\n", + "import os\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BF1j6f9HApxa" + }, + "source": [ + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager).\n", + "\n", + "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "1. [Enable the Vertex AI API and Compute Engine API, and Service Networking API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,servicenetworking.googleapis.com).\n", + "\n", + "1. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WReHDGG5g0XY" + }, + "source": [ + "#### Set your project ID\n", + "\n", + "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oM1iC_MfAts1" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "PROJECT_ID = \"\"\n", + "\n", + "# Get your Google Cloud project ID from gcloud\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID: \", PROJECT_ID)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qJYoRfYng0XZ" + }, + "source": [ + "Otherwise, set your project ID here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "riG_qUokg0XZ" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None:\n", + " PROJECT_ID = \"\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "q7tcBkCDI1_M" + }, + "source": [ + "#### Timestamp\n", + "\n", + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HpIK91y1IzDr" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t6Ggbb4DI6by" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebooks**, your environment is already\n", + "authenticated. Skip this step." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RpIzUmpOI9G7" + }, + "source": [ + "**If you are using Colab**, run the cell below and follow the instructions\n", + "when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "1. In the Cloud Console, go to the [**Create service account key**\n", + " page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n", + "\n", + "2. Click **Create service account**.\n", + "\n", + "3. In the **Service account name** field, enter a name, and\n", + " click **Create**.\n", + "\n", + "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n", + "into the filter box, and select\n", + " **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "5. Click *Create*. A JSON file that contains your key downloads to your\n", + "local environment.\n", + "\n", + "6. Enter the path to your service account key as the\n", + "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AW9vQHeoI-q_" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your GCP account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "# The Google Cloud Notebook product has specific requirements\n", + "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n", + "\n", + "# If on Google Cloud Notebooks, then don't execute this code\n", + "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, log in using gcloud\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " ! gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zgPO1eR3CYjk" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "Set the name of your Cloud Storage bucket below. It must be unique across all\n", + "Cloud Storage buckets.\n", + "\n", + "You may also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Make sure to [choose a region where Vertex AI services are\n", + "available](https://cloud.google.com/vertex-ai/docs/general/locations#available_regions). You may\n", + "not use a Multi-Regional Storage bucket for training with Vertex AI." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MzGDU7TWdts_" + }, + "outputs": [], + "source": [ + "BUCKET_URI = \"gs://[your-bucket-name]\" # @param {type:\"string\"}\n", + "REGION = \"[your-region]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cf221059d072" + }, + "outputs": [], + "source": [ + "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n", + " BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP\n", + "\n", + "if REGION == \"[your-region]\":\n", + " REGION = \"us-central1\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-EcIXiGsCePi" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NIq7R4HZCfIc" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ucvCsknMCims" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "vhOb7YnwClBb" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XoEqT2Y4DJmf" + }, + "source": [ + "### Import libraries and define constants" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Y9Uo3tifg1kx" + }, + "source": [ + "Import the Vertex AI (unified) client library into your Python environment. \n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f2d05ab4126a" + }, + "outputs": [], + "source": [ + "import h5py" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pRUOFELefqf1" + }, + "outputs": [], + "source": [ + "PROJECT_NUMBER = !gcloud projects list --filter=\"PROJECT_ID:'{PROJECT_ID}'\" --format='value(PROJECT_NUMBER)'\n", + "PROJECT_NUMBER = PROJECT_NUMBER[0]\n", + "\n", + "PARENT = \"projects/{}/locations/{}\".format(PROJECT_ID, REGION)\n", + "\n", + "print(\"PROJECT_ID: {}\".format(PROJECT_ID))\n", + "print(\"REGION: {}\".format(REGION))\n", + "\n", + "!gcloud config set project {PROJECT_ID}\n", + "!gcloud config set ai_platform/region {REGION}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lR6Wwv-hCCN-" + }, + "source": [ + "## Prepare the Data\n", + "\n", + "The GloVe dataset consists of a set of pre-trained embeddings. The embeddings are split into a \"train\" split, and a \"test\" split.\n", + "We will create a vector search index from the \"train\" split, and use the embedding vectors in the \"test\" split as query vectors to test the vector search index.\n", + "\n", + "NOTE: While the data split uses the term \"train\", these are pre-trained embeddings and thus are ready to be indexed for search. The terms \"train\" and \"test\" split are used just to be consistent with usual machine learning terminology.\n", + "\n", + "Download the GloVe dataset.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9wzS85TeB9dG" + }, + "outputs": [], + "source": [ + "! gsutil cp gs://cloud-samples-data/vertex-ai/matching_engine/glove-100-angular.hdf5 ." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4fAO9CMoCNtq" + }, + "source": [ + "Read the data into memory.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lZ3JQTS6CN-3" + }, + "outputs": [], + "source": [ + "# The number of nearest neighbors to be retrieved from database for each query.\n", + "NUM_NEIGHBOURS = 10\n", + "\n", + "h5 = h5py.File(\"glove-100-angular.hdf5\", \"r\")\n", + "train = h5[\"train\"]\n", + "test = h5[\"test\"]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pE6bBBo7GjJK" + }, + "outputs": [], + "source": [ + "train[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aQIQSyF9GtSv" + }, + "source": [ + "Save the train split in JSONL format.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "18wCiTwfG40P" + }, + "outputs": [], + "source": [ + "with open(\"glove100.json\", \"w\") as f:\n", + " for i in range(len(train)):\n", + " f.write('{\"id\":\"' + str(i) + '\",')\n", + " f.write('\"embedding\":[' + \",\".join(str(x) for x in train[i]) + \"]}\")\n", + " f.write(\"\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QuVl8DrWG8NS" + }, + "source": [ + "Upload the training data to GCS." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3PgsA_vbI8Vg" + }, + "outputs": [], + "source": [ + "EMBEDDINGS_INITIAL_URI = f\"{BUCKET_URI}/matching_engine/initial/\"\n", + "! gsutil cp glove100.json {EMBEDDINGS_INITIAL_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mglUPwHpJH98" + }, + "source": [ + "## Create Indexes\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qhIBCQ7dDSbW" + }, + "source": [ + "### Create ANN Index (for Production Usage)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qiIg9b5zJLi1" + }, + "outputs": [], + "source": [ + "DIMENSIONS = 100\n", + "DISPLAY_NAME = \"glove_100_1\"\n", + "DISPLAY_NAME_BRUTE_FORCE = DISPLAY_NAME + \"_brute_force\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "svLYiDf0OD2G" + }, + "source": [ + "Create the ANN index configuration:\n", + "\n", + "Please read the documentation to understand the various configuration parameters that can be used to tune the index\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y4zooldkGoM4" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xzY7TpUSJcTV" + }, + "outputs": [], + "source": [ + "tree_ah_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(\n", + " display_name=DISPLAY_NAME,\n", + " contents_delta_uri=EMBEDDINGS_INITIAL_URI,\n", + " dimensions=DIMENSIONS,\n", + " approximate_neighbors_count=150,\n", + " distance_measure_type=\"DOT_PRODUCT_DISTANCE\",\n", + " leaf_node_embedding_count=500,\n", + " leaf_nodes_to_search_percent=7,\n", + " description=\"Glove 100 ANN index\",\n", + " labels={\"label_name\": \"label_value\"},\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "17jrQi501QyX" + }, + "outputs": [], + "source": [ + "INDEX_RESOURCE_NAME = tree_ah_index.resource_name\n", + "INDEX_RESOURCE_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kSsqZuyoA1SG" + }, + "source": [ + "### Create Brute Force Index (for Ground Truth)\n", + "\n", + "The brute force index uses a naive brute force method to find the nearest neighbors. This method is not fast or efficient. Hence brute force indices are not recommended for production usage. They are to be used to find the \"ground truth\" set of neighbors, so that the \"ground truth\" set can be used to measure recall of the indices being tuned for production usage. To ensure an apples to apples comparison, the `distanceMeasureType` and `featureNormType`, `dimensions` of the brute force index should match those of the production indices being tuned.\n", + "\n", + "Create the brute force index configuration:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DXnBLqjXBsv8" + }, + "outputs": [], + "source": [ + "brute_force_index = aiplatform.MatchingEngineIndex.create_brute_force_index(\n", + " display_name=DISPLAY_NAME,\n", + " contents_delta_uri=EMBEDDINGS_INITIAL_URI,\n", + " dimensions=DIMENSIONS,\n", + " approximate_neighbors_count=150,\n", + " distance_measure_type=\"DOT_PRODUCT_DISTANCE\",\n", + " description=\"Glove 100 index (brute force)\",\n", + " labels={\"label_name\": \"label_value\"},\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_oD5SieYJbbW" + }, + "outputs": [], + "source": [ + "INDEX_BRUTE_FORCE_RESOURCE_NAME = brute_force_index.resource_name\n", + "INDEX_BRUTE_FORCE_RESOURCE_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "omlgEZ-sGoM5" + }, + "source": [ + "## Update Indexes\n", + "\n", + "Create incremental data file.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DDAvm_mj_BVs" + }, + "outputs": [], + "source": [ + "with open(\"glove100_incremental.json\", \"w\") as f:\n", + " f.write(\n", + " '{\"id\":\"0\",\"embedding\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}\\n'\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZU7TU7C7GoM6" + }, + "source": [ + "Copy the incremental data file to a new subdirectory.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RLWcDvNLGoM6" + }, + "outputs": [], + "source": [ + "EMBEDDINGS_UPDATE_URI = f\"{BUCKET_URI}/matching-engine/incremental/\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FgpEDX0oGoM6" + }, + "outputs": [], + "source": [ + "! gsutil cp glove100_incremental.json {EMBEDDINGS_UPDATE_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aiXtF_x0GoM6" + }, + "source": [ + "Create update index request\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tvedBONtGoM6" + }, + "outputs": [], + "source": [ + "tree_ah_index = tree_ah_index.update_embeddings(\n", + " contents_delta_uri=EMBEDDINGS_UPDATE_URI,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HKPDojFpGoM6" + }, + "outputs": [], + "source": [ + "INDEX_RESOURCE_NAME = tree_ah_index.resource_name\n", + "INDEX_RESOURCE_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qV2xjAnDDObD" + }, + "source": [ + "## Create an IndexEndpoint with VPC Network" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BpZQoJyxDlbO" + }, + "outputs": [], + "source": [ + "VPC_NETWORK_NAME = \"projects/{}/global/networks/{}\".format(PROJECT_NUMBER, NETWORK_NAME)\n", + "VPC_NETWORK_NAME" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QuARXzJVGyQX" + }, + "outputs": [], + "source": [ + "my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(\n", + " display_name=\"index_endpoint_for_demo\",\n", + " description=\"index endpoint description\",\n", + " network=VPC_NETWORK_NAME,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0N0R2reAGoM7" + }, + "outputs": [], + "source": [ + "my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint(\n", + " index_endpoint_name=\"projects/1012616486416/locations/us-central1/indexEndpoints/1370185002255384576\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PJ3bcZqi-cfM" + }, + "outputs": [], + "source": [ + "INDEX_ENDPOINT_NAME = my_index_endpoint.resource_name\n", + "INDEX_ENDPOINT_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "np2cgVuuIe9k" + }, + "source": [ + "## Deploy Indexes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8Ew1UgcIIiJG" + }, + "source": [ + "### Deploy ANN Index" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nLOYTGygIlMK" + }, + "outputs": [], + "source": [ + "DEPLOYED_INDEX_ID = \"tree_ah_glove_deployed\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_uK4WOgqN1NG" + }, + "outputs": [], + "source": [ + "my_index_endpoint = my_index_endpoint.deploy_index(\n", + " index=tree_ah_index, deployed_index_id=DEPLOYED_INDEX_ID\n", + ")\n", + "\n", + "my_index_endpoint.deployed_indexes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RNZnXmO5AhDO" + }, + "source": [ + "### Deploy Brute Force Index" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3p9e4828AkSv" + }, + "outputs": [], + "source": [ + "DEPLOYED_BRUTE_FORCE_INDEX_ID = \"glove_brute_force_deployed\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-2kgd01SA4rk" + }, + "outputs": [], + "source": [ + "my_index_endpoint = my_index_endpoint.deploy_index(\n", + " index=brute_force_index, deployed_index_id=DEPLOYED_BRUTE_FORCE_INDEX_ID\n", + ")\n", + "\n", + "my_index_endpoint.deployed_indexes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6LCGvBNvBd8D" + }, + "source": [ + "## Create Online Queries\n", + "\n", + "After you built your indexes, you may query against the deployed index through the online querying gRPC API (Match service) within the virtual machine instances from the same region (for example 'us-central1' in this tutorial). " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IcXa9lSuB9AT" + }, + "source": [ + "Test your query:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "A3KYVw5HB-4v" + }, + "outputs": [], + "source": [ + "# Test query\n", + "query = [\n", + " -0.11333,\n", + " 0.48402,\n", + " 0.090771,\n", + " -0.22439,\n", + " 0.034206,\n", + " -0.55831,\n", + " 0.041849,\n", + " -0.53573,\n", + " 0.18809,\n", + " -0.58722,\n", + " 0.015313,\n", + " -0.014555,\n", + " 0.80842,\n", + " -0.038519,\n", + " 0.75348,\n", + " 0.70502,\n", + " -0.17863,\n", + " 0.3222,\n", + " 0.67575,\n", + " 0.67198,\n", + " 0.26044,\n", + " 0.4187,\n", + " -0.34122,\n", + " 0.2286,\n", + " -0.53529,\n", + " 1.2582,\n", + " -0.091543,\n", + " 0.19716,\n", + " -0.037454,\n", + " -0.3336,\n", + " 0.31399,\n", + " 0.36488,\n", + " 0.71263,\n", + " 0.1307,\n", + " -0.24654,\n", + " -0.52445,\n", + " -0.036091,\n", + " 0.55068,\n", + " 0.10017,\n", + " 0.48095,\n", + " 0.71104,\n", + " -0.053462,\n", + " 0.22325,\n", + " 0.30917,\n", + " -0.39926,\n", + " 0.036634,\n", + " -0.35431,\n", + " -0.42795,\n", + " 0.46444,\n", + " 0.25586,\n", + " 0.68257,\n", + " -0.20821,\n", + " 0.38433,\n", + " 0.055773,\n", + " -0.2539,\n", + " -0.20804,\n", + " 0.52522,\n", + " -0.11399,\n", + " -0.3253,\n", + " -0.44104,\n", + " 0.17528,\n", + " 0.62255,\n", + " 0.50237,\n", + " -0.7607,\n", + " -0.071786,\n", + " 0.0080131,\n", + " -0.13286,\n", + " 0.50097,\n", + " 0.18824,\n", + " -0.54722,\n", + " -0.42664,\n", + " 0.4292,\n", + " 0.14877,\n", + " -0.0072514,\n", + " -0.16484,\n", + " -0.059798,\n", + " 0.9895,\n", + " -0.61738,\n", + " 0.054169,\n", + " 0.48424,\n", + " -0.35084,\n", + " -0.27053,\n", + " 0.37829,\n", + " 0.11503,\n", + " -0.39613,\n", + " 0.24266,\n", + " 0.39147,\n", + " -0.075256,\n", + " 0.65093,\n", + " -0.20822,\n", + " -0.17456,\n", + " 0.53571,\n", + " -0.16537,\n", + " 0.13582,\n", + " -0.56016,\n", + " 0.016964,\n", + " 0.1277,\n", + " 0.94071,\n", + " -0.22608,\n", + " -0.021106,\n", + "]\n", + "\n", + "response = my_index_endpoint.match(\n", + " deployed_index_id=DEPLOYED_INDEX_ID, queries=[query], num_neighbors=NUM_NEIGHBOURS\n", + ")\n", + "\n", + "response" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_mNwdU9_B_Ez" + }, + "source": [ + "### Batch Query\n", + "\n", + "You can run multiple queries in a single match call:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "A0XL0PJ1GoM9" + }, + "outputs": [], + "source": [ + "# Test query\n", + "queries = [\n", + " [\n", + " -0.11333,\n", + " 0.48402,\n", + " 0.090771,\n", + " -0.22439,\n", + " 0.034206,\n", + " -0.55831,\n", + " 0.041849,\n", + " -0.53573,\n", + " 0.18809,\n", + " -0.58722,\n", + " 0.015313,\n", + " -0.014555,\n", + " 0.80842,\n", + " -0.038519,\n", + " 0.75348,\n", + " 0.70502,\n", + " -0.17863,\n", + " 0.3222,\n", + " 0.67575,\n", + " 0.67198,\n", + " 0.26044,\n", + " 0.4187,\n", + " -0.34122,\n", + " 0.2286,\n", + " -0.53529,\n", + " 1.2582,\n", + " -0.091543,\n", + " 0.19716,\n", + " -0.037454,\n", + " -0.3336,\n", + " 0.31399,\n", + " 0.36488,\n", + " 0.71263,\n", + " 0.1307,\n", + " -0.24654,\n", + " -0.52445,\n", + " -0.036091,\n", + " 0.55068,\n", + " 0.10017,\n", + " 0.48095,\n", + " 0.71104,\n", + " -0.053462,\n", + " 0.22325,\n", + " 0.30917,\n", + " -0.39926,\n", + " 0.036634,\n", + " -0.35431,\n", + " -0.42795,\n", + " 0.46444,\n", + " 0.25586,\n", + " 0.68257,\n", + " -0.20821,\n", + " 0.38433,\n", + " 0.055773,\n", + " -0.2539,\n", + " -0.20804,\n", + " 0.52522,\n", + " -0.11399,\n", + " -0.3253,\n", + " -0.44104,\n", + " 0.17528,\n", + " 0.62255,\n", + " 0.50237,\n", + " -0.7607,\n", + " -0.071786,\n", + " 0.0080131,\n", + " -0.13286,\n", + " 0.50097,\n", + " 0.18824,\n", + " -0.54722,\n", + " -0.42664,\n", + " 0.4292,\n", + " 0.14877,\n", + " -0.0072514,\n", + " -0.16484,\n", + " -0.059798,\n", + " 0.9895,\n", + " -0.61738,\n", + " 0.054169,\n", + " 0.48424,\n", + " -0.35084,\n", + " -0.27053,\n", + " 0.37829,\n", + " 0.11503,\n", + " -0.39613,\n", + " 0.24266,\n", + " 0.39147,\n", + " -0.075256,\n", + " 0.65093,\n", + " -0.20822,\n", + " -0.17456,\n", + " 0.53571,\n", + " -0.16537,\n", + " 0.13582,\n", + " -0.56016,\n", + " 0.016964,\n", + " 0.1277,\n", + " 0.94071,\n", + " -0.22608,\n", + " -0.021106,\n", + " ],\n", + " [\n", + " -0.99544,\n", + " -2.3651,\n", + " -0.24332,\n", + " -1.0321,\n", + " 0.42052,\n", + " -1.1817,\n", + " -0.16451,\n", + " -1.683,\n", + " 0.49673,\n", + " -0.27258,\n", + " -0.025397,\n", + " 0.34188,\n", + " 1.5523,\n", + " 1.3532,\n", + " 0.33297,\n", + " -0.0056677,\n", + " -0.76525,\n", + " 0.49587,\n", + " 1.2211,\n", + " 0.83394,\n", + " -0.20031,\n", + " -0.59657,\n", + " 0.38485,\n", + " -0.23487,\n", + " -1.0725,\n", + " 0.95856,\n", + " 0.16161,\n", + " -1.2496,\n", + " 1.6751,\n", + " 0.73899,\n", + " 0.051347,\n", + " -0.42702,\n", + " 0.16257,\n", + " -0.16772,\n", + " 0.40146,\n", + " 0.29837,\n", + " 0.96204,\n", + " -0.36232,\n", + " -0.47848,\n", + " 0.78278,\n", + " 0.14834,\n", + " 1.3407,\n", + " 0.47834,\n", + " -0.39083,\n", + " -1.037,\n", + " -0.24643,\n", + " -0.75841,\n", + " 0.7669,\n", + " -0.37363,\n", + " 0.52741,\n", + " 0.018563,\n", + " -0.51301,\n", + " 0.97674,\n", + " 0.55232,\n", + " 1.1584,\n", + " 0.73715,\n", + " 1.3055,\n", + " -0.44743,\n", + " -0.15961,\n", + " 0.85006,\n", + " -0.34092,\n", + " -0.67667,\n", + " 0.2317,\n", + " 1.5582,\n", + " 1.2308,\n", + " -0.62213,\n", + " -0.032801,\n", + " 0.1206,\n", + " -0.25899,\n", + " -0.02756,\n", + " -0.52814,\n", + " -0.93523,\n", + " 0.58434,\n", + " -0.24799,\n", + " 0.37692,\n", + " 0.86527,\n", + " 0.069626,\n", + " 1.3096,\n", + " 0.29975,\n", + " -1.3651,\n", + " -0.32048,\n", + " -0.13741,\n", + " 0.33329,\n", + " -1.9113,\n", + " -0.60222,\n", + " -0.23921,\n", + " 0.12664,\n", + " -0.47961,\n", + " -0.89531,\n", + " 0.62054,\n", + " 0.40869,\n", + " -0.08503,\n", + " 0.6413,\n", + " -0.84044,\n", + " -0.74325,\n", + " -0.19426,\n", + " 0.098722,\n", + " 0.32648,\n", + " -0.67621,\n", + " -0.62692,\n", + " ],\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xeUZO3bAGoM-" + }, + "source": [ + "### Compute Recall\n", + "\n", + "Use deployed brute force Index as the ground truth to calculate the recall of ANN Index:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "U9dNIbkEGoM-" + }, + "outputs": [], + "source": [ + "# Retrieve nearest neighbors for both the tree-AH index and the brute-force index\n", + "tree_ah_response_test = my_index_endpoint.match(\n", + " deployed_index_id=DEPLOYED_INDEX_ID,\n", + " queries=list(test),\n", + " num_neighbors=NUM_NEIGHBOURS,\n", + ")\n", + "brute_force_response_test = my_index_endpoint.match(\n", + " deployed_index_id=DEPLOYED_BRUTE_FORCE_INDEX_ID,\n", + " queries=list(test),\n", + " num_neighbors=NUM_NEIGHBOURS,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "V-eMF05UGoM-" + }, + "outputs": [], + "source": [ + "# Calculate recall by determining how many neighbors were correctly retrieved as compared to the brute-force option.\n", + "correct_neighbors = 0\n", + "for tree_ah_neighbors, brute_force_neighbors in zip(\n", + " tree_ah_response_test, brute_force_response_test\n", + "):\n", + " tree_ah_neighbor_ids = [neighbor.id for neighbor in tree_ah_neighbors]\n", + " brute_force_neighbor_ids = [neighbor.id for neighbor in brute_force_neighbors]\n", + "\n", + " correct_neighbors += len(\n", + " set(tree_ah_neighbor_ids).intersection(brute_force_neighbor_ids)\n", + " )\n", + "\n", + "recall = correct_neighbors / (len(test) * NUM_NEIGHBOURS)\n", + "\n", + "print(\"Recall: {}\".format(recall))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TpV-iwP9qw9c" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "You can also manually delete resources that you created by running the following code." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sx_vKniMq9ZX" + }, + "outputs": [], + "source": [ + "# Force undeployment of indexes and delete endpoint\n", + "my_index_endpoint.delete(force=True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "omj7N9iWv-Tq" + }, + "outputs": [], + "source": [ + "# Delete indexes\n", + "tree_ah_index.delete(force=True)\n", + "brute_force_index.delete(force=True)" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "sdk_matching_engine_for_indexing.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}