Compare commits

...
Author SHA1 Message Date
Andrew FerlitschandGitHub 79328b0ab5 Merge branch 'main' into official_fix14 2022-09-28 14:12:21 -07:00
Andrew Ferlitsch ccd518dd03 fix: link, title, format 2022-09-28 17:57:25 +00:00
Andrew Ferlitsch a8e95d37ca fix: link, title, format 2022-09-28 17:56:58 +00:00
@@ -29,12 +29,12 @@
"id": "JAPoU8Sm5E6e"
},
"source": [
"<table align=\"left\">\n",
"# Create Vertex AI Matching Engine index\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Run in Vertex Workbench\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
@@ -43,25 +43,32 @@
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Run in Vertex Workbench\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
"id": "b0a74aaf1481"
},
"source": [
"## Overview\n",
"\n",
"This example demonstrates how to use the GCP ANN Service. It is a high scale, low latency solution, to find similar vectors (or more specifically \"embeddings\") for a large corpus. Moreover, it is a fully managed offering, further reducing operational overhead. It is built upon [Approximate Nearest Neighbor (ANN) technology](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) developed by Google Research.\n",
"\n",
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [GloVe dataset](https://nlp.stanford.edu/projects/glove/).\n",
"\n",
"\"GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.\"\n",
"\n",
"This example demonstrates how to use the GCP ANN Service. It is a high scale, low latency solution, to find similar vectors (or more specifically \"embeddings\") for a large corpus. Moreover, it is a fully managed offering, further reducing operational overhead. It is built upon [Approximate Nearest Neighbor (ANN) technology](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) developed by Google Research."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "34a4b245e795"
},
"source": [
"### Objective\n",
"\n",
"In this notebook, you will learn how to create Approximate Nearest Neighbor (ANN) Index, query against indexes, and validate the performance of the index. \n",
@@ -72,9 +79,28 @@
"* Create an IndexEndpoint with VPC Network\n",
"* Deploy ANN Index and Brute Force Index\n",
"* Perform online query\n",
"* Compute recall\n",
"* Compute recall\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [GloVe dataset](https://nlp.stanford.edu/projects/glove/).\n",
"\n",
"\"GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5e2eba58ad71"
},
"source": [
"### Costs \n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
@@ -89,21 +115,14 @@
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S5zc4kbEiYCm"
},
"source": [
"## Before you begin"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d1e95a984673"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
@@ -168,18 +187,7 @@
"metadata": {
"id": "3e2b43c2d2bf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your browser has been opened to visit:\n",
"\n",
" https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=UY9jjYfhoSedWWUOWXp5Pmicq0Ic04&access_type=offline&code_challenge=OQefcewSwkT7ZwfzzOVidtngvZspdY1NgN6rltw8x7A&code_challenge_method=S256\n",
"\n"
]
}
],
"outputs": [],
"source": [
"import os\n",
"import sys\n",
@@ -219,19 +227,11 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"id": "beb72f394541"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Project ID: python-docs-samples-tests\n"
]
}
],
"outputs": [],
"source": [
"import os\n",
"\n",