xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: clean and rebrand - permission (#2335)

Browse Source 
* fix: clean and rebrand

* fix: VPC setup

* rebrand

* rebrand

* fix: PROJECTT

* fix: VPC network

* fix: redis

* fix: import os
This commit is contained in:
Andrew Ferlitsch
2023-10-25 21:20:59 +00:00
committed by GitHub
parent d0037863ff
commit 8dad9cd13f
1 changed files with 189 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/matching_engine/sdk_matching_engine_create_stack_overflow_embeddings.ipynb
+189 -223
View File
@@ -24,13 +24,12 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Using Vertex AI Matching Engine for StackOverflow Questions\n",
"# Using Vertex AI Vector Search for StackOverflow Questions\n",
"![ ](https://www.google-analytics.com/collect?v=2&tid=G-L6X3ECH596&cid=1&en=page_view&sid=1&dt=sdk_matching_engine_create_stack_overflow_embeddings.ipynb&dl=notebooks%2Fofficial%2Fmatching_engine%2Fsdk_matching_engine_create_stack_overflow_embeddings.ipynb)\n",
"<table align=\"left\">\n",
" <td>\n",
@@ -61,11 +60,11 @@
"source": [
"## Overview\n",
"\n",
"This example demonstrates how to encode custom text embeddings using the StackOverflow dataset and the sentence-T5 model. These are uploaded to the Vertex AI Matching Engine service. It is a high scale, low latency solution, to find similar vectors (or more specifically \"embeddings\") for a large corpus. Moreover, it is a fully managed offering, further reducing operational overhead. It is built upon [Approximate Nearest Neighbor (ANN) technology](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) developed by Google Research.\n",
"This example demonstrates how to encode custom text embeddings using the StackOverflow dataset and the sentence-T5 model. These are uploaded to the Vertex AI Vector Search service. It is a high scale, low latency solution, to find similar vectors (or more specifically \"embeddings\") for a large corpus. Moreover, it is a fully managed offering, further reducing operational overhead. It is built upon [Approximate Nearest Neighbor (ANN) technology](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) developed by Google Research.\n",
"\n",
"**Pre-requisite**: This notebook requires you to already have a VPC network set up. See the \"Prepare a VPC network\" section in [Create Vertex AI Matching Engine index notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb).\n",
"**Pre-requisite**: This notebook requires you to already have a VPC network set up. See the \"Prepare a VPC network\" section in [Create Vertex AI Vector Search index notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb).\n",
"\n",
"Learn more about [Vertex AI Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview)."
"Learn more about [Vertex AI Vector Search](https://cloud.google.com/vertex-ai/docs/matching-engine/overview)."
]
},
{
@@ -80,7 +79,7 @@
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Matching Engine`\n",
"- `Vertex AI Vector Search`\n",
"\n",
"The steps performed include:\n",
"\n",
@@ -124,8 +123,8 @@
"source": [
"# Install the packages\n",
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
" google-cloud-storage \\\n",
" 'google-cloud-bigquery[pandas]'"
" google-cloud-storage \\\n",
" 'google-cloud-bigquery[pandas]'"
]
},
{
@@ -147,12 +146,11 @@
"source": [
"# Install the packages\n",
"! pip3 install --upgrade tensorflow \\\n",
" tensorflow_text \\\n",
" tensorflow-hub"
" tensorflow_text \\\n",
" tensorflow-hub"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "54ac7ebac10b"
@@ -220,7 +218,7 @@
},
"outputs": [],
"source": [
"PROJECT_ID = \"[YOUR-PROJECT-ID]\"\n",
"PROJECT_ID = \"[your-project-id]\"\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
@@ -239,7 +237,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {
"id": "474be5183c27"
},
@@ -280,7 +278,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {
"id": "fbc9cd30cc4b"
},
@@ -300,7 +298,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {
"id": "a336a05c6149"
},
@@ -340,7 +338,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"id": "EzrelQZ22IZj"
},
@@ -366,13 +364,13 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"metadata": {
"id": "MzGDU7TWdts_"
},
"outputs": [],
"source": [
"BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}"
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -392,7 +390,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -437,125 +435,11 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"id": "b43937b6065d"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>title</th>\n",
" <th>body</th>\n",
" <th>tags</th>\n",
" <th>answers</th>\n",
" <th>score</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>28819470</td>\n",
" <td>Numbers of Day in Month</td>\n",
" <td>&lt;p&gt;I have a data frame with a date time index,...</td>\n",
" <td>python|pandas</td>\n",
" <td>&lt;p&gt;There is now a &lt;a href=\"https://pandas.pyda...</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>35064304</td>\n",
" <td>\"RuntimeError: Make sure the Graphviz executab...</td>\n",
" <td>&lt;p&gt;I downloaded &lt;code&gt;Graphviz 2.38&lt;/code&gt; MSI...</td>\n",
" <td>python|installation|graphviz</td>\n",
" <td>&lt;p&gt;&lt;strong&gt;Step 1:&lt;/strong&gt; Install Graphviz b...</td>\n",
" <td>22</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>18783390</td>\n",
" <td>python pip specify a library directory and an ...</td>\n",
" <td>&lt;p&gt;I am using pip and trying to install a pyth...</td>\n",
" <td>python|shared-libraries|pip|include-path|pyodbc</td>\n",
" <td>&lt;p&gt;Another way to indicate the location of inc...</td>\n",
" <td>19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1066933</td>\n",
" <td>How to extract top-level domain name (TLD) fro...</td>\n",
" <td>&lt;p&gt;how would you extract the domain name from ...</td>\n",
" <td>python|url|parsing|dns|extract</td>\n",
" <td>&lt;p&gt;No, there is no \"intrinsic\" way of knowing ...</td>\n",
" <td>52</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3817529</td>\n",
" <td>syntax for creating a dictionary into another ...</td>\n",
" <td>&lt;blockquote&gt;\\n &lt;p&gt;&lt;strong&gt;Possible Duplicate:...</td>\n",
" <td>python</td>\n",
" <td>&lt;p&gt;You can declare a dictionary inside a dicti...</td>\n",
" <td>106</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id title \\\n",
"0 28819470 Numbers of Day in Month \n",
"1 35064304 \"RuntimeError: Make sure the Graphviz executab... \n",
"2 18783390 python pip specify a library directory and an ... \n",
"3 1066933 How to extract top-level domain name (TLD) fro... \n",
"4 3817529 syntax for creating a dictionary into another ... \n",
"\n",
" body \\\n",
"0 <p>I have a data frame with a date time index,... \n",
"1 <p>I downloaded <code>Graphviz 2.38</code> MSI... \n",
"2 <p>I am using pip and trying to install a pyth... \n",
"3 <p>how would you extract the domain name from ... \n",
"4 <blockquote>\\n <p><strong>Possible Duplicate:... \n",
"\n",
" tags \\\n",
"0 python|pandas \n",
"1 python|installation|graphviz \n",
"2 python|shared-libraries|pip|include-path|pyodbc \n",
"3 python|url|parsing|dns|extract \n",
"4 python \n",
"\n",
" answers score \n",
"0 <p>There is now a <a href=\"https://pandas.pyda... 11 \n",
"1 <p><strong>Step 1:</strong> Install Graphviz b... 22 \n",
"2 <p>Another way to indicate the location of inc... 19 \n",
"3 <p>No, there is no \"intrinsic\" way of knowing ... 52 \n",
"4 <p>You can declare a dictionary inside a dicti... 106 "
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"# Convert to a dataframe\n",
"df = rows.to_dataframe()\n",
@@ -652,7 +536,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "ba45d58bf96e"
@@ -691,63 +574,24 @@
},
{
"cell_type": "code",
"execution_count": 21,
"execution_count": null,
"metadata": {
"id": "d296e181205d"
},
"outputs": [
{
"data": {
"text/plain": [
"768"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"DIMENSIONS = len(question_embeddings[0])\n",
"\n",
"DIMENSIONS"
"print(DIMENSIONS)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": null,
"metadata": {
"id": "95e408daf219"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Query question = Numbers of Day in Month\n",
"\t0: Numbers of Day in Month: 0.9999998807907104\n",
"\t1: Python: Number of the Week in a Month: 0.8649452924728394\n",
"\t2: Simulate Poisson arrival times given count of arrivals per day: 0.8294509053230286\n",
"\t3: How to workout if a datetime is older than x months in Python: 0.808755099773407\n",
"\t4: How to efficiently add seconds place to date time: 0.7867542505264282\n",
"\t5: Show if restaurant is open or closed based on weekday, opening and closing time: 0.7858442068099976\n",
"\t6: How to subtract datetimes based on transition events in another column: 0.7808101177215576\n",
"\t7: Doing DateTime Comparisons in Filter SQLAlchemy: 0.7684456706047058\n",
"\t8: Python datetime module and getting current time: 0.7682346701622009\n",
"\t9: Multiples of 10 in a list: 0.7647985816001892\n",
"\t10: Counting differences between two strings: 0.7633612155914307\n",
"\t11: Function That Computes Sum of Squares of Numbers in List: 0.7624084949493408\n",
"\t12: Python/Matplotlib - Colorbar Range and Display Values: 0.7616406679153442\n",
"\t13: Most computationally efficient way to count consecutive repeating values: 0.7592378854751587\n",
"\t14: Plotting data with a string as the x-axis: 0.758903980255127\n",
"\t15: How to write an efficient hit counter for websites: 0.758114755153656\n",
"\t16: Number of channels in convLSTM for classifying videos: 0.7568756937980652\n",
"\t17: Sort a set with four pieces of data per element: 0.7568261623382568\n",
"\t18: Interactive Data Visualiation - Python: 0.7562029957771301\n",
"\t19: How to make Images/PDF of Timetable using Python: 0.7535030841827393\n"
]
}
],
"outputs": [],
"source": [
"question_index = 0\n",
"\n",
@@ -787,7 +631,7 @@
"# Create temporary file to write embeddings to\n",
"embeddings_file = tempfile.NamedTemporaryFile(suffix=\".json\", delete=False)\n",
"\n",
"embeddings_file.name"
"print(embeddings_file.name)"
]
},
{
@@ -830,7 +674,7 @@
"id": "QuVl8DrWG8NS"
},
"source": [
"Upload the training data to a Google Cloud Storage bucket."
"Upload the training data to a Cloud Storage bucket."
]
},
{
@@ -866,7 +710,7 @@
},
{
"cell_type": "code",
"execution_count": 39,
"execution_count": null,
"metadata": {
"id": "qiIg9b5zJLi1"
},
@@ -889,7 +733,7 @@
},
{
"cell_type": "code",
"execution_count": 40,
"execution_count": null,
"metadata": {
"id": "Y4zooldkGoM4"
},
@@ -929,7 +773,7 @@
"outputs": [],
"source": [
"INDEX_RESOURCE_NAME = tree_ah_index.resource_name\n",
"INDEX_RESOURCE_NAME"
"print(INDEX_RESOURCE_NAME)"
]
},
{
@@ -955,10 +799,117 @@
{
"cell_type": "markdown",
"metadata": {
"id": "qV2xjAnDDObD"
"id": "4f4f0bc64ddb"
},
"source": [
"## Create an IndexEndpoint with VPC Network"
"## Setup VPC peering network\n",
"\n",
"To use a `Vector Search Index`, you setup a VPC peering network between your project and the `Vertex AI Vector Search` service project. This eliminates additional hops in network traffic and allows using efficient gRPC protocol.\n",
"\n",
"Learn more about [VPC peering](https://cloud.google.com/vertex-ai/docs/general/vpc-peering).\n",
"\n",
"**IMPORTANT: you can only setup one VPC peering to servicenetworking.googleapis.com per project.**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d85e8f48291a"
},
"source": [
"### Create VPC peering\n",
"\n",
"For simplicity, we setup VPC peering to the ucaip-haystack-vpc-network network. You can create a different network for your project.\n",
"\n",
"If you setup VPC peering with any other network, make sure that the network already exists and that your VM is running on that network."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a107544fbabf"
},
"outputs": [],
"source": [
"# This is for display only; you can name the range anything.\n",
"PEERING_RANGE_NAME = \"vertex-ai-prediction-peering-range\"\n",
"NETWORK = \"ucaip-haystack-vpc-network\"\n",
"\n",
"# NOTE: `prefix-length=16` means a CIDR block with mask /16 will be\n",
"# reserved for use by Google services, such as Vertex AI.\n",
"! gcloud compute addresses create $PEERING_RANGE_NAME \\\n",
" --global \\\n",
" --prefix-length=16 \\\n",
" --description=\"peering range for Google service\" \\\n",
" --network=$NETWORK \\\n",
" --purpose=VPC_PEERING"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8e29cad1a0be"
},
"source": [
"### Create the VPC connection\n",
"\n",
"Next, create the connection for VPC peering.\n",
"\n",
"*Note:* If you get a PERMISSION DENIED, you may not have the neccessary role 'Compute Network Admin' set for your default service account. In the Cloud Console, do the following steps.\n",
"\n",
"1. Goto `IAM & Admin`\n",
"2. Find your service account.\n",
"3. Click edit icon.\n",
"4. Select `Add Another Role`.\n",
"5. Enter 'Compute Network Admin'.\n",
"6. Select `Save`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f3f6c85ffc63"
},
"outputs": [],
"source": [
"! gcloud services vpc-peerings connect \\\n",
" --service=servicenetworking.googleapis.com \\\n",
" --network=$NETWORK \\\n",
" --ranges=$PEERING_RANGE_NAME \\\n",
" --project=$PROJECT_ID"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "944d772b1397"
},
"source": [
"Check the status of your peering connections."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b946ce37cc16"
},
"outputs": [],
"source": [
"! gcloud compute networks peerings list --network $NETWORK"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7a5e1b83ae61"
},
"source": [
"#### Construct the full network name\n",
"\n",
"You need to have the full network resource name when you subsequently create an `Vector Search Index Endpoint` resource for VPC peering."
]
},
{
@@ -971,11 +922,27 @@
"source": [
"# Retrieve the project number\n",
"PROJECT_NUMBER = !gcloud projects list --filter=\"PROJECT_ID:'{PROJECT_ID}'\" --format='value(PROJECT_NUMBER)'\n",
"PROJECT_NUMBER = PROJECT_NUMBER[0]\n",
"\n",
"VPC_NETWORK = \"[your-network-name]\"\n",
"VPC_NETWORK_FULL = \"projects/{}/global/networks/{}\".format(PROJECT_NUMBER, VPC_NETWORK)\n",
"VPC_NETWORK_FULL"
"PROJECT_NUMBER = PROJECT_NUMBER[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cd58eb809f71"
},
"outputs": [],
"source": [
"full_network_name = f\"projects/{PROJECT_NUMBER}/global/networks/{NETWORK}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qV2xjAnDDObD"
},
"source": [
"## Create an IndexEndpoint with VPC Network"
]
},
{
@@ -989,7 +956,7 @@
"my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(\n",
" display_name=DISPLAY_NAME,\n",
" description=DISPLAY_NAME,\n",
" network=VPC_NETWORK_FULL,\n",
" network=full_network_name,\n",
")"
]
},
@@ -1038,7 +1005,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "6LCGvBNvBd8D"
@@ -1114,14 +1080,13 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "05514825ba7d"
},
"source": [
"## Storing and retrieving titles from a Redis data store\n",
"When you productionize this code into a service, you will need to convert the nearest nearest id's returned from Vertex AI Matching Engine into data usable by downstream services.\n",
"When you productionize this code into a service, you will need to convert the nearest nearest id's returned from Vertex AI Vector Search into data usable by downstream services.\n",
"\n",
"In this case, you'll need to convert the id's to titles.\n",
"\n",
@@ -1152,18 +1117,21 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# Get host and port info\n",
"REDIS_HOST = ! gcloud redis instances list --filter=\"INSTANCE_NAME:'{REDIS_INSTANCE_NAME}'\" --region {REGION} --format='value(HOST)'\n",
"REDIS_PORT = ! gcloud redis instances list --filter=\"INSTANCE_NAME:'{REDIS_INSTANCE_NAME}'\" --region {REGION} --format='value(PORT)'\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" REDIS_HOST = ! gcloud redis instances list --filter=\"INSTANCE_NAME:'{REDIS_INSTANCE_NAME}'\" --region {REGION} --format='value(HOST)'\n",
" REDIS_PORT = ! gcloud redis instances list --filter=\"INSTANCE_NAME:'{REDIS_INSTANCE_NAME}'\" --region {REGION} --format='value(PORT)'\n",
"\n",
"if isinstance(REDIS_HOST, list):\n",
" REDIS_HOST = REDIS_HOST[0]\n",
" if isinstance(REDIS_HOST, list):\n",
" REDIS_HOST = REDIS_HOST[0]\n",
"\n",
"if isinstance(REDIS_PORT, list):\n",
" REDIS_PORT = REDIS_PORT[0]\n",
" if isinstance(REDIS_PORT, list):\n",
" REDIS_PORT = REDIS_PORT[0]\n",
"\n",
"print(f\"REDIS_HOST = {REDIS_HOST}\")\n",
"print(f\"REDIS_PORT = {REDIS_PORT}\")"
" print(f\"REDIS_HOST = {REDIS_HOST}\")\n",
" print(f\"REDIS_PORT = {REDIS_PORT}\")"
]
},
{
@@ -1177,7 +1145,8 @@
"# Connect to the instance\n",
"import redis\n",
"\n",
"redis_client = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT)"
"if not os.getenv(\"IS_TESTING\"):\n",
" redis_client = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT)"
]
},
{
@@ -1189,7 +1158,8 @@
"outputs": [],
"source": [
"# Convert the id -> title relationship into a dict and write to redis\n",
"redis_client.mset({str(id): str(title) for id, title in zip(df.id, df.title)})"
"if not os.getenv(\"IS_TESTING\"):\n",
" redis_client.mset({str(id): str(title) for id, title in zip(df.id, df.title)})"
]
},
{
@@ -1200,11 +1170,12 @@
},
"outputs": [],
"source": [
"# Verify that redis can retrieve the correct information\n",
"[\n",
" f\"Actual = {title}, Retrieved = {redis_client.get(str(id))}\"\n",
" for id, title in list(zip(df.id, df.title))[:10]\n",
"]"
"if not os.getenv(\"IS_TESTING\"):\n",
" # Verify that redis can retrieve the correct information\n",
" [\n",
" f\"Actual = {title}, Retrieved = {redis_client.get(str(id))}\"\n",
" for id, title in list(zip(df.id, df.title))[:10]\n",
" ]"
]
},
{
@@ -1229,19 +1200,14 @@
"outputs": [],
"source": [
"# Force undeployment of indexes and delete endpoint\n",
"my_index_endpoint.delete(force=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "omj7N9iWv-Tq"
},
"outputs": [],
"source": [
"my_index_endpoint.delete(force=True)\n",
"\n",
"# Delete indexes\n",
"tree_ah_index.delete()"
"tree_ah_index.delete()\n",
"\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
]
},
{