update: touchups

Added new notebook that creates Vertex AI AutoML text entity extraction dataset from PDFs using Vision API (#683 )
2026-09-27 07:31:58 +00:00 · 2022-06-23 16:56:20 +00:00 · 2022-06-23 16:46:46 +00:00 · 2022-06-23 08:38:14 -07:00 · 2022-06-22 21:31:55 -05:00 · 2022-06-22 21:49:09 -04:00
3 changed files with 1085 additions and 104 deletions
@@ -21,4 +21,6 @@
 /vertex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb @entrpn
 /vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb @RajeshThallam
 /vertex_endpoints/optimized_tensorflow_runtime @vlasenkoalexey
-/notebooks/community/ml_ops/stage2/get_started_with_visionapi_and_automl.ipynb @mansari
+/notebooks/community/ml_ops/stage2/get_started_with_visionapi_and_automl.ipynb @mansari
+/notebooks/community/neo4j/graph_paysim.ipynb @benofben @laeg
+/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb @mansari
@@ -0,0 +1,994 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "copyright"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2022 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4JIDiHvGasba"
+      },
+      "source": [
+        "This notebook was contributed by [Mohammad Al-Ansari](https://github.com/Mansari)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2xDiUNIZINWp"
+      },
+      "source": [
+        "# E2E ML on GCP: MLOps stage 1 : data management: create an unlabelled Vertex AI AutoML text entity extraction dataset from PDFs using Vision API\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td>\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb\">\n",
+        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
+        "      View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb\">\n",
+        "     <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
+        "      Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "<br/><br/><br/>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "H0alLPo_A-LK"
+      },
+      "source": [
+        "## Overview\n",
+        "\n",
+        "This notebook will create an unlabelled `Vertex AI AutoML` text entity extraction dataset based on a collection of PDF files stored in a Cloud Storage bucket. \n",
+        "\n",
+        "The notebook can be modified to create different types of text datasets including sentiment analysis and classification."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "W4IBLTKOA5nl"
+      },
+      "source": [
+        "### Dataset\n",
+        "\n",
+        "The dataset used for this tutorial is the [Patent PDF Samples with Extracted Structured Data](https://console.cloud.google.com/marketplace/product/global-patents/labeled-patents) from Google Public Data Sets. \n",
+        "\n",
+        "This dataset includes data extracted from over 300 patent documents issued in the US and EU. The dataset includes links to Cloud Storage blobs for the first page of each patent, in addition to a number of extracted entities. \n",
+        "\n",
+        "The data is published as a [public dataset](https://cloud.google.com/bigquery/public-data) on `BigQuery`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3f8c2f702ccd"
+      },
+      "source": [
+        "### Objective\n",
+        "\n",
+        "In this tutorial, you learn to use `Vision API` to extract text from PDF files stored on a Cloud Storage bucket. You will then process the results and create an unlabelled `Vertex AI Dataset`, compatible with `AutoML`, for text entity extraction.\n",
+        "\n",
+        "You can then either use Google Cloud console to annotate / label the dataset, or create a labelling job as demonstrated in [this notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb).\n",
+        "\n",
+        "This tutorial uses the following Google Cloud services:\n",
+        "\n",
+        "- `Vision AI`\n",
+        "- `Vertex AI AutoML`\n",
+        "\n",
+        "The steps performed include:\n",
+        "\n",
+        "1. Using `Vision API` to perform Optical Character Recognition (OCR) to extract text from PDF files.\n",
+        "2. Processing the results and saving them to text files.\n",
+        "3. Generating a `Vertex AI Dataset` import file.\n",
+        "4. Creating a new unlabelled text entity extraction `Vertex AI Dataset` resource in `Vertex AI`."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CgLDJ419LPJs"
+      },
+      "source": [
+        "### Costs\n",
+        "\n",
+        "This tutorial uses billable components of Google Cloud:\n",
+        "\n",
+        "* Vision API\n",
+        "* Vertex AI\n",
+        "* Cloud Storage\n",
+        "\n",
+        "Learn about [Vertex AI\n",
+        "pricing](https://cloud.google.com/vertex-ai/pricing), [Vision API pricing](https://cloud.google.com/vision/pricing), [Cloud Storage\n",
+        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Calculator](https://cloud.google.com/products/calculator/)\n",
+        "to generate a cost estimate based on your projected usage."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "va2g7m9wLTjA"
+      },
+      "source": [
+        "### Set up your local development environment\n",
+        "\n",
+        "If you are using Colab or Vertex AI Workbench Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
+        "\n",
+        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
+        "\n",
+        "- The Vision API SDK\n",
+        "- The Vertex AI SDK\n",
+        "- The Cloud Storage SDK\n",
+        "- Git\n",
+        "- Python 3\n",
+        "- virtualenv\n",
+        "- Jupyter notebook running in a virtual environment with Python 3\n",
+        "\n",
+        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
+        "\n",
+        "1. [Install and initialize the SDKs](https://cloud.google.com/sdk/docs/).\n",
+        "\n",
+        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
+        "\n",
+        "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.  Activate the virtual environment.\n",
+        "\n",
+        "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
+        "\n",
+        "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
+        "\n",
+        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "X2tZAmugAe6h"
+      },
+      "source": [
+        "## Installation\n",
+        "\n",
+        "Install the packages required for executing this notebook. You can ignore errors for the `pip` dependecy resolver as they do not impact this notebook."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "BQOsJ1hZAZu0"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "# The Vertex AI Workbench Notebook product has specific requirements\n",
+        "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
+        "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
+        "    \"/opt/deeplearning/metadata/env_version\"\n",
+        ")\n",
+        "\n",
+        "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
+        "USER_FLAG = \"\"\n",
+        "if IS_WORKBENCH_NOTEBOOK:\n",
+        "    USER_FLAG = \"--user\"\n",
+        "\n",
+        "! pip3 install --upgrade google-cloud-storage google-cloud-vision google-cloud-aiplatform $USER_FLAG -q"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yzvvcmCuAon3"
+      },
+      "source": [
+        "### Restart the kernel\n",
+        "\n",
+        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6qEonzbuAoI_"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "if not os.getenv(\"IS_TESTING\"):\n",
+        "    # Automatically restart kernel after installs\n",
+        "    import IPython\n",
+        "\n",
+        "    app = IPython.Application.instance()\n",
+        "    app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pGbbyN7rAuRM"
+      },
+      "source": [
+        "## Before you begin\n",
+        "\n",
+        "### GPU runtime\n",
+        "\n",
+        "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n",
+        "\n",
+        "### Set up your Google Cloud project\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
+        "\n",
+        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "\n",
+        "3. [Enable the following APIs: Vision API, Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com,aiplatform.googleapis.com,compute_component,storage-component.googleapis.com)\n",
+        "\n",
+        "4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).\n",
+        "\n",
+        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
+        "Cloud SDK uses the right project for all the commands in this notebook.\n",
+        "\n",
+        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AE97adtnAzrr"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "nWlzLu5ELxWd"
+      },
+      "outputs": [],
+      "source": [
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
+        "    # Get your GCP project id from gcloud\n",
+        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
+        "    PROJECT_ID = shell_output[0]\n",
+        "    print(\"Project ID:\", PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GB5b27r0LxqE"
+      },
+      "outputs": [],
+      "source": [
+        "! gcloud config set project $PROJECT_ID"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pMJdU1K5xG7D"
+      },
+      "source": [
+        "### Regions\n",
+        "\n",
+        "#### Vision AI\n",
+        "\n",
+        "You can now specify continent-level data storage and Optical Character Regonition (OCR) processing by setting the `VISION_AI_REGION` variable. You can select one of the following options:\n",
+        "\n",
+        "* USA country only: `us`\n",
+        "* The European Union: `eu`\n",
+        "\n",
+        "Learn more about [Vision AI regions for OCR](https://cloud.google.com/vision/docs/pdf#regionalization)\n",
+        "\n",
+        "#### Vertex AI\n",
+        "\n",
+        "You can also change the `VERTEX_AI_REGION` variable, which is used for operations throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
+        "\n",
+        "- Americas: `us-central1`\n",
+        "- Europe: `europe-west4`\n",
+        "- Asia Pacific: `asia-east1`\n",
+        "\n",
+        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
+        "\n",
+        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5EhEAOK5xIKc"
+      },
+      "outputs": [],
+      "source": [
+        "VISION_AI_REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if VISION_AI_REGION == \"[your-region]\":\n",
+        "    VISION_AI_REGION = \"us\"\n",
+        "\n",
+        "VERTEX_AI_REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if VERTEX_AI_REGION == \"[your-region]\":\n",
+        "    VERTEX_AI_REGION = \"us-central1\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xkgvWoXkxM1r"
+      },
+      "source": [
+        "### Timestamp\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append onto the name of resources which will be created in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gr0HTpQZxNy4"
+      },
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AA-ns5CcBA9U"
+      },
+      "source": [
+        "### Vertex AI dataset import schema\n",
+        "\n",
+        "This constant tells Vertex AI the schema for importing the dataset. In this tutorial you are going to use the value for text extraction, but you can also change it to any of the values below for other use cases:\n",
+        "\n",
+        "- \n",
+        "`aiplatform.schema.dataset.ioformat.text.single_label_classification`\n",
+        "\n",
+        "- \n",
+        "`aiplatform.schema.dataset.ioformat.text.multi_label_classification`\n",
+        "\n",
+        "- \n",
+        "`aiplatform.schema.dataset.ioformat.text.extraction`\n",
+        "\n",
+        "- \n",
+        "`aiplatform.schema.dataset.ioformat.text.sentiment`\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jnOb6Pp-4w5P"
+      },
+      "outputs": [],
+      "source": [
+        "from google.cloud import aiplatform\n",
+        "\n",
+        "DATASET_IMPORT_SCHEMA = aiplatform.schema.dataset.ioformat.text.extraction"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ekbg-G7UA-bK"
+      },
+      "source": [
+        "### Authenticate your Google Cloud account\n",
+        "\n",
+        "**If you are using Vertex AI Workbench**, your environment is already authenticated. Skip this step. If you receive errors still, you may have to grant the service account that is your Workbench notebook is running under access to the services listed below.\n",
+        "\n",
+        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
+        "\n",
+        "**Otherwise**, follow these steps:\n",
+        "\n",
+        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
+        "\n",
+        "**Click Create service account**.\n",
+        "\n",
+        "In the **Service account name** field, enter a name, and click **Create**.\n",
+        "\n",
+        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
+        "\n",
+        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "lCRrULxKBAfa"
+      },
+      "outputs": [],
+      "source": [
+        "# If you are running this notebook in Colab, run this cell and follow the\n",
+        "# instructions to authenticate your GCP account. This provides access to your\n",
+        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
+        "# requests.\n",
+        "\n",
+        "import os\n",
+        "import sys\n",
+        "\n",
+        "# If on Vertex AI Workbench, then don't execute this code\n",
+        "IS_COLAB = \"google.colab\" in sys.modules\n",
+        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
+        "    \"DL_ANACONDA_HOME\"\n",
+        "):\n",
+        "    if \"google.colab\" in sys.modules:\n",
+        "        from google.colab import auth as google_auth\n",
+        "\n",
+        "        google_auth.authenticate_user()\n",
+        "\n",
+        "    # If you are running this notebook locally, replace the string below with the\n",
+        "    # path to your service account key and run this cell to authenticate your GCP\n",
+        "    # account.\n",
+        "    elif not os.getenv(\"IS_TESTING\"):\n",
+        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rHB6fbonMMbI"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions. This bucket will be also used to store the output of the Vision API SDK PDF-to-text conversion process.\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZSM5j0nfMOVK"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "i6H2iQX2MP-s"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
+        "    BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
+        "    BUCKET_URI = \"gs://\" + BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AOsnYE5cMQX4"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "33RgSjhyMR6C"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $VERTEX_AI_REGION -p $PROJECT_ID $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "UpKfi0VfMTwe"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "G9dMjMnkMVNt"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "k2qH7YCI0vnG"
+      },
+      "source": [
+        "### Set up variables\n",
+        "\n",
+        "Next, set up some variables used throughout the tutorial.\n",
+        "\n",
+        "### Import libraries and define constants"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TB5-_2Xh01NH"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "from google.cloud import aiplatform, storage, vision"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-v7gY_KABIn8"
+      },
+      "source": [
+        "### Initialize Vision API SDK for Python\n",
+        "\n",
+        "Initialize the `Vision AI` SDK for Python for your project and region."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DRbf--kWBLpx"
+      },
+      "outputs": [],
+      "source": [
+        "vision_client_options = {\n",
+        "    \"quota_project_id\": PROJECT_ID,\n",
+        "    \"api_endpoint\": f\"{VISION_AI_REGION}-vision.googleapis.com\",\n",
+        "}\n",
+        "vision_client = vision.ImageAnnotatorClient(client_options=vision_client_options)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CA4nNVbBZ25d"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the `Vertex AI` SDK for Python for your project, region and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "awWpNW1vZ6uV"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(\n",
+        "    project=PROJECT_ID, location=VERTEX_AI_REGION, staging_bucket=BUCKET_URI\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "debBBljMDqkM"
+      },
+      "source": [
+        "### Initialize Cloud Storage SDK for Python\n",
+        "\n",
+        "Initialize the `Cloud Storage` SDK for Python for your project."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZtzmI9tpDr4e"
+      },
+      "outputs": [],
+      "source": [
+        "storage_client = storage.Client(project=PROJECT_ID)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mvD0BxVXMtJe"
+      },
+      "source": [
+        "## Tutorial\n",
+        "\n",
+        "Now you are ready to start creating an unlabelled `Vertex AI Dataset` text entity extraction dataset from PDF files."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EurEFM3GBap9"
+      },
+      "source": [
+        "### Convert PDF files to text using Vision API\n",
+        "\n",
+        "First, you make a `Vision API` request to OCR to text the PDFs from the Patent samples stored in the Cloud Storage bucket.\n",
+        "\n",
+        "*Note:* `Visions API` only allows batches of 100 document submissions at a time."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "uXVPOvjTBeK3"
+      },
+      "outputs": [],
+      "source": [
+        "ORIGIN_BUCKET_NAME = \"gcs-public-data--labeled-patents\"\n",
+        "# You can add a path if needed\n",
+        "ORIGIN_BUCKET_PATH = \"\"\n",
+        "\n",
+        "DESTINATION_BUCKET_NAME = BUCKET_NAME\n",
+        "DESTINATION_BUCKET_PATH = \"ocr-output\"\n",
+        "\n",
+        "gcs_destination_uri = f\"gs://{DESTINATION_BUCKET_NAME}/{DESTINATION_BUCKET_PATH}\"\n",
+        "\n",
+        "# Specify the feature for the Vision API processor\n",
+        "feature = vision.Feature(type_=vision.Feature.Type.DOCUMENT_TEXT_DETECTION)\n",
+        "\n",
+        "# Retrieve a list of all files in the bucket and path\n",
+        "blobs = storage_client.list_blobs(\n",
+        "    ORIGIN_BUCKET_NAME, prefix=ORIGIN_BUCKET_PATH, delimiter=\"/\"\n",
+        ")\n",
+        "\n",
+        "# Create a collection of requests. The SDK requires a separate request per each\n",
+        "# file that we want to extract text from\n",
+        "async_requests = []\n",
+        "\n",
+        "# Visions API only supports processing up to 100 documents at a time\n",
+        "# so we will process the first 100 elements only\n",
+        "sliced_blob_list = list(blobs)[:100]\n",
+        "\n",
+        "# Loop through the source bucket and create a request for each file there\n",
+        "for blob in sliced_blob_list:\n",
+        "    # Build input_config\n",
+        "    # Ensure we are only processing PDF files\n",
+        "    if blob.name.endswith(\".pdf\"):\n",
+        "        gcs_source = vision.GcsSource(uri=f\"gs://{ORIGIN_BUCKET_NAME}/{blob.name}\")\n",
+        "        input_config = vision.InputConfig(\n",
+        "            gcs_source=gcs_source, mime_type=\"application/pdf\"\n",
+        "        )\n",
+        "\n",
+        "        # Build output config\n",
+        "        # Get file name\n",
+        "        file_name = os.path.splitext(os.path.basename(blob.name))[0]\n",
+        "        gcs_destination = vision.GcsDestination(\n",
+        "            uri=f\"{gcs_destination_uri}/{file_name}-\"\n",
+        "        )\n",
+        "        output_config = vision.OutputConfig(gcs_destination=gcs_destination)\n",
+        "\n",
+        "        # Build request object and add to the collection\n",
+        "        async_request = vision.AsyncAnnotateFileRequest(\n",
+        "            features=[feature], input_config=input_config, output_config=output_config\n",
+        "        )\n",
+        "\n",
+        "        async_requests.append(async_request)\n",
+        "\n",
+        "print(f\"Created {len(async_requests)} requests\")\n",
+        "\n",
+        "# Submit the batch OCR job\n",
+        "\n",
+        "operation = vision_client.async_batch_annotate_files(requests=async_requests)\n",
+        "print(\"Submitting the batch OCR job\")\n",
+        "\n",
+        "print(\"Waiting for the operation to finish... this will take a short while\")\n",
+        "\n",
+        "response = operation.result(timeout=420)\n",
+        "\n",
+        "print(\"Completed!\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7b15473e1937"
+      },
+      "source": [
+        "#### Quick peek at extracted annotated JSON files\n",
+        "\n",
+        "Next, you take a peek at the contents of one of the extracted JSON annotated files."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4366442c1373"
+      },
+      "outputs": [],
+      "source": [
+        "json_files = ! gsutil ls {gcs_destination_uri}\n",
+        "\n",
+        "example = json_files[0]\n",
+        "! gsutil cat {example} | head -n 1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "QWmeHWPIHako"
+      },
+      "source": [
+        "### Process results and build the import file\n",
+        "\n",
+        "The `Vision API` output is in JSON format, and contains detailed text extraction data. You only need the full text output, so you will processs the JSON results, extract the text output, and save it in new text files to be used later in the tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WDLtiejKHug6"
+      },
+      "outputs": [],
+      "source": [
+        "import json\n",
+        "\n",
+        "print(\"Extracting text from Vision API output and saving it to text files\")\n",
+        "\n",
+        "ocr_blobs = storage_client.list_blobs(\n",
+        "    DESTINATION_BUCKET_NAME, prefix=DESTINATION_BUCKET_PATH\n",
+        ")\n",
+        "\n",
+        "output_bucket = storage_client.bucket(DESTINATION_BUCKET_NAME)\n",
+        "\n",
+        "# begin building the import file content\n",
+        "import_file_entries = []\n",
+        "\n",
+        "for ocr_blob in ocr_blobs:\n",
+        "    # Only process .json files, in case we previously processed files and had .txt files\n",
+        "    if ocr_blob.name.endswith(\".json\"):\n",
+        "        print(f\"Extracting text from {ocr_blob.name}\")\n",
+        "        # read each blob into a stream\n",
+        "        contents = ocr_blob.download_as_string()\n",
+        "        # load as JSON\n",
+        "        json_object = json.loads(contents)\n",
+        "        # extract text\n",
+        "        full_text = \"\"\n",
+        "        for response in json_object[\"responses\"]:\n",
+        "            if response[\"fullTextAnnotation\"]:\n",
+        "                full_text += response[\"fullTextAnnotation\"][\"text\"] + \"\\r\\n\"\n",
+        "\n",
+        "        # save as a blob\n",
+        "        output_blob_name = f\"{ocr_blob.name}.txt\"\n",
+        "        import_file_blob = output_bucket.blob(output_blob_name)\n",
+        "        import_file_blob.upload_from_string(full_text)\n",
+        "\n",
+        "        # create import file listing\n",
+        "        import_file_entry = {\n",
+        "            \"textGcsUri\": f\"gs://{DESTINATION_BUCKET_NAME}/{output_blob_name}\"\n",
+        "        }\n",
+        "\n",
+        "        import_file_entries.append(import_file_entry)\n",
+        "\n",
+        "print(\"Extraction completed!\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0a5aae0eab44"
+      },
+      "source": [
+        "#### Quick peek at extracted text files\n",
+        "\n",
+        "Next, you take a peek at the contents of one of the extracted text files."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "76ce5f57b1ae"
+      },
+      "outputs": [],
+      "source": [
+        "example = import_file_entries[0][\"textGcsUri\"]\n",
+        "\n",
+        "! gsutil cat {example}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hqTLS_AmLWQP"
+      },
+      "source": [
+        "### Generate and save import file to be used in `Vertex AI Dataset` resource\n",
+        "\n",
+        "You will now build the import file that will be used to create the `Vertex AI Dataset` resource."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_xFvOdQ_LWne"
+      },
+      "outputs": [],
+      "source": [
+        "IMPORT_FILE_PATH = \"import_file\"\n",
+        "\n",
+        "# Convert import file entries to JSON Lines format\n",
+        "import_file_content = \"\"\n",
+        "for entry in import_file_entries:\n",
+        "    import_file_content += json.dumps(entry) + \"\\n\"\n",
+        "\n",
+        "print(f\"Created import file based on {len(import_file_entries)} annotations\")\n",
+        "\n",
+        "# Upload content to GCS to be used in our next step\n",
+        "gcs_annotation_file_name = f\"{IMPORT_FILE_PATH}/import_file_{TIMESTAMP}.jsonl\"\n",
+        "import_file_blob = output_bucket.blob(gcs_annotation_file_name)\n",
+        "import_file_blob.upload_from_string(import_file_content)\n",
+        "\n",
+        "print(f\"Uploaded import file to {output_bucket.name}/{gcs_annotation_file_name}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6dVjFftOaKdw"
+      },
+      "source": [
+        "### Create an unlabelled `Vertex AI Dataset` resource\n",
+        "\n",
+        "Next, you create the `Dataset` resource using the `create` method for the `TextDataset` class, which takes the following parameters:\n",
+        "\n",
+        "- `display_name`: The human readable name for the `Dataset` resource.\n",
+        "- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n",
+        "- `import_schema_uri`: The data labeling schema for the data items.\n",
+        "\n",
+        "This operation may take ten to twenty minutes."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ciM9HLGCaOTJ"
+      },
+      "outputs": [],
+      "source": [
+        "print(\"Creating dataset ...\")\n",
+        "\n",
+        "dataset = aiplatform.TextDataset.create(\n",
+        "    display_name=\"Text Dataset \" + TIMESTAMP,\n",
+        "    gcs_source=[f\"gs://{output_bucket.name}/{gcs_annotation_file_name}\"],\n",
+        "    import_schema_uri=DATASET_IMPORT_SCHEMA,\n",
+        ")\n",
+        "\n",
+        "print(\"Completed!\")\n",
+        "\n",
+        "print(dataset.resource_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2vagHf5T6Jd4"
+      },
+      "source": [
+        "**Congratulations, your dataset is now ready for annotations!**\n",
+        "\n",
+        "You have two options:\n",
+        "\n",
+        "*   Use Google Cloud Console to manually annotate the dataset in `Vertex AI`. Checkout [this link](https://cloud.google.com/vertex-ai/docs/datasets/label-using-console#entity-extraction) for more details on how to do so.\n",
+        "*   Create a labelling job to request data labelling. Check out [this link](https://cloud.google.com/vertex-ai/docs/datasets/data-labeling-job) and [this notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb) for more details and examples.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cleanup:migration,new"
+      },
+      "source": [
+        "# Cleaning up\n",
+        "\n",
+        "To clean up all GCP resources used in this project, you can [delete the GCP\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aoJ18d8Y_jAy"
+      },
+      "outputs": [],
+      "source": [
+        "# Set this to true only if you'd like to delete your bucket\n",
+        "delete_bucket = False\n",
+        "\n",
+        "# Delete the dataset using the Vertex AI fully qualified identifier for the dataset\n",
+        "dataset.delete()\n",
+        "\n",
+        "# Delete the bucket created\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -r $BUCKET_URI"
+      ]
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "collapsed_sections": [],
+      "name": "get_started_with_visionapi_and_vertex_datasets.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}
@@ -8,7 +8,7 @@
      },
      "outputs": [],
      "source": [
-        "# Copyright 2021 Google LLC\n",
+        "# Copyright 2022 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
@@ -42,6 +42,11 @@
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/laeg/vertex-ai-samples/main/notebooks/community/neo4j/graph_paysim.ipynb\">\n",
+        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
+        "    </a>\n",
+        "</td>\n",
        "</table>"
      ]
    },
@@ -143,7 +148,7 @@
      },
      "outputs": [],
      "source": [
-        "!pip install --quiet --upgrade neo4j"
+        "!pip install --quiet --upgrade graphdatascience==1.0.0"
      ]
    },
    {
@@ -255,7 +260,7 @@
      "outputs": [],
      "source": [
        "import pandas as pd\n",
-        "from neo4j import GraphDatabase"
+        "from graphdatascience import GraphDataScience"
      ]
    },
    {
@@ -266,7 +271,19 @@
      },
      "outputs": [],
      "source": [
-        "driver = GraphDatabase.driver(DB_URL, auth=(DB_USER, DB_PASS))"
+        "# If you are connecting the client to an AuraDS instance, you can get the recommended non-default configuration settings of the Python Driver applied automatically. To achieve this, set the constructor argument aura_ds=True\n",
+        "gds = GraphDataScience(DB_URL, auth=(DB_USER, DB_PASS), aura_ds=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f14915ddd1fb"
+      },
+      "outputs": [],
+      "source": [
+        "gds.set_database(DB_NAME)"
      ]
    },
    {
@@ -287,19 +304,16 @@
      "outputs": [],
      "source": [
        "# node labels\n",
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "    CALL db.labels() YIELD label\n",
-        "    CALL apoc.cypher.run('MATCH (:`'+label+'`) RETURN count(*) as freq', {})\n",
-        "    YIELD value\n",
-        "    RETURN label, value.freq AS freq\n",
+        "result = gds.run_cypher(\n",
        "    \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "display(df)"
+        "CALL db.labels() YIELD label\n",
+        "CALL apoc.cypher.run('MATCH (:`'+label+'`) RETURN count(*) as freq', {})\n",
+        "YIELD value\n",
+        "RETURN label, value.freq AS freq\n",
+        "\"\"\"\n",
+        ")\n",
+        "\n",
+        "display(result)"
      ]
    },
    {
@@ -311,20 +325,17 @@
      "outputs": [],
      "source": [
        "# relationship types\n",
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "      CALL db.relationshipTypes() YIELD relationshipType as type\n",
-        "      CALL apoc.cypher.run('MATCH ()-[:`'+type+'`]->() RETURN count(*) as freq', {})\n",
-        "      YIELD value\n",
-        "      RETURN type AS relationshipType, value.freq AS freq\n",
-        "      ORDER by freq DESC\n",
-        "      \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "display(df)"
+        "result = gds.run_cypher(\n",
+        "    \"\"\"\n",
+        "CALL db.relationshipTypes() YIELD relationshipType as type\n",
+        "CALL apoc.cypher.run('MATCH ()-[:`'+type+'`]->() RETURN count(*) as freq', {})\n",
+        "YIELD value\n",
+        "RETURN type AS relationshipType, value.freq AS freq\n",
+        "ORDER by freq DESC\n",
+        "\"\"\"\n",
+        ")\n",
+        "\n",
+        "display(result)"
      ]
    },
    {
@@ -336,23 +347,20 @@
      "outputs": [],
      "source": [
        "# transaction types\n",
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "    MATCH (t:Transaction)\n",
-        "    WITH sum(t.amount) AS globalSum, count(t) AS globalCnt\n",
-        "    WITH *, 10^3 AS scaleFactor\n",
-        "    UNWIND ['CashIn', 'CashOut', 'Payment', 'Debit', 'Transfer'] AS txType\n",
-        "      CALL apoc.cypher.run('MATCH (t:' + txType + ')\n",
-        "        RETURN sum(t.amount) as txAmount, count(t) AS txCnt', {})\n",
-        "      YIELD value\n",
-        "    RETURN txType,value.txAmount AS TotalMarketValue\n",
+        "result = gds.run_cypher(\n",
        "    \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "display(df)"
+        "      MATCH (t:Transaction)\n",
+        "      WITH sum(t.amount) AS globalSum, count(t) AS globalCnt\n",
+        "      WITH *, 10^3 AS scaleFactor\n",
+        "      UNWIND ['CashIn', 'CashOut', 'Payment', 'Debit', 'Transfer'] AS txType\n",
+        "        CALL apoc.cypher.run('MATCH (t:' + txType + ')\n",
+        "          RETURN sum(t.amount) as txAmount, count(t) AS txCnt', {})\n",
+        "        YIELD value\n",
+        "      RETURN txType,value.txAmount AS TotalMarketValue\n",
+        "    \"\"\"\n",
+        ")\n",
+        "\n",
+        "display(result)"
      ]
    },
    {
@@ -375,18 +383,14 @@
      },
      "outputs": [],
      "source": [
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "    CALL gds.graph.create.cypher('client_graph', \n",
-        "      'MATCH (c:Client) RETURN id(c) as id, c.num_transactions as num_transactions, c.total_transaction_amnt as total_transaction_amnt, c.is_fraudster as is_fraudster',\n",
-        "      'MATCH (c:Client)-[:PERFORMED]->(t:Transaction)-[:TO]->(c2:Client) return id(c) as source, id(c2) as target, sum(t.amount) as amount, \"TRANSACTED_WITH\" as type ')\n",
-        "    \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "display(df)"
+        "# We get a tuple back with an object that represents the graph projection and the results of the GDS call\n",
+        "G, results = gds.graph.project.cypher(\n",
+        "    \"client_graph\",\n",
+        "    \"MATCH (c:Client) RETURN id(c) as id, c.num_transactions as num_transactions, c.total_transaction_amnt as total_transaction_amnt, c.is_fraudster as is_fraudster\",\n",
+        "    'MATCH (c:Client)-[:PERFORMED]->(t:Transaction)-[:TO]->(c2:Client) return id(c) as source, id(c2) as target, sum(t.amount) as amount, \"TRANSACTED_WITH\" as type ',\n",
+        ")\n",
+        "\n",
+        "display(results)"
      ]
    },
    {
@@ -406,25 +410,19 @@
      },
      "outputs": [],
      "source": [
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "    CALL gds.fastRP.mutate('client_graph',{\n",
-        "      relationshipWeightProperty:'amount',\n",
-        "      iterationWeights: [0.0, 1.00, 1.00, 0.80, 0.60],\n",
-        "      featureProperties: ['num_transactions', 'total_transaction_amnt'],\n",
-        "      propertyRatio: 0.25, \n",
-        "      nodeSelfInfluence: 0.15,\n",
-        "      embeddingDimension: 16,\n",
-        "      randomSeed: 1, \n",
-        "      mutateProperty:'embedding'\n",
-        "    })\n",
-        "    \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "display(df)"
+        "results = gds.fastRP.mutate(\n",
+        "    G,\n",
+        "    relationshipWeightProperty=\"amount\",\n",
+        "    iterationWeights=[0.0, 1.00, 1.00, 0.80, 0.60],\n",
+        "    featureProperties=[\"num_transactions\", \"total_transaction_amnt\"],\n",
+        "    propertyRatio=0.25,\n",
+        "    nodeSelfInfluence=0.15,\n",
+        "    embeddingDimension=16,\n",
+        "    randomSeed=1,\n",
+        "    mutateProperty=\"embedding\",\n",
+        ")\n",
+        "\n",
+        "display(result)"
      ]
    },
    {
@@ -444,19 +442,11 @@
      },
      "outputs": [],
      "source": [
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "    CALL gds.graph.streamNodeProperties\n",
-        "    ('client_graph', ['embedding', 'num_transactions', 'total_transaction_amnt', 'is_fraudster'])\n",
-        "    YIELD nodeId, nodeProperty, propertyValue\n",
-        "    RETURN nodeId, nodeProperty, propertyValue\n",
-        "    \"\"\"\n",
-        "        ).data()\n",
-        "    )\n",
-        "df = pd.DataFrame(result)\n",
-        "df.head()"
+        "node_properties = gds.graph.streamNodeProperties(\n",
+        "    G, [\"embedding\", \"num_transactions\", \"total_transaction_amnt\", \"is_fraudster\"]\n",
+        ")\n",
+        "\n",
+        "node_properties.head()"
      ]
    },
    {
@@ -476,7 +466,9 @@
      },
      "outputs": [],
      "source": [
-        "x = df.pivot(index=\"nodeId\", columns=\"nodeProperty\", values=\"propertyValue\")\n",
+        "x = node_properties.pivot(\n",
+        "    index=\"nodeId\", columns=\"nodeProperty\", values=\"propertyValue\"\n",
+        ")\n",
        "x = x.reset_index()\n",
        "x.columns.name = None\n",
        "x.head()"
@@ -699,8 +691,8 @@
        "id": "ArK3cfKsdT1x"
      },
      "source": [
-        "## Train and deploy a model on GCP\n",
-        "We'll use the engineered features to train an AutoML Tables model, then deploy it to an endpoint"
+        "## Train and deploy a model with Vertex AI\n",
+        "We'll use the engineered features to train an AutoML Tabular Data, then deploy it to an endpoint"
      ]
    },
    {
@@ -782,8 +774,8 @@
        "id": "-NnDaATyWY7z"
      },
      "source": [
-        "## Loading Data into GCP Feature Store\n",
-        "In this section, we'll take our dataframe with newly engineered features and load that into GCP feature store."
+        "## Loading Data into Vertex AI Feature Store\n",
+        "In this section, we'll take our dataframe with newly engineered features and load that into Vertex AI Feature Store."
      ]
    },
    {
@@ -1083,14 +1075,7 @@
      },
      "outputs": [],
      "source": [
-        "with driver.session(database=DB_NAME) as session:\n",
-        "    result = session.read_transaction(\n",
-        "        lambda tx: tx.run(\n",
-        "            \"\"\"\n",
-        "        CALL gds.graph.drop('client_graph')\n",
-        "        \"\"\"\n",
-        "        ).data()\n",
-        "    )"
+        "gds.graph.drop(G)"
      ]
    },
    {
Author	SHA1	Message	Date
Andrew Ferlitsch	6132e76d90	update: touchups	2022-06-23 16:56:20 +00:00
Andrew Ferlitsch	4cb8cbf83e	update: touchups	2022-06-23 16:46:46 +00:00
Mohammad Al-AnsariandGitHub	d5057da9bb	Added new notebook that creates Vertex AI AutoML text entity extraction dataset from PDFs using Vision API (#683 ) * Added new Stage 1 notebook to create unlabelled Vertex AI AutoML text entity extraction dataset from collection of PDF files on Google Cloud Storage * Linted notebook * Removed TODOs * Updates per PR comments * Revered to multiple imports per line	2022-06-23 08:38:14 -07:00
laexample GitHub Karl Weinmeister Ivan Cheung	24904a5999	feat: adding code owners and updating graph_paysim (#645 ) * adding code owners and updating graph_paysim * formatted Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> Co-authored-by: Ivan Cheung <ivans.mailbox@gmail.com>	2022-06-22 21:31:55 -05:00
Ivan CheungandGitHub	881a2b45c5	Improved git diff logic (#687 )	2022-06-22 21:49:09 -04:00