Compare commits

...
Author SHA1 Message Date
Andrew Ferlitsch 6132e76d90 update: touchups 2022-06-23 16:56:20 +00:00
Andrew Ferlitsch 4cb8cbf83e update: touchups 2022-06-23 16:46:46 +00:00
@@ -38,7 +38,7 @@
"id": "2xDiUNIZINWp"
},
"source": [
"# E2E ML on GCP: MLOps stage 1 : experiments: create an unlabelled Vertex AI AutoML text entity extraction dataset from PDFs using Vision API\n",
"# E2E ML on GCP: MLOps stage 1 : data management: create an unlabelled Vertex AI AutoML text entity extraction dataset from PDFs using Vision API\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
@@ -70,7 +70,7 @@
"source": [
"## Overview\n",
"\n",
"This notebook will create an unlabelled `Vertex AI AutoML` text entity extraction dataset based on a collection of PDF files stored in a Google Cloud Storage bucket. \n",
"This notebook will create an unlabelled `Vertex AI AutoML` text entity extraction dataset based on a collection of PDF files stored in a Cloud Storage bucket. \n",
"\n",
"The notebook can be modified to create different types of text datasets including sentiment analysis and classification."
]
@@ -85,7 +85,7 @@
"\n",
"The dataset used for this tutorial is the [Patent PDF Samples with Extracted Structured Data](https://console.cloud.google.com/marketplace/product/global-patents/labeled-patents) from Google Public Data Sets. \n",
"\n",
"This dataset includes data extracted from over 300 patent documents issued in the US and EU. The dataset includes links to Google Cloud Storage blobs for the first page of each patent, in addition to a number of extracted entities. \n",
"This dataset includes data extracted from over 300 patent documents issued in the US and EU. The dataset includes links to Cloud Storage blobs for the first page of each patent, in addition to a number of extracted entities. \n",
"\n",
"The data is published as a [public dataset](https://cloud.google.com/bigquery/public-data) on `BigQuery`."
]
@@ -98,9 +98,9 @@
"source": [
"### Objective\n",
"\n",
"In this tutorial, you will use `Vision API` to extract text from PDF files stored on a Google Cloud Storage bucket. You will then process the results and create an unlabelled `Vertex AI AutoML` dataset for text entity extraction.\n",
"In this tutorial, you learn to use `Vision API` to extract text from PDF files stored on a Cloud Storage bucket. You will then process the results and create an unlabelled `Vertex AI Dataset`, compatible with `AutoML`, for text entity extraction.\n",
"\n",
"You can then either use Google Cloud console to annotate / label the dataset, or create a labelling job as mentioned in [this notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb).\n",
"You can then either use Google Cloud console to annotate / label the dataset, or create a labelling job as demonstrated in [this notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb).\n",
"\n",
"This tutorial uses the following Google Cloud services:\n",
"\n",
@@ -109,10 +109,10 @@
"\n",
"The steps performed include:\n",
"\n",
"1. Using `Vision API` to perform Optical Character Recognition (OCR) to extract text from PDF files\n",
"2. Processing the results and saving them to text files\n",
"3. Generating a `Vertex AI AutoML` import file.\n",
"4. Creating a new unlabelled text entity extraction data set in `Vertex AI`."
"1. Using `Vision API` to perform Optical Character Recognition (OCR) to extract text from PDF files.\n",
"2. Processing the results and saving them to text files.\n",
"3. Generating a `Vertex AI Dataset` import file.\n",
"4. Creating a new unlabelled text entity extraction `Vertex AI Dataset` resource in `Vertex AI`."
]
},
{
@@ -657,7 +657,7 @@
"source": [
"## Tutorial\n",
"\n",
"Now you are ready to start creating an unlabelled `Vertex AI AutoML` text entity extraction dataset from PDF files."
"Now you are ready to start creating an unlabelled `Vertex AI Dataset` text entity extraction dataset from PDF files."
]
},
{
@@ -666,11 +666,11 @@
"id": "EurEFM3GBap9"
},
"source": [
"### Convert PDF files to Text using Vision API\n",
"### Convert PDF files to text using Vision API\n",
"\n",
"You will use the Google Cloud Storage bucket that contains the PDFs from the Patent.\n",
"First, you make a `Vision API` request to OCR to text the PDFs from the Patent samples stored in the Cloud Storage bucket.\n",
"\n",
"`Visions API` only allows batches of 100 document submissions at a time, so the source bucket listing will be "
"*Note:* `Visions API` only allows batches of 100 document submissions at a time."
]
},
{
@@ -745,6 +745,31 @@
"print(\"Completed!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7b15473e1937"
},
"source": [
"#### Quick peek at extracted annotated JSON files\n",
"\n",
"Next, you take a peek at the contents of one of the extracted JSON annotated files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4366442c1373"
},
"outputs": [],
"source": [
"json_files = ! gsutil ls {gcs_destination_uri}\n",
"\n",
"example = json_files[0]\n",
"! gsutil cat {example} | head -n 1"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -806,15 +831,39 @@
"print(\"Extraction completed!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0a5aae0eab44"
},
"source": [
"#### Quick peek at extracted text files\n",
"\n",
"Next, you take a peek at the contents of one of the extracted text files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "76ce5f57b1ae"
},
"outputs": [],
"source": [
"example = import_file_entries[0][\"textGcsUri\"]\n",
"\n",
"! gsutil cat {example}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hqTLS_AmLWQP"
},
"source": [
"### Generate and save import file to be used in Vertex AI dataset\n",
"### Generate and save import file to be used in `Vertex AI Dataset` resource\n",
"\n",
"You will now build the import file that will be used to create the `Vertex AI Dataset' resource."
"You will now build the import file that will be used to create the `Vertex AI Dataset` resource."
]
},
{
@@ -848,9 +897,9 @@
"id": "6dVjFftOaKdw"
},
"source": [
"### Create unlabelled dataset\n",
"### Create an unlabelled `Vertex AI Dataset` resource\n",
"\n",
"Now you will create the `Dataset` resource using the `create` method for the `TextDataset` class, which takes the following parameters:\n",
"Next, you create the `Dataset` resource using the `create` method for the `TextDataset` class, which takes the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `Dataset` resource.\n",
"- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.\n",