fix: auto review (#793)

* feat: tune template

* feat: tune template

* fix: auto review

* fix: auto review

* fix: auto review

* fix: auto review

* fix: auti review

* fix: auti review

* fix: auti review

* fix: auti review
This commit is contained in:
Andrew Ferlitsch
2022-08-03 09:22:59 -07:00
committed by GitHub
parent aaa7d5c259
commit 785ca9f864
14 changed files with 325 additions and 201 deletions
@@ -64,17 +64,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Apache Airflow and Vertex AI Pipelines." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Apache Airflow and Vertex AI Pipelines."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is [Condensed Game Data](gs://example-datasets/game_data_condensed.csv), which comes from the [Apache Beam examples](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/game). The version used in this tutorial is stored in a Cloud Storage bucket."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -97,8 +86,26 @@
"- Create Cloud Composer environment.\n", "- Create Cloud Composer environment.\n",
"- Upload Airflow DAG to Composer environment that performs data processing -- i.e., creates a BigQuery table from a CSV file.\n", "- Upload Airflow DAG to Composer environment that performs data processing -- i.e., creates a BigQuery table from a CSV file.\n",
"- Create a `Vertex AI Pipeline` that triggers the Airflow DAG.\n", "- Create a `Vertex AI Pipeline` that triggers the Airflow DAG.\n",
"- Execute the `Vertex AI Pipeline`.\n", "- Execute the `Vertex AI Pipeline`."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n", "\n",
"The dataset used for this tutorial is [Condensed Game Data](gs://example-datasets/game_data_condensed.csv), which comes from the [Apache Beam examples](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/game). The version used in this tutorial is stored in a Cloud Storage bucket."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b8a374d1a7dc"
},
"source": [
"### Costs\n", "### Costs\n",
"\n", "\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
@@ -747,7 +754,7 @@
"source": [ "source": [
"# This code is modified version of https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/rest/get_client_id.py\n", "# This code is modified version of https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/rest/get_client_id.py\n",
"\n", "\n",
"shell_output=! python3 get_composer_config.py $PROJECT_ID $REGION $COMPOSER_ENV_NAME\n", "shell_output = ! python3 get_composer_config.py $PROJECT_ID $REGION $COMPOSER_ENV_NAME\n",
"COMPOSER_WEB_URI = shell_output[0]\n", "COMPOSER_WEB_URI = shell_output[0]\n",
"COMPOSER_DAG_GCS = shell_output[1]\n", "COMPOSER_DAG_GCS = shell_output[1]\n",
"COMPOSER_CLIENT_ID = shell_output[2]\n", "COMPOSER_CLIENT_ID = shell_output[2]\n",
@@ -977,7 +984,7 @@
" dag_name: str,\n", " dag_name: str,\n",
" composer_client_id: str,\n", " composer_client_id: str,\n",
" composer_webserver_id: str,\n", " composer_webserver_id: str,\n",
" response: Output[Artifact]\n", " response: Output[Artifact],\n",
"):\n", "):\n",
" # [START composer_trigger]\n", " # [START composer_trigger]\n",
"\n", "\n",
@@ -988,10 +995,9 @@
" from google.auth.transport.requests import Request\n", " from google.auth.transport.requests import Request\n",
" from google.oauth2 import id_token\n", " from google.oauth2 import id_token\n",
"\n", "\n",
" IAM_SCOPE = \"https://www.googleapis.com/auth/iam\"\n",
" OAUTH_TOKEN_URI = \"https://www.googleapis.com/oauth2/v4/token\"\n",
"\n", "\n",
" IAM_SCOPE = 'https://www.googleapis.com/auth/iam'\n",
" OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'\n",
" \n",
" data = '{\"replace_microseconds\":\"false\"}'\n", " data = '{\"replace_microseconds\":\"false\"}'\n",
" context = None\n", " context = None\n",
"\n", "\n",
@@ -1008,13 +1014,13 @@
" \"\"\"\n", " \"\"\"\n",
"\n", "\n",
" # Form webserver URL to make REST API calls\n", " # Form webserver URL to make REST API calls\n",
" webserver_url = f'{composer_webserver_id}/api/experimental/dags/{dag_name}/dag_runs'\n", " webserver_url = f\"{composer_webserver_id}/api/experimental/dags/{dag_name}/dag_runs\"\n",
" # print(webserver_url)\n", " # print(webserver_url)\n",
"\n", "\n",
" # This code is copied from\n", " # This code is copied from\n",
" # https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py\n", " # https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py\n",
" # START COPIED IAP CODE\n", " # START COPIED IAP CODE\n",
" def make_iap_request(url, client_id, method='GET', **kwargs):\n", " def make_iap_request(url, client_id, method=\"GET\", **kwargs):\n",
" \"\"\"Makes a request to an application protected by Identity-Aware Proxy.\n", " \"\"\"Makes a request to an application protected by Identity-Aware Proxy.\n",
" Args:\n", " Args:\n",
" url: The Identity-Aware Proxy-protected URL to fetch.\n", " url: The Identity-Aware Proxy-protected URL to fetch.\n",
@@ -1028,8 +1034,8 @@
" The page body, or raises an exception if the page couldn't be retrieved.\n", " The page body, or raises an exception if the page couldn't be retrieved.\n",
" \"\"\"\n", " \"\"\"\n",
" # Set the default timeout, if missing\n", " # Set the default timeout, if missing\n",
" if 'timeout' not in kwargs:\n", " if \"timeout\" not in kwargs:\n",
" kwargs['timeout'] = 90\n", " kwargs[\"timeout\"] = 90\n",
"\n", "\n",
" # Obtain an OpenID Connect (OIDC) token from metadata server or using service\n", " # Obtain an OpenID Connect (OIDC) token from metadata server or using service\n",
" # account.\n", " # account.\n",
@@ -1039,32 +1045,41 @@
" # Authorization header containing \"Bearer \" followed by a\n", " # Authorization header containing \"Bearer \" followed by a\n",
" # Google-issued OpenID Connect token for the service account.\n", " # Google-issued OpenID Connect token for the service account.\n",
" resp = requests.request(\n", " resp = requests.request(\n",
" method, url,\n", " method,\n",
" headers={'Authorization': 'Bearer {}'.format(\n", " url,\n",
" google_open_id_connect_token)}, **kwargs)\n", " headers={\"Authorization\": \"Bearer {}\".format(google_open_id_connect_token)},\n",
" **kwargs,\n",
" )\n",
" if resp.status_code == 403:\n", " if resp.status_code == 403:\n",
" raise Exception('Service account does not have permission to '\n", " raise Exception(\n",
" 'access the IAP-protected application.')\n", " \"Service account does not have permission to \"\n",
" \"access the IAP-protected application.\"\n",
" )\n",
" elif resp.status_code != 200:\n", " elif resp.status_code != 200:\n",
" raise Exception(\n", " raise Exception(\n",
" 'Bad response from application: {!r} / {!r} / {!r}'.format(\n", " \"Bad response from application: {!r} / {!r} / {!r}\".format(\n",
" resp.status_code, resp.headers, resp.text))\n", " resp.status_code, resp.headers, resp.text\n",
" )\n",
" )\n",
" else:\n", " else:\n",
" print(f\"response = {resp.text}\")\n", " print(f\"response = {resp.text}\")\n",
" # not executed when testing locally\n", " # not executed when testing locally\n",
" if response:\n", " if response:\n",
" file_path = os.path.join(response.path)\n", " file_path = os.path.join(response.path)\n",
" os.makedirs(file_path)\n", " os.makedirs(file_path)\n",
" with open(os.path.join(file_path, \"airflow_response.json\"), 'w') as f:\n", " with open(os.path.join(file_path, \"airflow_response.json\"), \"w\") as f:\n",
" json.dump(resp.text, f)\n", " json.dump(resp.text, f)\n",
"\n", "\n",
" # END COPIED IAP CODE\n", " # END COPIED IAP CODE\n",
"\n", "\n",
" \n",
" # Make a POST request to IAP which then Triggers the DAG\n", " # Make a POST request to IAP which then Triggers the DAG\n",
" make_iap_request(\n", " make_iap_request(\n",
" webserver_url, composer_client_id, method='POST', json={\"conf\": data, \"replace_microseconds\": 'false'})\n", " webserver_url,\n",
" \n", " composer_client_id,\n",
" method=\"POST\",\n",
" json={\"conf\": data, \"replace_microseconds\": \"false\"},\n",
" )\n",
"\n",
" # [END composer_trigger]" " # [END composer_trigger]"
] ]
}, },
@@ -1094,7 +1109,7 @@
" dag_name=COMPOSER_DAG_NAME,\n", " dag_name=COMPOSER_DAG_NAME,\n",
" composer_client_id=COMPOSER_CLIENT_ID,\n", " composer_client_id=COMPOSER_CLIENT_ID,\n",
" composer_webserver_id=COMPOSER_WEB_URI,\n", " composer_webserver_id=COMPOSER_WEB_URI,\n",
" response=None\n", " response=None,\n",
" )\n", " )\n",
"except Exception as e:\n", "except Exception as e:\n",
" print(e)" " print(e)"
@@ -1121,12 +1136,13 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"PATH=%env PATH\n", "PATH = %env PATH\n",
"%env PATH={PATH}:/home/jupyter/.local/bin\n", "%env PATH={PATH}:/home/jupyter/.local/bin\n",
"\n", "\n",
"PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/\"\n", "PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/\"\n",
"print(PIPELINE_ROOT)\n", "print(PIPELINE_ROOT)\n",
"\n", "\n",
"\n",
"@dsl.pipeline(\n", "@dsl.pipeline(\n",
" name=\"pipeline-trigger-airflow-dag\",\n", " name=\"pipeline-trigger-airflow-dag\",\n",
" description=\"Trigger Airflow DAG from Vertex AI Pipelines\",\n", " description=\"Trigger Airflow DAG from Vertex AI Pipelines\",\n",
@@ -1140,7 +1156,7 @@
" data_processing_task = trigger_airflow_dag(\n", " data_processing_task = trigger_airflow_dag(\n",
" dag_name=data_processing_task_dag_name,\n", " dag_name=data_processing_task_dag_name,\n",
" composer_client_id=COMPOSER_CLIENT_ID,\n", " composer_client_id=COMPOSER_CLIENT_ID,\n",
" composer_webserver_id=COMPOSER_WEB_URI\n", " composer_webserver_id=COMPOSER_WEB_URI,\n",
" )" " )"
] ]
}, },
@@ -1171,9 +1187,8 @@
" display_name=\"airflow_pipeline\",\n", " display_name=\"airflow_pipeline\",\n",
" template_path=\"pipeline-trigger-airflow-dag.json\",\n", " template_path=\"pipeline-trigger-airflow-dag.json\",\n",
" pipeline_root=PIPELINE_ROOT,\n", " pipeline_root=PIPELINE_ROOT,\n",
" parameter_values={\n", " parameter_values={},\n",
" },\n", " enable_caching=False,\n",
" enable_caching=False\n",
")\n", ")\n",
"\n", "\n",
"pipeline.run()\n", "pipeline.run()\n",
@@ -1213,7 +1228,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"COMPOSER_WEB_URI + '/admin/airflow/tree?dag_id=dag_gcs_to_bq_orch'" "COMPOSER_WEB_URI + \"/admin/airflow/tree?dag_id=dag_gcs_to_bq_orch\""
] ]
}, },
{ {
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in the given image from the five classes of flowers: daisy, dandelion, rose, sunflower, or tulip."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -100,8 +89,26 @@
" - Training a Vertex AI AutoML trained model.\n", " - Training a Vertex AI AutoML trained model.\n",
" - Test the serving binary with a batch prediction job.\n", " - Test the serving binary with a batch prediction job.\n",
" - Deploying a Vertex AI AutoML trained model.\n", " - Deploying a Vertex AI AutoML trained model.\n",
"- Execute a Vertex AI pipeline.\n", "- Execute a Vertex AI pipeline.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n", "\n",
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in the given image from the five classes of flowers: daisy, dandelion, rose, sunflower, or tulip."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eef426a35e17"
},
"source": [
"### Costs\n", "### Costs\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
"\n", "\n",
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML Tabular pipeline template." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML Tabular pipeline template."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:iris,lcn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -105,8 +94,26 @@
"- Export AutoML model as an OSS TF model.\n", "- Export AutoML model as an OSS TF model.\n",
"- Create `Endpoint` resource.\n", "- Create `Endpoint` resource.\n",
"- Deploy exported OSS TF model.\n", "- Deploy exported OSS TF model.\n",
"- Make a prediction.\n", "- Make a prediction."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:iris,lcn"
},
"source": [
"### Dataset\n",
"\n", "\n",
"The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). This dataset does not require any feature engineering. The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4fc0ad661ebb"
},
"source": [
"### Costs \n", "### Costs \n",
"\n", "\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery and TFDV pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery and TFDV pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:gsod,lrg"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -99,26 +88,31 @@
"- Execute a Vertex AI pipeline." "- Execute a Vertex AI pipeline."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:gsod,lrg"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
"id": "0c997d8d92ce" "id": "0c997d8d92ce"
}, },
"source": [ "source": [
"### Costs \n", "### Costs\n",
"\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
"\n", "\n",
"* Vertex AI\n", "- Vertex AI\n",
"* Cloud Storage\n", "- Cloud Storage\n",
"- BigQuery\n",
"\n", "\n",
"\n", "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
"Learn about [Vertex AI\n",
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
"Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
] ]
}, },
{ {
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery ML pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery ML pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:penguins,lcn,bq"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -105,26 +94,31 @@
"- Make a prediction with the deployed Vertex AI model." "- Make a prediction with the deployed Vertex AI model."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:penguins,lcn,bq"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
"id": "0c997d8d92ce" "id": "0c997d8d92ce"
}, },
"source": [ "source": [
"### Costs \n", "### Costs\n",
"\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
"\n", "\n",
"* Vertex AI\n", "- Vertex AI\n",
"* Cloud Storage\n", "- Cloud Storage\n",
"- BigQuery\n",
"\n", "\n",
"\n", "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
"Learn about [Vertex AI\n",
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
"Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
] ]
}, },
{ {
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with custom training pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with custom training pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -109,6 +98,17 @@
"- Execute a Vertex AI pipeline." "- Execute a Vertex AI pipeline."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:flowers,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -67,17 +67,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Dataflow pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Dataflow pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:gsod,lrg"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -101,6 +90,39 @@
"- Execute a Vertex AI pipeline." "- Execute a Vertex AI pipeline."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:gsod,lrg"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0c997d8d92ce"
},
"source": [
"### Costs \n",
"\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"\n",
"Learn about [Vertex AI\n",
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
"Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -85,8 +85,26 @@
"- `DataprocPySparkBatchOp` for running PySpark batch workloads.\n", "- `DataprocPySparkBatchOp` for running PySpark batch workloads.\n",
"- `DataprocSparkBatchOp` for running Spark batch workloads.\n", "- `DataprocSparkBatchOp` for running Spark batch workloads.\n",
"- `DataprocSparkSqlBatchOp` for running Spark SQL batch workloads.\n", "- `DataprocSparkSqlBatchOp` for running Spark SQL batch workloads.\n",
"- `DataprocSparkRBatchOp` for running SparkR batch workloads.\n", "- `DataprocSparkRBatchOp` for running SparkR batch workloads."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4ced09c1b4ce"
},
"source": [
"### Dataset\n",
"\n", "\n",
"No dataset is used in this tutorial. References to an example dataset are for demonstration purposes."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "25697c6fccd3"
},
"source": [
"### Costs\n", "### Costs\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
"\n", "\n",
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Hyperparameter Tuning pipeline components." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Hyperparameter Tuning pipeline components."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:horses_or_humans,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Horses or Humans](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The trained model predicts whether an image is a horse or human being."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -100,8 +89,26 @@
" - If the metrics exceed a specified threshold.\n", " - If the metrics exceed a specified threshold.\n",
" - Get the location of the model artifacts for the best tuned model.\n", " - Get the location of the model artifacts for the best tuned model.\n",
" - Upload the model artifacts to a `Vertex AI Model` resource.\n", " - Upload the model artifacts to a `Vertex AI Model` resource.\n",
"- Execute a Vertex AI pipeline.\n", "- Execute a Vertex AI pipeline."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:horses_or_humans,icn"
},
"source": [
"### Dataset\n",
"\n", "\n",
"The dataset used for this tutorial is the [Horses or Humans](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The trained model predicts whether an image is a horse or human being."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5e2eba58ad71"
},
"source": [
"### Costs \n", "### Costs \n",
"\n", "\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
@@ -87,8 +87,26 @@
"- Executing a KFP pipeline using Vertex AI Pipelines.\n", "- Executing a KFP pipeline using Vertex AI Pipelines.\n",
"- Loading component and pipeline definitions from a source code repository.\n", "- Loading component and pipeline definitions from a source code repository.\n",
"- Building sequential, parallel, multiple output components.\n", "- Building sequential, parallel, multiple output components.\n",
"- Building control flow into pipelines.\n", "- Building control flow into pipelines."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4ced09c1b4ce"
},
"source": [
"### Dataset\n",
"\n", "\n",
"No dataset is used in this tutorial. References to an example dataset are for demonstration purposes."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eef426a35e17"
},
"source": [
"### Costs\n", "### Costs\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
"\n", "\n",
@@ -65,18 +65,20 @@
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
"id": "8b10d2fb975d" "id": "c16c6df3c48d"
}, },
"source": [ "source": [
"## Overview\n", "## Overview\n",
"\n", "\n",
"This tutorial demonstrates how to manage machine resources when training as a component in `Vertex AI Pipelines`.\n", "This tutorial demonstrates how to manage machine resources when training as a component in `Vertex AI Pipelines`."
"\n", ]
"\n", },
"### Dataset\n", {
"\n", "cell_type": "markdown",
"The dataset is the MNIST dataset. The dataset consists of 28x28 grayscale images of the digits 0 .. 9. \n", "metadata": {
"\n", "id": "8b10d2fb975d"
},
"source": [
"### Objective\n", "### Objective\n",
"\n", "\n",
"In this tutorial, you convert a self-contained custom training component into a `Vertex AI CustomJob`, whereby:\n", "In this tutorial, you convert a self-contained custom training component into a `Vertex AI CustomJob`, whereby:\n",
@@ -93,8 +95,26 @@
"- Create a custom component with a self-contained training job.\n", "- Create a custom component with a self-contained training job.\n",
"- Execute pipeline using component-level settings for machine resources\n", "- Execute pipeline using component-level settings for machine resources\n",
"- Convert the self-contained training component into a `Vertex AI CustomJob`.\n", "- Convert the self-contained training component into a `Vertex AI CustomJob`.\n",
"- Execute pipeline using customjob-level settings for machine resources \n", "- Execute pipeline using customjob-level settings for machine resources "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "39c8466c1f07"
},
"source": [
"### Dataset\n",
"\n", "\n",
"The dataset is the MNIST dataset. The dataset consists of 28x28 grayscale images of the digits 0 .. 9."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "35bee437737d"
},
"source": [
"### Costs\n", "### Costs\n",
"\n", "\n",
"This tutorial uses billable components of Google Cloud:\n", "This tutorial uses billable components of Google Cloud:\n",
@@ -65,6 +65,33 @@
"<img src=\"https://storage.googleapis.com/rafacarv-public-bucket-do-not-delete/abalone/automl_and_bqml.png\" />" "<img src=\"https://storage.googleapis.com/rafacarv-public-bucket-do-not-delete/abalone/automl_and_bqml.png\" />"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "6c75b63ad57e"
},
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to use `Vertex AI Predictions` for rapid prototyping a model.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Pipelines`\n",
"- `Vertex AI AutoML`\n",
"- `Vertex AI BigQuery ML`\n",
"- `Google Cloud Pipeline Components`\n",
"\n",
"The steps performed include:\n",
"\n",
"- Creating a BigQuery and Vertex AI training dataset.\n",
"- Training a BigQuery ML and AutoML model.\n",
"- Extracting evaluation metrics from the BigQueryML and AutoML models.\n",
"- Selecting the best trained model.\n",
"- Deploying the best trained model.\n",
"- Testing the deployed model infrastructure."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -153,33 +180,6 @@
"</body>\n" "</body>\n"
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "6c75b63ad57e"
},
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to use `Vertex AI Predictions` for rapid prototyping a model.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Pipelines`\n",
"- `Vertex AI AutoML`\n",
"- `Vertex AI BigQuery ML`\n",
"- `Google Cloud Pipeline Components`\n",
"\n",
"The steps performed include:\n",
"\n",
"- Creating a BigQuery and Vertex AI training dataset.\n",
"- Training a BigQuery ML and AutoML model.\n",
"- Extracting evaluation metrics from the BigQueryML and AutoML models.\n",
"- Selecting the best trained model.\n",
"- Deploying the best trained model.\n",
"- Testing the deployed model infrastructure."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -75,17 +75,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with TFX and Vertex AI Pipelines." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with TFX and Vertex AI Pipelines."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:bank,lbn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -112,6 +101,17 @@
"- Execute the pipeline using `Vertex AI Pipelines`." "- Execute the pipeline using `Vertex AI Pipelines`."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:bank,lbn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization." "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:bq,chicago,lbn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Chicago Taxi](https://www.kaggle.com/chicago/chicago-taxi-trips-bq). The version of the dataset you will use in this tutorial is stored in a public BigQuery table. The trained model predicts whether someone would leave a tip for a taxi fare."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {
@@ -169,6 +158,39 @@
" - Deploy the trained `Vertex AI Model` resource to the `Vertex AI Endpoint` resource." " - Deploy the trained `Vertex AI Model` resource to the `Vertex AI Endpoint` resource."
] ]
}, },
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:bq,chicago,lbn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [Chicago Taxi](https://www.kaggle.com/chicago/chicago-taxi-trips-bq). The version of the dataset in this tutorial is stored in a public BigQuery table. The trained model predicts whether someone would leave a tip for a taxi fare."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0c997d8d92ce"
},
"source": [
"### Costs \n",
"\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"\n",
"Learn about [Vertex AI\n",
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
"Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": { "metadata": {