mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
fix: auto review (#793)
* feat: tune template * feat: tune template * fix: auto review * fix: auto review * fix: auto review * fix: auto review * fix: auti review * fix: auti review * fix: auti review * fix: auti review
This commit is contained in:
+54
-39
@@ -64,17 +64,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Apache Airflow and Vertex AI Pipelines."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Apache Airflow and Vertex AI Pipelines."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:flowers,icn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is [Condensed Game Data](gs://example-datasets/game_data_condensed.csv), which comes from the [Apache Beam examples](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/game). The version used in this tutorial is stored in a Cloud Storage bucket."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -97,8 +86,26 @@
|
|||||||
"- Create Cloud Composer environment.\n",
|
"- Create Cloud Composer environment.\n",
|
||||||
"- Upload Airflow DAG to Composer environment that performs data processing -- i.e., creates a BigQuery table from a CSV file.\n",
|
"- Upload Airflow DAG to Composer environment that performs data processing -- i.e., creates a BigQuery table from a CSV file.\n",
|
||||||
"- Create a `Vertex AI Pipeline` that triggers the Airflow DAG.\n",
|
"- Create a `Vertex AI Pipeline` that triggers the Airflow DAG.\n",
|
||||||
"- Execute the `Vertex AI Pipeline`.\n",
|
"- Execute the `Vertex AI Pipeline`."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:flowers,icn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"The dataset used for this tutorial is [Condensed Game Data](gs://example-datasets/game_data_condensed.csv), which comes from the [Apache Beam examples](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/game). The version used in this tutorial is stored in a Cloud Storage bucket."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "b8a374d1a7dc"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs\n",
|
"### Costs\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
@@ -747,7 +754,7 @@
|
|||||||
"source": [
|
"source": [
|
||||||
"# This code is modified version of https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/rest/get_client_id.py\n",
|
"# This code is modified version of https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/rest/get_client_id.py\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shell_output=! python3 get_composer_config.py $PROJECT_ID $REGION $COMPOSER_ENV_NAME\n",
|
"shell_output = ! python3 get_composer_config.py $PROJECT_ID $REGION $COMPOSER_ENV_NAME\n",
|
||||||
"COMPOSER_WEB_URI = shell_output[0]\n",
|
"COMPOSER_WEB_URI = shell_output[0]\n",
|
||||||
"COMPOSER_DAG_GCS = shell_output[1]\n",
|
"COMPOSER_DAG_GCS = shell_output[1]\n",
|
||||||
"COMPOSER_CLIENT_ID = shell_output[2]\n",
|
"COMPOSER_CLIENT_ID = shell_output[2]\n",
|
||||||
@@ -977,7 +984,7 @@
|
|||||||
" dag_name: str,\n",
|
" dag_name: str,\n",
|
||||||
" composer_client_id: str,\n",
|
" composer_client_id: str,\n",
|
||||||
" composer_webserver_id: str,\n",
|
" composer_webserver_id: str,\n",
|
||||||
" response: Output[Artifact]\n",
|
" response: Output[Artifact],\n",
|
||||||
"):\n",
|
"):\n",
|
||||||
" # [START composer_trigger]\n",
|
" # [START composer_trigger]\n",
|
||||||
"\n",
|
"\n",
|
||||||
@@ -988,10 +995,9 @@
|
|||||||
" from google.auth.transport.requests import Request\n",
|
" from google.auth.transport.requests import Request\n",
|
||||||
" from google.oauth2 import id_token\n",
|
" from google.oauth2 import id_token\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
" IAM_SCOPE = \"https://www.googleapis.com/auth/iam\"\n",
|
||||||
|
" OAUTH_TOKEN_URI = \"https://www.googleapis.com/oauth2/v4/token\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
" IAM_SCOPE = 'https://www.googleapis.com/auth/iam'\n",
|
|
||||||
" OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'\n",
|
|
||||||
" \n",
|
|
||||||
" data = '{\"replace_microseconds\":\"false\"}'\n",
|
" data = '{\"replace_microseconds\":\"false\"}'\n",
|
||||||
" context = None\n",
|
" context = None\n",
|
||||||
"\n",
|
"\n",
|
||||||
@@ -1008,13 +1014,13 @@
|
|||||||
" \"\"\"\n",
|
" \"\"\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Form webserver URL to make REST API calls\n",
|
" # Form webserver URL to make REST API calls\n",
|
||||||
" webserver_url = f'{composer_webserver_id}/api/experimental/dags/{dag_name}/dag_runs'\n",
|
" webserver_url = f\"{composer_webserver_id}/api/experimental/dags/{dag_name}/dag_runs\"\n",
|
||||||
" # print(webserver_url)\n",
|
" # print(webserver_url)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # This code is copied from\n",
|
" # This code is copied from\n",
|
||||||
" # https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py\n",
|
" # https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py\n",
|
||||||
" # START COPIED IAP CODE\n",
|
" # START COPIED IAP CODE\n",
|
||||||
" def make_iap_request(url, client_id, method='GET', **kwargs):\n",
|
" def make_iap_request(url, client_id, method=\"GET\", **kwargs):\n",
|
||||||
" \"\"\"Makes a request to an application protected by Identity-Aware Proxy.\n",
|
" \"\"\"Makes a request to an application protected by Identity-Aware Proxy.\n",
|
||||||
" Args:\n",
|
" Args:\n",
|
||||||
" url: The Identity-Aware Proxy-protected URL to fetch.\n",
|
" url: The Identity-Aware Proxy-protected URL to fetch.\n",
|
||||||
@@ -1028,8 +1034,8 @@
|
|||||||
" The page body, or raises an exception if the page couldn't be retrieved.\n",
|
" The page body, or raises an exception if the page couldn't be retrieved.\n",
|
||||||
" \"\"\"\n",
|
" \"\"\"\n",
|
||||||
" # Set the default timeout, if missing\n",
|
" # Set the default timeout, if missing\n",
|
||||||
" if 'timeout' not in kwargs:\n",
|
" if \"timeout\" not in kwargs:\n",
|
||||||
" kwargs['timeout'] = 90\n",
|
" kwargs[\"timeout\"] = 90\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Obtain an OpenID Connect (OIDC) token from metadata server or using service\n",
|
" # Obtain an OpenID Connect (OIDC) token from metadata server or using service\n",
|
||||||
" # account.\n",
|
" # account.\n",
|
||||||
@@ -1039,32 +1045,41 @@
|
|||||||
" # Authorization header containing \"Bearer \" followed by a\n",
|
" # Authorization header containing \"Bearer \" followed by a\n",
|
||||||
" # Google-issued OpenID Connect token for the service account.\n",
|
" # Google-issued OpenID Connect token for the service account.\n",
|
||||||
" resp = requests.request(\n",
|
" resp = requests.request(\n",
|
||||||
" method, url,\n",
|
" method,\n",
|
||||||
" headers={'Authorization': 'Bearer {}'.format(\n",
|
" url,\n",
|
||||||
" google_open_id_connect_token)}, **kwargs)\n",
|
" headers={\"Authorization\": \"Bearer {}\".format(google_open_id_connect_token)},\n",
|
||||||
|
" **kwargs,\n",
|
||||||
|
" )\n",
|
||||||
" if resp.status_code == 403:\n",
|
" if resp.status_code == 403:\n",
|
||||||
" raise Exception('Service account does not have permission to '\n",
|
" raise Exception(\n",
|
||||||
" 'access the IAP-protected application.')\n",
|
" \"Service account does not have permission to \"\n",
|
||||||
|
" \"access the IAP-protected application.\"\n",
|
||||||
|
" )\n",
|
||||||
" elif resp.status_code != 200:\n",
|
" elif resp.status_code != 200:\n",
|
||||||
" raise Exception(\n",
|
" raise Exception(\n",
|
||||||
" 'Bad response from application: {!r} / {!r} / {!r}'.format(\n",
|
" \"Bad response from application: {!r} / {!r} / {!r}\".format(\n",
|
||||||
" resp.status_code, resp.headers, resp.text))\n",
|
" resp.status_code, resp.headers, resp.text\n",
|
||||||
|
" )\n",
|
||||||
|
" )\n",
|
||||||
" else:\n",
|
" else:\n",
|
||||||
" print(f\"response = {resp.text}\")\n",
|
" print(f\"response = {resp.text}\")\n",
|
||||||
" # not executed when testing locally\n",
|
" # not executed when testing locally\n",
|
||||||
" if response:\n",
|
" if response:\n",
|
||||||
" file_path = os.path.join(response.path)\n",
|
" file_path = os.path.join(response.path)\n",
|
||||||
" os.makedirs(file_path)\n",
|
" os.makedirs(file_path)\n",
|
||||||
" with open(os.path.join(file_path, \"airflow_response.json\"), 'w') as f:\n",
|
" with open(os.path.join(file_path, \"airflow_response.json\"), \"w\") as f:\n",
|
||||||
" json.dump(resp.text, f)\n",
|
" json.dump(resp.text, f)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # END COPIED IAP CODE\n",
|
" # END COPIED IAP CODE\n",
|
||||||
"\n",
|
"\n",
|
||||||
" \n",
|
|
||||||
" # Make a POST request to IAP which then Triggers the DAG\n",
|
" # Make a POST request to IAP which then Triggers the DAG\n",
|
||||||
" make_iap_request(\n",
|
" make_iap_request(\n",
|
||||||
" webserver_url, composer_client_id, method='POST', json={\"conf\": data, \"replace_microseconds\": 'false'})\n",
|
" webserver_url,\n",
|
||||||
" \n",
|
" composer_client_id,\n",
|
||||||
|
" method=\"POST\",\n",
|
||||||
|
" json={\"conf\": data, \"replace_microseconds\": \"false\"},\n",
|
||||||
|
" )\n",
|
||||||
|
"\n",
|
||||||
" # [END composer_trigger]"
|
" # [END composer_trigger]"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
@@ -1094,7 +1109,7 @@
|
|||||||
" dag_name=COMPOSER_DAG_NAME,\n",
|
" dag_name=COMPOSER_DAG_NAME,\n",
|
||||||
" composer_client_id=COMPOSER_CLIENT_ID,\n",
|
" composer_client_id=COMPOSER_CLIENT_ID,\n",
|
||||||
" composer_webserver_id=COMPOSER_WEB_URI,\n",
|
" composer_webserver_id=COMPOSER_WEB_URI,\n",
|
||||||
" response=None\n",
|
" response=None,\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"except Exception as e:\n",
|
"except Exception as e:\n",
|
||||||
" print(e)"
|
" print(e)"
|
||||||
@@ -1121,12 +1136,13 @@
|
|||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"PATH=%env PATH\n",
|
"PATH = %env PATH\n",
|
||||||
"%env PATH={PATH}:/home/jupyter/.local/bin\n",
|
"%env PATH={PATH}:/home/jupyter/.local/bin\n",
|
||||||
"\n",
|
"\n",
|
||||||
"PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/\"\n",
|
"PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/\"\n",
|
||||||
"print(PIPELINE_ROOT)\n",
|
"print(PIPELINE_ROOT)\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"\n",
|
||||||
"@dsl.pipeline(\n",
|
"@dsl.pipeline(\n",
|
||||||
" name=\"pipeline-trigger-airflow-dag\",\n",
|
" name=\"pipeline-trigger-airflow-dag\",\n",
|
||||||
" description=\"Trigger Airflow DAG from Vertex AI Pipelines\",\n",
|
" description=\"Trigger Airflow DAG from Vertex AI Pipelines\",\n",
|
||||||
@@ -1140,7 +1156,7 @@
|
|||||||
" data_processing_task = trigger_airflow_dag(\n",
|
" data_processing_task = trigger_airflow_dag(\n",
|
||||||
" dag_name=data_processing_task_dag_name,\n",
|
" dag_name=data_processing_task_dag_name,\n",
|
||||||
" composer_client_id=COMPOSER_CLIENT_ID,\n",
|
" composer_client_id=COMPOSER_CLIENT_ID,\n",
|
||||||
" composer_webserver_id=COMPOSER_WEB_URI\n",
|
" composer_webserver_id=COMPOSER_WEB_URI,\n",
|
||||||
" )"
|
" )"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
@@ -1171,9 +1187,8 @@
|
|||||||
" display_name=\"airflow_pipeline\",\n",
|
" display_name=\"airflow_pipeline\",\n",
|
||||||
" template_path=\"pipeline-trigger-airflow-dag.json\",\n",
|
" template_path=\"pipeline-trigger-airflow-dag.json\",\n",
|
||||||
" pipeline_root=PIPELINE_ROOT,\n",
|
" pipeline_root=PIPELINE_ROOT,\n",
|
||||||
" parameter_values={\n",
|
" parameter_values={},\n",
|
||||||
" },\n",
|
" enable_caching=False,\n",
|
||||||
" enable_caching=False\n",
|
|
||||||
")\n",
|
")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"pipeline.run()\n",
|
"pipeline.run()\n",
|
||||||
@@ -1213,7 +1228,7 @@
|
|||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"COMPOSER_WEB_URI + '/admin/airflow/tree?dag_id=dag_gcs_to_bq_orch'"
|
"COMPOSER_WEB_URI + \"/admin/airflow/tree?dag_id=dag_gcs_to_bq_orch\""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
+19
-12
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:flowers,icn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in the given image from the five classes of flowers: daisy, dandelion, rose, sunflower, or tulip."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -100,8 +89,26 @@
|
|||||||
" - Training a Vertex AI AutoML trained model.\n",
|
" - Training a Vertex AI AutoML trained model.\n",
|
||||||
" - Test the serving binary with a batch prediction job.\n",
|
" - Test the serving binary with a batch prediction job.\n",
|
||||||
" - Deploying a Vertex AI AutoML trained model.\n",
|
" - Deploying a Vertex AI AutoML trained model.\n",
|
||||||
"- Execute a Vertex AI pipeline.\n",
|
"- Execute a Vertex AI pipeline.\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:flowers,icn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower in the given image from the five classes of flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "eef426a35e17"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs\n",
|
"### Costs\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
|||||||
+19
-12
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML Tabular pipeline template."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML Tabular pipeline template."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:iris,lcn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -105,8 +94,26 @@
|
|||||||
"- Export AutoML model as an OSS TF model.\n",
|
"- Export AutoML model as an OSS TF model.\n",
|
||||||
"- Create `Endpoint` resource.\n",
|
"- Create `Endpoint` resource.\n",
|
||||||
"- Deploy exported OSS TF model.\n",
|
"- Deploy exported OSS TF model.\n",
|
||||||
"- Make a prediction.\n",
|
"- Make a prediction."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:iris,lcn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). This dataset does not require any feature engineering. The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "4fc0ad661ebb"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs \n",
|
"### Costs \n",
|
||||||
"\n",
|
"\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
|
|||||||
+16
-22
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery and TFDV pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery and TFDV pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:gsod,lrg"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -99,26 +88,31 @@
|
|||||||
"- Execute a Vertex AI pipeline."
|
"- Execute a Vertex AI pipeline."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:gsod,lrg"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"id": "0c997d8d92ce"
|
"id": "0c997d8d92ce"
|
||||||
},
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"### Costs \n",
|
"### Costs\n",
|
||||||
"\n",
|
|
||||||
"\n",
|
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Vertex AI\n",
|
"- Vertex AI\n",
|
||||||
"* Cloud Storage\n",
|
"- Cloud Storage\n",
|
||||||
|
"- BigQuery\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||||
"Learn about [Vertex AI\n",
|
|
||||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
|
||||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
|
||||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
|
||||||
"to generate a cost estimate based on your projected usage."
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery ML pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with BigQuery ML pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:penguins,lcn,bq"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -105,26 +94,31 @@
|
|||||||
"- Make a prediction with the deployed Vertex AI model."
|
"- Make a prediction with the deployed Vertex AI model."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:penguins,lcn,bq"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset predicts the species."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"id": "0c997d8d92ce"
|
"id": "0c997d8d92ce"
|
||||||
},
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"### Costs \n",
|
"### Costs\n",
|
||||||
"\n",
|
|
||||||
"\n",
|
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Vertex AI\n",
|
"- Vertex AI\n",
|
||||||
"* Cloud Storage\n",
|
"- Cloud Storage\n",
|
||||||
|
"- BigQuery\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||||
"Learn about [Vertex AI\n",
|
|
||||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
|
||||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
|
||||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
|
||||||
"to generate a cost estimate based on your projected usage."
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
+11
-11
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with custom training pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with custom training pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:flowers,icn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -109,6 +98,17 @@
|
|||||||
"- Execute a Vertex AI pipeline."
|
"- Execute a Vertex AI pipeline."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:flowers,icn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
+33
-11
@@ -67,17 +67,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Dataflow pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Dataflow pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:gsod,lrg"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -101,6 +90,39 @@
|
|||||||
"- Execute a Vertex AI pipeline."
|
"- Execute a Vertex AI pipeline."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:gsod,lrg"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of the dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "0c997d8d92ce"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Costs \n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
|
"\n",
|
||||||
|
"* Vertex AI\n",
|
||||||
|
"* Cloud Storage\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"Learn about [Vertex AI\n",
|
||||||
|
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||||
|
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||||
|
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||||
|
"to generate a cost estimate based on your projected usage."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
+19
-1
@@ -85,8 +85,26 @@
|
|||||||
"- `DataprocPySparkBatchOp` for running PySpark batch workloads.\n",
|
"- `DataprocPySparkBatchOp` for running PySpark batch workloads.\n",
|
||||||
"- `DataprocSparkBatchOp` for running Spark batch workloads.\n",
|
"- `DataprocSparkBatchOp` for running Spark batch workloads.\n",
|
||||||
"- `DataprocSparkSqlBatchOp` for running Spark SQL batch workloads.\n",
|
"- `DataprocSparkSqlBatchOp` for running Spark SQL batch workloads.\n",
|
||||||
"- `DataprocSparkRBatchOp` for running SparkR batch workloads.\n",
|
"- `DataprocSparkRBatchOp` for running SparkR batch workloads."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "4ced09c1b4ce"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"No dataset is used in this tutorial. References to an example dataset are for demonstration purposes."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "25697c6fccd3"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs\n",
|
"### Costs\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
|||||||
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Hyperparameter Tuning pipeline components."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with Hyperparameter Tuning pipeline components."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:horses_or_humans,icn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [Horses or Humans](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The trained model predicts whether an image is a horse or human being."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -100,8 +89,26 @@
|
|||||||
" - If the metrics exceed a specified threshold.\n",
|
" - If the metrics exceed a specified threshold.\n",
|
||||||
" - Get the location of the model artifacts for the best tuned model.\n",
|
" - Get the location of the model artifacts for the best tuned model.\n",
|
||||||
" - Upload the model artifacts to a `Vertex AI Model` resource.\n",
|
" - Upload the model artifacts to a `Vertex AI Model` resource.\n",
|
||||||
"- Execute a Vertex AI pipeline.\n",
|
"- Execute a Vertex AI pipeline."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:horses_or_humans,icn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [Horses or Humans](https://www.tensorflow.org/datasets/catalog/horses_or_humans) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The trained model predicts whether an image is a horse or human being."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "5e2eba58ad71"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs \n",
|
"### Costs \n",
|
||||||
"\n",
|
"\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
|
|||||||
@@ -87,8 +87,26 @@
|
|||||||
"- Executing a KFP pipeline using Vertex AI Pipelines.\n",
|
"- Executing a KFP pipeline using Vertex AI Pipelines.\n",
|
||||||
"- Loading component and pipeline definitions from a source code repository.\n",
|
"- Loading component and pipeline definitions from a source code repository.\n",
|
||||||
"- Building sequential, parallel, multiple output components.\n",
|
"- Building sequential, parallel, multiple output components.\n",
|
||||||
"- Building control flow into pipelines.\n",
|
"- Building control flow into pipelines."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "4ced09c1b4ce"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"No dataset is used in this tutorial. References to an example dataset are for demonstration purposes."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "eef426a35e17"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs\n",
|
"### Costs\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
|||||||
@@ -65,18 +65,20 @@
|
|||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"id": "8b10d2fb975d"
|
"id": "c16c6df3c48d"
|
||||||
},
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"## Overview\n",
|
"## Overview\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This tutorial demonstrates how to manage machine resources when training as a component in `Vertex AI Pipelines`.\n",
|
"This tutorial demonstrates how to manage machine resources when training as a component in `Vertex AI Pipelines`."
|
||||||
"\n",
|
]
|
||||||
"\n",
|
},
|
||||||
"### Dataset\n",
|
{
|
||||||
"\n",
|
"cell_type": "markdown",
|
||||||
"The dataset is the MNIST dataset. The dataset consists of 28x28 grayscale images of the digits 0 .. 9. \n",
|
"metadata": {
|
||||||
"\n",
|
"id": "8b10d2fb975d"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Objective\n",
|
"### Objective\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In this tutorial, you convert a self-contained custom training component into a `Vertex AI CustomJob`, whereby:\n",
|
"In this tutorial, you convert a self-contained custom training component into a `Vertex AI CustomJob`, whereby:\n",
|
||||||
@@ -93,8 +95,26 @@
|
|||||||
"- Create a custom component with a self-contained training job.\n",
|
"- Create a custom component with a self-contained training job.\n",
|
||||||
"- Execute pipeline using component-level settings for machine resources\n",
|
"- Execute pipeline using component-level settings for machine resources\n",
|
||||||
"- Convert the self-contained training component into a `Vertex AI CustomJob`.\n",
|
"- Convert the self-contained training component into a `Vertex AI CustomJob`.\n",
|
||||||
"- Execute pipeline using customjob-level settings for machine resources \n",
|
"- Execute pipeline using customjob-level settings for machine resources "
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "39c8466c1f07"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
"\n",
|
"\n",
|
||||||
|
"The dataset is the MNIST dataset. The dataset consists of 28x28 grayscale images of the digits 0 .. 9."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "35bee437737d"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
"### Costs\n",
|
"### Costs\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This tutorial uses billable components of Google Cloud:\n",
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
|
|||||||
+27
-27
@@ -65,6 +65,33 @@
|
|||||||
"<img src=\"https://storage.googleapis.com/rafacarv-public-bucket-do-not-delete/abalone/automl_and_bqml.png\" />"
|
"<img src=\"https://storage.googleapis.com/rafacarv-public-bucket-do-not-delete/abalone/automl_and_bqml.png\" />"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "6c75b63ad57e"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Objective\n",
|
||||||
|
"\n",
|
||||||
|
"In this tutorial, you learn how to use `Vertex AI Predictions` for rapid prototyping a model.\n",
|
||||||
|
"\n",
|
||||||
|
"This tutorial uses the following Google Cloud ML services:\n",
|
||||||
|
"\n",
|
||||||
|
"- `Vertex AI Pipelines`\n",
|
||||||
|
"- `Vertex AI AutoML`\n",
|
||||||
|
"- `Vertex AI BigQuery ML`\n",
|
||||||
|
"- `Google Cloud Pipeline Components`\n",
|
||||||
|
"\n",
|
||||||
|
"The steps performed include:\n",
|
||||||
|
"\n",
|
||||||
|
"- Creating a BigQuery and Vertex AI training dataset.\n",
|
||||||
|
"- Training a BigQuery ML and AutoML model.\n",
|
||||||
|
"- Extracting evaluation metrics from the BigQueryML and AutoML models.\n",
|
||||||
|
"- Selecting the best trained model.\n",
|
||||||
|
"- Deploying the best trained model.\n",
|
||||||
|
"- Testing the deployed model infrastructure."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -153,33 +180,6 @@
|
|||||||
"</body>\n"
|
"</body>\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "6c75b63ad57e"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Objective\n",
|
|
||||||
"\n",
|
|
||||||
"In this tutorial, you learn how to use `Vertex AI Predictions` for rapid prototyping a model.\n",
|
|
||||||
"\n",
|
|
||||||
"This tutorial uses the following Google Cloud ML services:\n",
|
|
||||||
"\n",
|
|
||||||
"- `Vertex AI Pipelines`\n",
|
|
||||||
"- `Vertex AI AutoML`\n",
|
|
||||||
"- `Vertex AI BigQuery ML`\n",
|
|
||||||
"- `Google Cloud Pipeline Components`\n",
|
|
||||||
"\n",
|
|
||||||
"The steps performed include:\n",
|
|
||||||
"\n",
|
|
||||||
"- Creating a BigQuery and Vertex AI training dataset.\n",
|
|
||||||
"- Training a BigQuery ML and AutoML model.\n",
|
|
||||||
"- Extracting evaluation metrics from the BigQueryML and AutoML models.\n",
|
|
||||||
"- Selecting the best trained model.\n",
|
|
||||||
"- Deploying the best trained model.\n",
|
|
||||||
"- Testing the deployed model infrastructure."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
@@ -75,17 +75,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with TFX and Vertex AI Pipelines."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with TFX and Vertex AI Pipelines."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:bank,lbn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -112,6 +101,17 @@
|
|||||||
"- Execute the pipeline using `Vertex AI Pipelines`."
|
"- Execute the pipeline using `Vertex AI Pipelines`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:bank,lbn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
@@ -65,17 +65,6 @@
|
|||||||
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization."
|
"This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"metadata": {
|
|
||||||
"id": "dataset:bq,chicago,lbn"
|
|
||||||
},
|
|
||||||
"source": [
|
|
||||||
"### Dataset\n",
|
|
||||||
"\n",
|
|
||||||
"The dataset used for this tutorial is the [Chicago Taxi](https://www.kaggle.com/chicago/chicago-taxi-trips-bq). The version of the dataset you will use in this tutorial is stored in a public BigQuery table. The trained model predicts whether someone would leave a tip for a taxi fare."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -169,6 +158,39 @@
|
|||||||
" - Deploy the trained `Vertex AI Model` resource to the `Vertex AI Endpoint` resource."
|
" - Deploy the trained `Vertex AI Model` resource to the `Vertex AI Endpoint` resource."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "dataset:bq,chicago,lbn"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Dataset\n",
|
||||||
|
"\n",
|
||||||
|
"The dataset used for this tutorial is the [Chicago Taxi](https://www.kaggle.com/chicago/chicago-taxi-trips-bq). The version of the dataset in this tutorial is stored in a public BigQuery table. The trained model predicts whether someone would leave a tip for a taxi fare."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "0c997d8d92ce"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Costs \n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"This tutorial uses billable components of Google Cloud:\n",
|
||||||
|
"\n",
|
||||||
|
"* Vertex AI\n",
|
||||||
|
"* Cloud Storage\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"Learn about [Vertex AI\n",
|
||||||
|
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||||
|
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||||
|
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||||
|
"to generate a cost estimate based on your projected usage."
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|||||||
Reference in New Issue
Block a user