mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 22:51:56 +00:00
Compare commits
39
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Eric Schmidt
|
037a3b041a | ||
|
Eric Schmidt
|
f55e6c60cd | ||
|
Eric Schmidt
|
2cd815c640 | ||
|
Eric Schmidt
|
b1c0cc9a9d | ||
|
Eric Schmidt
|
af63d1c0e2 | ||
|
Eric Schmidt
|
0e97a83836 | ||
|
Eric Schmidt
|
f6645e0125 | ||
|
Eric Schmidt
|
71e6423066 | ||
|
Eric Schmidt
|
139ed95ffc | ||
|
Eric Schmidt
|
cafd192417 | ||
|
Eric Schmidt
|
0d7ec7cd60 | ||
|
Eric Schmidt
|
7fd934045a | ||
|
Eric Schmidt
|
e9b8aa02e9 | ||
|
Eric Schmidt
|
ac1af33c8c | ||
|
Andrew FerlitschandGitHub
|
5bc18b01e4 | ||
|
Andrew FerlitschandGitHub
|
e98b9d6eb4 | ||
|
Andrew FerlitschandGitHub
|
5e6b8bf597 | ||
|
Andrew FerlitschandGitHub
|
5b39e7d995 | ||
|
|
04c6ff4ec7 | ||
|
|
5586fd7c4d | ||
|
gericdongandGitHub
|
30e747b966 | ||
|
Andrew FerlitschandGitHub
|
080e2b5bb5 | ||
|
|
60e4416a7e | ||
|
gericdongandGitHub
|
b207b270b4 | ||
|
Andrew FerlitschandGitHub
|
497e93aba1 | ||
|
|
9a6c36d016 | ||
|
Andrew FerlitschandGitHub
|
c180408f41 | ||
|
gericdongandGitHub
|
869b19d342 | ||
|
Michael HuandGitHub
|
3d967b180d | ||
|
halio-gandGitHub
|
49710a9225 | ||
|
Andrew FerlitschandGitHub
|
9d31463585 | ||
|
Mend RenovateandGitHub
|
9335ea3591 | ||
|
Andrew FerlitschandGitHub
|
0f7343feee | ||
|
Ivan NardiniandGitHub
|
12cd965ce6 | ||
|
|
32632711ff | ||
|
junkourataandGitHub
|
d676d87cce | ||
|
Douglass ChenandGitHub
|
153a8044b8 | ||
|
Andrew FerlitschandGitHub
|
c0d9250416 | ||
|
|
fd16e39f91 |
@@ -103,7 +103,6 @@ class EndpointResourceCleanupManager(VertexAIResourceCleanupManager):
|
||||
models.id for models in resource._gca_resource.deployed_models
|
||||
]:
|
||||
resource._undeploy(deployed_model_id=deployed_model_id)
|
||||
|
||||
resource.delete(force=True)
|
||||
|
||||
|
||||
@@ -117,3 +116,7 @@ class MatchingEngineIndexResourceCleanupManager(VertexAIResourceCleanupManager):
|
||||
|
||||
class MatchingEngineIndexEndpointResourceCleanupManager(VertexAIResourceCleanupManager):
|
||||
vertex_ai_resource = aiplatform.MatchingEngineIndexEndpoint
|
||||
|
||||
def delete(self, resource):
|
||||
resource.undeploy_all()
|
||||
resource.delete(force=True)
|
||||
@@ -5,6 +5,6 @@ nbconvert
|
||||
black==22.10.0
|
||||
pyupgrade==2.38.4
|
||||
isort==5.10.1
|
||||
flake8==4.0.1
|
||||
flake8==6.0.0
|
||||
nbqa==1.5.3
|
||||
|
||||
|
||||
@@ -40,4 +40,4 @@
|
||||
/notebooks/community/pipelines/google_cloud_pipeline_components_bqml_pipeline_anomaly_detection.ipynb @inardini
|
||||
/notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb @Narwhalprime
|
||||
/notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb @Narwhalprime
|
||||
|
||||
/notebooks/community/feature_store/get_started_vertex_feature_store.ipynb @junkourata
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+95
-41
@@ -54,18 +54,20 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
"id": "239ba71252d3"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook shows how to use `Vertex AI Pipelines` and `BigQuery ML pipeline components` to train and evaluate a demand forecasting model.\n",
|
||||
"\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset is a modified version of the dataset in [Build and visualize demand forecast predictions using Datastream, Dataflow, BigQuery ML, and Looker\n",
|
||||
"](https://cloud.google.com/architecture/build-visualize-demand-forecast-prediction-datastream-dataflow-bigqueryml-looker) solution architecture\n",
|
||||
"\n",
|
||||
"This notebook shows how to use `Vertex AI Pipelines` and `BigQuery ML pipeline components` to train and evaluate a demand forecasting model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "25c28706c23e"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to train and evaluate a BigQuery ML model using Vertex AI Pipelines and BigQuery ML pipeline components. \n",
|
||||
@@ -87,8 +89,27 @@
|
||||
" - Generate the ARIMA Plus forecasts\n",
|
||||
" - Generate the ARIMA PLUS forecast explainations\n",
|
||||
"- Compile the pipeline.\n",
|
||||
"- Execute the pipeline.\n",
|
||||
"- Execute the pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "586acfa9b502"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset is a modified version of the dataset in [Build and visualize demand forecast predictions using Datastream, Dataflow, BigQuery ML, and Looker\n",
|
||||
"](https://cloud.google.com/architecture/build-visualize-demand-forecast-prediction-datastream-dataflow-bigqueryml-looker) solution architecture\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
@@ -352,9 +373,8 @@
|
||||
"id": "06571eb4063b"
|
||||
},
|
||||
"source": [
|
||||
"#### Timestamp\n",
|
||||
"\n",
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
|
||||
"#### UUID\n",
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -365,9 +385,16 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"import random\n",
|
||||
"import string\n",
|
||||
"\n",
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
"\n",
|
||||
"# Generate a uuid of a specifed length(default=8)\n",
|
||||
"def generate_uuid(length: int = 8) -> str:\n",
|
||||
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"UUID = generate_uuid()"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -485,7 +512,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
|
||||
" BUCKET_NAME = PROJECT_ID + \"-aip-\" + TIMESTAMP\n",
|
||||
" BUCKET_NAME = PROJECT_ID + \"-aip-\" + UUID\n",
|
||||
" BUCKET_URI = f\"gs://{BUCKET_NAME}\""
|
||||
]
|
||||
},
|
||||
@@ -706,6 +733,7 @@
|
||||
"KFP_COMPONENTS_PATH = \"components\"\n",
|
||||
"PIPELINES_PATH = \"pipelines\"\n",
|
||||
"\n",
|
||||
"! mkdir -m 777 -p {DATA_PATH}\n",
|
||||
"! mkdir -m 777 -p {KFP_COMPONENTS_PATH}\n",
|
||||
"! mkdir -m 777 -p {PIPELINES_PATH}"
|
||||
]
|
||||
@@ -771,7 +799,7 @@
|
||||
" --location={LOCATION} \\\n",
|
||||
" --source_format=CSV \\\n",
|
||||
" --skip_leading_rows=1\\\n",
|
||||
" fast_fresh.orders_{TIMESTAMP} \\\n",
|
||||
" fast_fresh.orders_{UUID} \\\n",
|
||||
" {RAW_DATA_URI} \\\n",
|
||||
" time_of_sale:DATETIME,order_id:INTEGER,product_name:STRING,price:NUMERIC,quantity:NUMERIC,payment_method:STRING,store_id:INTEGER,user_id:INTEGER"
|
||||
]
|
||||
@@ -782,7 +810,7 @@
|
||||
"id": "ZrgOD30o7HcL"
|
||||
},
|
||||
"source": [
|
||||
"## BQML Training Formalization\n",
|
||||
"## BigQuery ML Training Formalization\n",
|
||||
"\n",
|
||||
"In the next cells, you build the components and pipeline to train and evaluate the BQML demand forecasting model."
|
||||
]
|
||||
@@ -820,13 +848,13 @@
|
||||
"BQ_EVALUATE_MODEL_TABLE_PREFIX = \"orders_arima_model_evaluate\"\n",
|
||||
"BQ_FORECAST_TABLE_PREFIX = \"orders_arima_forecast\"\n",
|
||||
"BQ_EXPLAIN_FORECAST_TABLE_PREFIX = \"orders_arima_explain_forecast\"\n",
|
||||
"BQ_ORDERS_TABLE = f\"{BQ_ORDERS_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_TRAINING_TABLE = f\"{BQ_TRAINING_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_MODEL_TABLE = f\"{BQ_MODEL_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_EVALUATE_TS_TABLE = f\"{BQ_EVALUATE_TS_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_EVALUATE_MODEL_TABLE = f\"{BQ_EVALUATE_MODEL_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_FORECAST_TABLE = f\"{BQ_FORECAST_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_EXPLAIN_FORECAST_TABLE = f\"{BQ_EXPLAIN_FORECAST_TABLE_PREFIX}_{TIMESTAMP}\"\n",
|
||||
"BQ_ORDERS_TABLE = f\"{BQ_ORDERS_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_TRAINING_TABLE = f\"{BQ_TRAINING_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_MODEL_TABLE = f\"{BQ_MODEL_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_EVALUATE_TS_TABLE = f\"{BQ_EVALUATE_TS_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_EVALUATE_MODEL_TABLE = f\"{BQ_EVALUATE_MODEL_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_FORECAST_TABLE = f\"{BQ_FORECAST_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"BQ_EXPLAIN_FORECAST_TABLE = f\"{BQ_EXPLAIN_FORECAST_TABLE_PREFIX}_{UUID}\"\n",
|
||||
"\n",
|
||||
"BQ_TRAIN_CONFIGURATION = {\n",
|
||||
" \"destinationTable\": {\n",
|
||||
@@ -1022,7 +1050,7 @@
|
||||
"id": "pcSL1FHk69KT"
|
||||
},
|
||||
"source": [
|
||||
"### Build the BQML training pipeline\n",
|
||||
"### Build the BigQuery ML training pipeline\n",
|
||||
"\n",
|
||||
"Define your workflow using Kubeflow Pipelines DSL package. \n",
|
||||
"\n",
|
||||
@@ -1094,8 +1122,8 @@
|
||||
" location=location,\n",
|
||||
" ).set_display_name(\"get train data\")\n",
|
||||
"\n",
|
||||
" # Train the ARIMA PLUS model\n",
|
||||
" bq_arima_model_op = (\n",
|
||||
" # Run an ARIMA PLUS experiment\n",
|
||||
" bq_arima_model_exp_op = (\n",
|
||||
" BigqueryCreateModelJobOp(\n",
|
||||
" query=f\"\"\"\n",
|
||||
" -- create model table\n",
|
||||
@@ -1104,10 +1132,7 @@
|
||||
" MODEL_TYPE = \\'ARIMA_PLUS\\',\n",
|
||||
" TIME_SERIES_TIMESTAMP_COL = \\'hourly_timestamp\\',\n",
|
||||
" TIME_SERIES_DATA_COL = \\'total_sold\\',\n",
|
||||
" TIME_SERIES_ID_COL = [\\'product_name\\'],\n",
|
||||
" MODEL_REGISTRY = \\'vertex_ai\\',\n",
|
||||
" VERTEX_AI_MODEL_ID = \\'order_demand_forecasting\\',\n",
|
||||
" VERTEX_AI_MODEL_VERSION_ALIASES = [\\'staging\\']\n",
|
||||
" TIME_SERIES_ID_COL = [\\'product_name\\']\n",
|
||||
" ) AS\n",
|
||||
" SELECT\n",
|
||||
" hourly_timestamp,\n",
|
||||
@@ -1119,7 +1144,7 @@
|
||||
" project=project,\n",
|
||||
" location=location,\n",
|
||||
" )\n",
|
||||
" .set_display_name(\"train arima plus model\")\n",
|
||||
" .set_display_name(\"run arima+ model experiment\")\n",
|
||||
" .after(create_training_dataset_op)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
@@ -1128,12 +1153,12 @@
|
||||
" BigqueryMLArimaEvaluateJobOp(\n",
|
||||
" project=project,\n",
|
||||
" location=location,\n",
|
||||
" model=bq_arima_model_op.outputs[\"model\"],\n",
|
||||
" model=bq_arima_model_exp_op.outputs[\"model\"],\n",
|
||||
" show_all_candidate_models=False,\n",
|
||||
" job_configuration_query=bq_evaluate_time_series_configuration,\n",
|
||||
" )\n",
|
||||
" .set_display_name(\"evaluate arima plus time series\")\n",
|
||||
" .after(bq_arima_model_op)\n",
|
||||
" .after(bq_arima_model_exp_op)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Evaluate ARIMA Plus model\n",
|
||||
@@ -1141,12 +1166,12 @@
|
||||
" BigqueryEvaluateModelJobOp(\n",
|
||||
" project=project,\n",
|
||||
" location=location,\n",
|
||||
" model=bq_arima_model_op.outputs[\"model\"],\n",
|
||||
" model=bq_arima_model_exp_op.outputs[\"model\"],\n",
|
||||
" query_statement=f\"\"\"SELECT * FROM `{project}.{bq_dataset}.{bq_training_table}` WHERE split='TEST'\"\"\",\n",
|
||||
" job_configuration_query=bq_evaluate_model_configuration,\n",
|
||||
" )\n",
|
||||
" .set_display_name(\"evaluate arima plus model\")\n",
|
||||
" .after(bq_arima_model_op)\n",
|
||||
" .after(bq_arima_model_exp_op)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Plot model metrics\n",
|
||||
@@ -1164,6 +1189,34 @@
|
||||
" < PERF_THRESHOLD,\n",
|
||||
" name=\"avg. mae good\",\n",
|
||||
" ):\n",
|
||||
" # Train the ARIMA PLUS model\n",
|
||||
" bq_arima_model_op = (\n",
|
||||
" BigqueryCreateModelJobOp(\n",
|
||||
" query=f\"\"\"\n",
|
||||
" -- create model table\n",
|
||||
" CREATE OR REPLACE MODEL `{project}.{bq_dataset}.{bq_model_table}`\n",
|
||||
" OPTIONS(\n",
|
||||
" MODEL_TYPE = \\'ARIMA_PLUS\\',\n",
|
||||
" TIME_SERIES_TIMESTAMP_COL = \\'hourly_timestamp\\',\n",
|
||||
" TIME_SERIES_DATA_COL = \\'total_sold\\',\n",
|
||||
" TIME_SERIES_ID_COL = [\\'product_name\\'],\n",
|
||||
" MODEL_REGISTRY = \\'vertex_ai\\',\n",
|
||||
" VERTEX_AI_MODEL_ID = \\'order_demand_forecasting\\',\n",
|
||||
" VERTEX_AI_MODEL_VERSION_ALIASES = [\\'staging\\']\n",
|
||||
" ) AS\n",
|
||||
" SELECT\n",
|
||||
" DATETIME_TRUNC(time_of_sale, HOUR) as hourly_timestamp,\n",
|
||||
" product_name,\n",
|
||||
" SUM(quantity) AS total_sold,\n",
|
||||
" FROM `{project}.{bq_dataset}.{bq_orders_table}`\n",
|
||||
" GROUP BY hourly_timestamp, product_name;\n",
|
||||
" \"\"\",\n",
|
||||
" project=project,\n",
|
||||
" location=location,\n",
|
||||
" )\n",
|
||||
" .set_display_name(\"train arima+ model\")\n",
|
||||
" .after(get_evaluation_model_metrics_op)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Generate the ARIMA PLUS forecasts\n",
|
||||
" bq_arima_forecast_op = (\n",
|
||||
@@ -1224,7 +1277,7 @@
|
||||
"source": [
|
||||
"### Execute your pipeline\n",
|
||||
"\n",
|
||||
"Next, you execute the pipeline. It takes the following parameters which we set as default:\n",
|
||||
"Next, we execute the pipeline. It takes the following parameters which we set as default:\n",
|
||||
"\n",
|
||||
"- `bq_dataset`: The BigQuery dataset to train on.\n",
|
||||
"- `bq_orders_table` : The BigQuery table of raw data.\n",
|
||||
@@ -1266,7 +1319,7 @@
|
||||
"source": [
|
||||
"### View BigQuery ML training pipeline results\n",
|
||||
"\n",
|
||||
"Finally, you will view the artifact outputs of each task in the pipeline."
|
||||
"Finally, you view the artifact outputs of each task in the pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1342,8 +1395,8 @@
|
||||
"print(\"bigquery-ml-arima-evaluate-job\")\n",
|
||||
"artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-ml-arima-evaluate-job\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"get-model-evaluation-metrics\")\n",
|
||||
"artifacts = print_pipeline_output(bqml_pipeline, \"get-model-evaluation-metrics\")\n",
|
||||
"print(\"bigquery-evaluate-model-job\")\n",
|
||||
"artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-evaluate-model-job\")\n",
|
||||
"print(\"\\n\\n\")\n",
|
||||
"print(\"bigquery-forecast-model-job\")\n",
|
||||
"artifacts = print_pipeline_output(bqml_pipeline, \"bigquery-forecast-model-job\")\n",
|
||||
@@ -1407,7 +1460,8 @@
|
||||
"\n",
|
||||
"# Remove local resorces\n",
|
||||
"! rm -rf {KFP_COMPONENTS_PATH}\n",
|
||||
"! rm -rf {PIPELINES_PATH}"
|
||||
"! rm -rf {PIPELINES_PATH}\n",
|
||||
"! rm -rf {DATA_PATH}"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+5
-5
@@ -42,12 +42,12 @@
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/natural_language/cloud_natural_language_pipeline.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/natural_language/cloud_natural_language_pipeline.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_cloud_natural_language_pipeline.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
@@ -328,7 +328,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"cloud-ml-language-test\" # @param {type:\"string\"}\n",
|
||||
"PROJECT_ID = \"your-project-id\" # @param {type:\"string\"}\n",
|
||||
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
|
||||
" # Get your GCP project id from gcloud\n",
|
||||
" shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
@@ -368,7 +368,7 @@
|
||||
"source": [
|
||||
"REGION = \"us\" # @param {type:\"string\"}\n",
|
||||
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
||||
"TRAINING_DATA_LOCATION = \"gs://dougchen-20221130-pipeline-colab-test/data-00001-of-00001.jsonl\" # @param {type:\"string\"}\n",
|
||||
"TRAINING_DATA_LOCATION = \"gs://your-training-data-location\" # @param {type:\"string\"}\n",
|
||||
"TASK_TYPE = \"CLASSIFICATION\" # @param [\"CLASSIFICATION\", \"MULTILABEL_CLASSIFICATION\"]"
|
||||
]
|
||||
},
|
||||
@@ -740,7 +740,7 @@
|
||||
"\n",
|
||||
"This sends a create pipeline job request to Vertex Pipelines. Note that this task run synchronously and may take a while to complete.\n",
|
||||
"\n",
|
||||
"You may view the progress of the job at any time by clicking on the generated links (after \"View Pipeline Job\" in the console output of the cell below). Once the pipeline finishes, you may examine the artifacts produced from this pipeline. See "
|
||||
"You may view the progress of the job at any time by clicking on the generated links (after \"View Pipeline Job\" in the console output of the cell below). Once the pipeline finishes, you may examine the artifacts produced from this pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+3
-3
@@ -42,12 +42,12 @@
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/community-content/ready_to_go_text_classification_pipeline/ready_to_go_text_classification_pipeline.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/community-content/ready_to_go_text_classification_pipeline/ready_to_go_text_classification_pipeline.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/pipelines/google_cloud_pipeline_components_ready_to_go_text_classification_pipeline.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
@@ -706,7 +706,7 @@
|
||||
")\n",
|
||||
"\n",
|
||||
"# The GCS directory for keeping staging files for model evaluation.\n",
|
||||
"ROOT_DIR = 'f\"{BASE_OUTPUT_DIR}/root\"' # @param {type:\"string\"}"
|
||||
"ROOT_DIR = \"'f\\\"{BASE_OUTPUT_DIR}/root\\\"'\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -695,7 +695,26 @@ class ObjectiveRule(NotebookRule):
|
||||
ret = notebook.report_error(ErrorCode.ERROR_OBJECTIVE_MISSING_DESC, "Objective section missing desc")
|
||||
else:
|
||||
self.desc = self.desc.lstrip()
|
||||
sentences = self.desc.split('.')
|
||||
|
||||
bracket = False
|
||||
paren = False
|
||||
sentences = ""
|
||||
for _ in range(len(self.desc)):
|
||||
if self.desc[_] == '[':
|
||||
bracket = True
|
||||
continue
|
||||
elif self.desc[_] == ']':
|
||||
bracket = False
|
||||
continue
|
||||
elif self.desc[_] == '(':
|
||||
paren = True
|
||||
elif self.desc[_] == ')':
|
||||
paren = False
|
||||
continue
|
||||
|
||||
if not paren:
|
||||
sentences += self.desc[_]
|
||||
sentences = sentences.split('.')
|
||||
if len(sentences) > 1:
|
||||
self.desc = sentences[0] + '.\n'
|
||||
if self.desc.startswith('In this tutorial, you learn') or self.desc.startswith('In this notebook, you learn'):
|
||||
@@ -1121,32 +1140,49 @@ def add_index(path: str,
|
||||
print(f' {tag.strip()}<br/>\n')
|
||||
print(' </td>')
|
||||
print(' <td>')
|
||||
print(f' <b>{title}</b><br/>\n')
|
||||
print(f' <b>{title}</b>.\n')
|
||||
if args.desc:
|
||||
desc = replace_cl(desc.replace('`', ''))
|
||||
print('<br/>')
|
||||
print(f' {desc}<br/>\n')
|
||||
|
||||
if args.steps:
|
||||
steps = replace_cl(steps.replace('\n', '<br/>').replace('-', ' -').replace('**', '').replace('*', ' -').replace('`', ''))
|
||||
print('<br/>' + steps + '<br/>')
|
||||
print("<devsite-expandable>\n")
|
||||
print(' <h6 class="showalways">Notebook steps</h6>\n')
|
||||
print(' <ul style="font-size: .75em">\n')
|
||||
|
||||
if ":" in steps:
|
||||
steps = steps.split(':')[1].replace('*', '').replace('-', '').strip().split('\n')
|
||||
else:
|
||||
steps = []
|
||||
|
||||
for step in steps:
|
||||
print(f' <li>{replace_cl(step)}</li>\n')
|
||||
#steps = replace_cl(steps.replace('\n', '<br/>').replace('-', ' -').replace('**', '').replace('*', ' -').replace('`', ''))
|
||||
#print('<br/>' + steps + '<br/>')
|
||||
print(' </ul>\n')
|
||||
print("</devsite-expandable>\n")
|
||||
|
||||
if args.linkback and linkbacks:
|
||||
num = len(tags)
|
||||
for _ in range(num):
|
||||
if linkbacks[_].startswith("vertex-ai"):
|
||||
print(f'<br/> Learn more about <a href="https://cloud.google.com/{linkbacks[_]}" target="_blank">{replace_cl(tags[_])}</a>.\n')
|
||||
print(f' Learn more about <a href="https://cloud.google.com/{linkbacks[_]}." target="_blank">{replace_cl(tags[_])}</a>.\n')
|
||||
else:
|
||||
print(f'<br/> Learn more about <a href="{linkbacks[_]}" target="_blank">{replace_cl(tags[_])}</a>.\n')
|
||||
print(f' Learn more about <a href="{linkbacks[_]}." target="_blank">{replace_cl(tags[_])}</a>.\n')
|
||||
|
||||
if args.steps:
|
||||
steps = replace_cl(steps.replace('\n', '<br/>').replace('-', ' -').replace('**', '').replace('*', ' -').replace('`', ''))
|
||||
print('<br/><br/>' + steps + '<br/>')
|
||||
|
||||
print(' </td>')
|
||||
print(' <td>')
|
||||
if colab_link:
|
||||
print(f' <a href="{colab_link}" target="_blank">Colab</a><br/>\n')
|
||||
print(f' <a href="{colab_link}" target="_blank" class="external" track-type="notebookTutorial" track-name="colabLink">Colab</a><br/>\n')
|
||||
if git_link:
|
||||
print(f' <a href="{git_link}" target="_blank">GitHub</a><br/>\n')
|
||||
print(f' <a href="{git_link}" target="_blank" class="external" track-type="notebookTutorial" track-name="gitHubLink">GitHub</a><br/>\n')
|
||||
if workbench_link:
|
||||
print(f' <a href="{workbench_link}" target="_blank">Vertex AI Workbench</a><br/>\n')
|
||||
print(f' <a href="{workbench_link}" target="_blank" class="external" track-type="notebookTutorial" track-name="workbenchLink">Vertex AI Workbench</a><br/>\n')
|
||||
print(' </td>')
|
||||
print(' </tr>\n')
|
||||
elif args.repo:
|
||||
@@ -1215,8 +1251,8 @@ def replace_cl(text : str ) -> str:
|
||||
'Vertex AI Data Labeling': '{{vertex_data_labeling_name}}',
|
||||
'Vertex AI Experiments': '{{vertex_experiments_name}}',
|
||||
'Vertex Experiments': '{{vertex_experiments_name}}',
|
||||
'Vertex AI Matching Engine': '{vertex_matching_engine_name}}',
|
||||
'Vertex Matching Engine': '{vertex_matching_engine_name}}',
|
||||
'Vertex AI Matching Engine': '{{vertex_matching_engine_name}}',
|
||||
'Vertex Matching Engine': '{{vertex_matching_engine_name}}',
|
||||
'Vertex Model Monitoring': '{{vertex_model_monitoring_name}}',
|
||||
'Vertex AI Model Monitoring': '{{vertex_model_monitoring_name}}',
|
||||
'Vertex Feature Store': '{{vertex_featurestore_name}}',
|
||||
@@ -1282,9 +1318,14 @@ if args.web:
|
||||
print('}')
|
||||
print('</style>')
|
||||
print('<table>')
|
||||
print(' <th width="180px">Services</th>')
|
||||
print(' <th>Description</th>')
|
||||
print(' <th width="80px">Open in</th>')
|
||||
print(' <thead>')
|
||||
print(' <tr>')
|
||||
print(' <th width="180px">Services</th>')
|
||||
print(' <th>Description</th>')
|
||||
print(' <th width="80px">Open in</th>')
|
||||
print(' </tr>')
|
||||
print(' </thead>')
|
||||
print(' <tbody>')
|
||||
|
||||
if args.notebook_dir:
|
||||
if not os.path.isdir(args.notebook_dir):
|
||||
@@ -1320,6 +1361,7 @@ else:
|
||||
exit(1)
|
||||
|
||||
if args.web:
|
||||
print(' </tbody>\n')
|
||||
print('</table>\n')
|
||||
|
||||
exit(exit_code)
|
||||
|
||||
@@ -29,15 +29,14 @@
|
||||
/pipelines/google_cloud_pipelines_dataproc_tabular @inardini
|
||||
/automl/automl_forecasting_bqml_arima_plus_comparison.ipynb @TheMichaelHu
|
||||
/automl/automl_tabular_on_vertex_pipelines.ipynb @helinwang
|
||||
/custom/custom_training_tensorboard_profiler.ipynb @itseric
|
||||
/custom/custom_training_tensorboard_profiler.ipynb @gericdong
|
||||
/custom/get_started_with_vertex_endpoint_and_shared_vm.ipynb @andrewferlitsch
|
||||
/workbench/spark/spark_sample_notebook.ipynb @bradmiro
|
||||
/workbench/spark/spark_ml.ipynb @bradmiro
|
||||
/model_registry/bqml_vertexai_model_registry.ipynb @soheilazangeneh
|
||||
/workbench/exploratory_data_analysis/explore_data_in_bigquery_with_workbench.ipynb @alokpattani
|
||||
/model_evaluation/automl_tabular_classification_model_evaluation.ipynb @soheilazangeneh
|
||||
/model_evaluation/automl_tabular_regression_model_evaluation.ipynb @soheilazangeneh
|
||||
/tabular_workflows/tabnet_on_vertex_pipelines.ipynb @sakagarwal
|
||||
/tabular_workflows/wide_and_deep_on_vertex_pipelines.ipynb @sakagarwal
|
||||
/tabular_workflows/prophet_on_vertex_pipelines.ipynb @TheMichaelHu
|
||||
/model_evaluation/custom_tabular_classification_model_evaluation.ipynb @soheilazangeneh
|
||||
/sdk/SDK_FBProphet_Forecasting_Online.ipynb @brianchunkang
|
||||
/automl/sdk_automl_forecasting_hierarchical_batch.ipynb @ivanmkc
|
||||
|
||||
@@ -833,6 +833,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "99b7a9287ba6"
|
||||
@@ -840,7 +841,7 @@
|
||||
"source": [
|
||||
"For AutoML models, manual scaling can be adjusted by setting both min and max nodes i.e., `starting_replica_count` and `max_replica_count` as the same value(in this example, set to 1). The node count can be increased or decreased as required by load.\n",
|
||||
" \n",
|
||||
"`batch_predict` can export predictions either to BigQuery or GCS. The BigQuery options are commented out below and the predictions will be exported to the BUCKET_URI."
|
||||
"`batch_predict` can export predictions either to BigQuery or GCS. This example exports to BigQuery."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -352,7 +352,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"EMAIL = \"[your-email-address]\" # @param {type: \"string\"}"
|
||||
"EMAIL = \"[your-email-address]\" # @param {type: \"string\"}\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
" EMAIL = \"noreply@google.com\""
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -64,7 +64,7 @@
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use the Vertex AI SDK to create tabular binary classification models and do batch prediction with explanation using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model.\n",
|
||||
"\n",
|
||||
"Learn more about [AutoML Tabular](https://cloud.google.com/vertex-ai/docs/tabular-data/overview) and [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
|
||||
"Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview). Learn more about [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+1045
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -64,7 +64,7 @@
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use the Vertex AI SDK to create tabular binary classification models and do online prediction using a Google Cloud [AutoML](https://cloud.google.com/vertex-ai/docs/start/automl-users) model.\n",
|
||||
"\n",
|
||||
"Learn more about [Migrate to Vertex AI](https://cloud.google.com/vertex-ai/docs/start/migrating-to-vertex-ai) and [AutoML Tabular](https://cloud.google.com/vertex-ai/docs/start/automl-users#tables)."
|
||||
"Learn more about [Migrate to Vertex AI](https://cloud.google.com/vertex-ai/docs/start/migrating-to-vertex-ai). Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+1
-1
@@ -63,7 +63,7 @@
|
||||
"\n",
|
||||
"This notebook demonstrates how to use the Vertex AI classification model evaluation component to evaluate an AutoML Tabular classification model. Model evaluation helps determine your model's performance based on the evaluation metrics and improve the model whenever necessary. \n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction) and [AutoML Tabular](https://cloud.google.com/vertex-ai/docs/start/automl-users#tables)."
|
||||
"Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction). Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+1
-1
@@ -63,7 +63,7 @@
|
||||
"\n",
|
||||
"This notebook demonstrates how to use Vertex AI regression model evaluation component to evaluate an AutoML Tabular regression model. Model evaluation helps you determine your model performance based on the evaluation metrics and improve the model if necessary. \n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction) and [AutoML Tabular](https://cloud.google.com/vertex-ai/docs/start/automl-users#tables)."
|
||||
"Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction). Learn more about [Regression for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+27
-61
@@ -31,7 +31,7 @@
|
||||
"id": "fsv4jGuU89rX"
|
||||
},
|
||||
"source": [
|
||||
"# E2E ML on GCP: MLOps stage 7 : monitoring: Vertex AI Model Monitoring for AutoML tabular models\n",
|
||||
"# Vertex AI Model Monitoring for AutoML tabular models\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
" <td>\n",
|
||||
@@ -91,7 +91,6 @@
|
||||
"- Deploy the `Model` resource to the `Endpoint` resource.\n",
|
||||
"- Configure the `Endpoint` resource for model monitoring.\n",
|
||||
"- Generate synthetic prediction requests for skew.\n",
|
||||
"- Wait for email alert notification.\n",
|
||||
"- Generate synthetic prediction requests for drift.\n",
|
||||
"- Wait for email alert notification.\n",
|
||||
"\n",
|
||||
@@ -106,7 +105,7 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). The version of this dataset you use only the fields year, month and day to predict the value of mean daily temperature (mean_temp)."
|
||||
"The dataset used for this tutorial is the GSOD dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). In this notebook, you use only the fields year, month and day from the dataset to predict the value of mean daily temperature (mean_temp)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -563,7 +562,7 @@
|
||||
"source": [
|
||||
"### Create BigQuery client\n",
|
||||
"\n",
|
||||
"In this tutorial, you use data from the same public BigQuery table that was used to train the pre-trained model. You create a client interface, which you subsequently use to access the data."
|
||||
"In this tutorial, you explore the monitoring data stored in BigQuery. You create a client interface, which you subsequently use to access the data."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -668,23 +667,6 @@
|
||||
"print(dataset.resource_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_transformations:gsod"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRANSFORMATIONS = [\n",
|
||||
" {\"auto\": {\"column_name\": \"year\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"month\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"day\"}},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -703,7 +685,7 @@
|
||||
"- `optimization_prediction_type`: The type task to train the model for.\n",
|
||||
" - `classification`: A tabuar classification model.\n",
|
||||
" - `regression`: A tabular regression model.\n",
|
||||
"- `column_transformations`: (Optional): Transformations to apply to the input columns\n",
|
||||
"- `column_transformations`: (Optional): Transformations to apply to the input columns. In this example, you set the column transformations to use the default transformation based on their data type.\n",
|
||||
"- `optimization_objective`: The optimization objective to minimize or maximize.\n",
|
||||
" - binary classification:\n",
|
||||
" - `minimize-log-loss`\n",
|
||||
@@ -719,6 +701,23 @@
|
||||
" - `minimize-rmsle`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "set_transformations:gsod"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRANSFORMATIONS = [\n",
|
||||
" {\"auto\": {\"column_name\": \"year\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"month\"}},\n",
|
||||
" {\"auto\": {\"column_name\": \"day\"}},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"label_column = \"mean_temp\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -1065,7 +1064,9 @@
|
||||
"You are receiving this mail because you are using the Vertex AI Model Monitoring service.\n",
|
||||
"This mail is to inform you that we received your request to set up drift or skew detection for the Prediction Endpoint listed below. Starting from now, incoming prediction requests will be sampled and logged for analysis.\n",
|
||||
"Raw requests and responses will be collected from prediction service and saved in bq://[your-project-id].model_deployment_monitoring_[endpoint-id].serving_predict .\n",
|
||||
"</blockquote>"
|
||||
"</blockquote>\n",
|
||||
"\n",
|
||||
"*Note:* You do not need to wait for the email notification to continue to the next step."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1130,7 +1131,7 @@
|
||||
"\n",
|
||||
"Next, you extract the first 1000 instances from the BigQuery training table to use for prediction requests. You modify the data (synthetic) to trigger the skew detection in the prediction requests from the training distribution versus serving distribution, as follows:\n",
|
||||
"\n",
|
||||
"- `year`: Set all values to 3 (was 2)."
|
||||
"- `year`: Set all values to 3."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1275,7 +1276,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b91a0e19ff8b"
|
||||
"id": "2e64ffaae2de"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1283,41 +1284,6 @@
|
||||
" time.sleep(60 * 45)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2b5859ea4ae9"
|
||||
},
|
||||
"source": [
|
||||
"### Logging sampled requests\n",
|
||||
"\n",
|
||||
"On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
|
||||
"\n",
|
||||
"Next, you wait for the logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "bd177a8decbb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"while True:\n",
|
||||
" time.sleep(180)\n",
|
||||
"\n",
|
||||
" ENDPOINT_ID = endpoint.resource_name.split(\"/\")[-1]\n",
|
||||
"\n",
|
||||
" table = bigquery.TableReference.from_string(\n",
|
||||
" f\"{PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",
|
||||
" )\n",
|
||||
" rows = bqclient.list_rows(table)\n",
|
||||
" print(rows.total_rows)\n",
|
||||
" if rows.total_rows > 505:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -1419,7 +1385,7 @@
|
||||
" )\n",
|
||||
" rows = bqclient.list_rows(table)\n",
|
||||
" print(rows.total_rows)\n",
|
||||
" if rows.total_rows > 1050:\n",
|
||||
" if rows.total_rows > 505:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
|
||||
+1527
File diff suppressed because one or more lines are too long
@@ -68,7 +68,7 @@
|
||||
"\n",
|
||||
"<a href=\"https://storage.googleapis.com/amy-jo/images/mp/beans.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/beans.png\" width=\"95%\"/></a>\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [AutoML components](https://cloud.google.com/vertex-ai/docs/pipelines/vertex-automl-component)."
|
||||
"Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [AutoML components](https://cloud.google.com/vertex-ai/docs/pipelines/vertex-automl-component). Learn more about [Classification for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -63,7 +63,7 @@
|
||||
"\n",
|
||||
"This notebook shows how to use the components defined in [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) to build an AutoML tabular regression workflow on Vertex AI Pipelines.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [AutoML components](https://cloud.google.com/vertex-ai/docs/pipelines/vertex-automl-component)."
|
||||
"Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) and [AutoML components](https://cloud.google.com/vertex-ai/docs/pipelines/vertex-automl-component). Learn more about [Regression for tabular data](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -207,7 +207,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"PROJECT_ID = \"andy-1234-221921\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Set the project id\n",
|
||||
"! gcloud config set project {PROJECT_ID}"
|
||||
@@ -590,8 +590,9 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@component\n",
|
||||
"def consumer(text1: str, text2: str, text3: str):\n",
|
||||
" print(f\"text1: {text1}; text2: {text2}; text3: {text3}\")"
|
||||
"def consumer(text1: str, text2: str, text3: str) -> str:\n",
|
||||
" print(f\"text1: {text1}; text2: {text2}; text3: {text3}\")\n",
|
||||
" return f\"text1: {text1}; text2: {text2}; text3: {text3}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,794 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "JAPoU8Sm5E6e"
|
||||
},
|
||||
"source": [
|
||||
"# Vertex AI TensorBoard Hyperparameter Tuning with the HParams Dashboard\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tensorboard/tensorboard_hyperparameter_tuning_with_hparams.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tensorboard/tensorboard_hyperparameter_tuning_with_hparams.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/tensorboard/tensorboard_hyperparameter_tuning_with_hparams.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "24743cf4a1e1"
|
||||
},
|
||||
"source": [
|
||||
"**_NOTE_**: This notebook has been tested in the following environments:\n",
|
||||
"\n",
|
||||
"* Python version = 3.8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"### What is Vertex AI TensorBoard\n",
|
||||
"\n",
|
||||
"[Open source TensorBoard](https://www.tensorflow.org/tensorboard/get_started)\n",
|
||||
"(TB) is a Google open source project for machine learning experiment\n",
|
||||
"visualization. Vertex AI TensorBoard is an enterprise-ready managed\n",
|
||||
"version of TensorBoard.\n",
|
||||
"\n",
|
||||
"Vertex AI TensorBoard provides various detailed visualizations, including the following:\n",
|
||||
"\n",
|
||||
"* Tracking and visualizing metrics, such as loss and accuracy over time.\n",
|
||||
"* Visualizing model computational graphs (ops and layers).\n",
|
||||
"* Viewing histograms of weights, biases, or other tensors as they change over time.\n",
|
||||
"* Projecting embeddings to a lower dimensional space.\n",
|
||||
"* Displaying image, text, and audio samples.\n",
|
||||
"\n",
|
||||
"In addition to the powerful visualizations from\n",
|
||||
"TensorBoard, Vertex AI TensorBoard provides the following benefits:\n",
|
||||
"\n",
|
||||
"* A persistent, shareable link to your experiment's dashboard.\n",
|
||||
"\n",
|
||||
"* A searchable list of all experiments in a project.\n",
|
||||
"\n",
|
||||
"* Integrations with Vertex AI services for model training.\n",
|
||||
"\n",
|
||||
"* Enterprise-grade security, privacy, and compliance.\n",
|
||||
"\n",
|
||||
"With Vertex AI TensorBoard, you can track, visualize, and compare\n",
|
||||
"ML experiments and share them with your team.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI TensorBoard](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d975e698c9a4"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"This tutorial shows you how to log hyperparameter experiment results in TensorFlow and visualize the results in TensorBoard's Hparams dashboard.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Vertex AI services and resources:\n",
|
||||
"\n",
|
||||
"- Vertex AI TensorBoard\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"* Adapt TensorFlow runs to log hyperparameters and metrics.\n",
|
||||
"* Start runs and log them all under one parent directory.\n",
|
||||
"* Visualize the results in TensorBoard's HParams dashboard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "08d289fa873f"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"This tutorial uses the [FashionMNIST](https://github.com/zalandoresearch/fashion-mnist) dataset.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "aed92deeb4a0"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses the following billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
|
||||
"and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1fD9UZaygyPG"
|
||||
},
|
||||
"source": [
|
||||
"## Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Vertex AI Workbench**, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
|
||||
"\n",
|
||||
"Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
|
||||
"\n",
|
||||
"- Git\n",
|
||||
"- Python 3\n",
|
||||
"- virtualenv\n",
|
||||
"- Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"To quickly set up your environment to meet the requirements of this tutorial, perform the following:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
|
||||
"\n",
|
||||
"2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
|
||||
"\n",
|
||||
"3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3 and activate the virtual environment.\n",
|
||||
"\n",
|
||||
"4. Install Jupyter by running the following command in a terminal shell:\n",
|
||||
"<br> `pip3 install jupyter`\n",
|
||||
"\n",
|
||||
"5. Launch Jupyter by running the following command in a terminal shell: <br> `jupyter notebook`\n",
|
||||
"\n",
|
||||
"6. Open this tutorial notebook in the Jupyter Notebook Dashboard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "i7EUnXsZhAGF"
|
||||
},
|
||||
"source": [
|
||||
"## Install dependencies\n",
|
||||
"\n",
|
||||
"Install the following packages required to run this tutorial notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "th7tWguZiSN2"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# The Vertex AI Workbench Notebook product has specific requirements\n",
|
||||
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
|
||||
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
|
||||
" \"/opt/deeplearning/metadata/env_version\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
|
||||
"USER_FLAG = \"\"\n",
|
||||
"if IS_WORKBENCH_NOTEBOOK:\n",
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform[tensorboard] tensorflow==2.7 {USER_FLAG} -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "58707a750154"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only: Uncomment the following cell to restart the kernel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "f200f10a1da3"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
|
||||
"# import IPython\n",
|
||||
"\n",
|
||||
"# app = IPython.Application.instance()\n",
|
||||
"# app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, install the [Cloud SDK](https://cloud.google.com/sdk)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WReHDGG5g0XY"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, try the following:\n",
|
||||
"* Run `gcloud config list`.\n",
|
||||
"* Run `gcloud projects list`.\n",
|
||||
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "oM1iC_MfAts1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Set the project id\n",
|
||||
"! gcloud config set project {PROJECT_ID}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "region"
|
||||
},
|
||||
"source": [
|
||||
"#### Set the region\n",
|
||||
"\n",
|
||||
"**Optional**: Update the 'REGION' variable to specify the region that you want to use. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "nsN5NJKSu-GU"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "sBCra4QMA2wR"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"To authenticate your Google Cloud account, follow the instructions for your Jupyter environment:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "74ccc9e52986"
|
||||
},
|
||||
"source": [
|
||||
"* **Vertex AI Workbench**\n",
|
||||
"<br>You are already authenticated."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "de775a3773ba"
|
||||
},
|
||||
"source": [
|
||||
"* **Local JupyterLab instance**\n",
|
||||
"<br>Uncomment and run the following code:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "254614fa0c46"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! gcloud auth login"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ef21552ccea8"
|
||||
},
|
||||
"source": [
|
||||
"* **Colab**\n",
|
||||
"<br>Uncomment and run the following code:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "603adbbf0532"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# from google.colab import auth\n",
|
||||
"\n",
|
||||
"# auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "960505627ddf"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "PyQmSRbKA8r-"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "init_aip:mbsdk,all"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize the Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "KllitKlIu-GW"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WjWD61gONRkw"
|
||||
},
|
||||
"source": [
|
||||
"### Load TensorBoard and TensorFlow components\n",
|
||||
"\n",
|
||||
"Load the TensorBoard notebook extension and import TensorFlow and the TensorBoard HParams plugin.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "KSayPNqxfJC_"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Load the TensorBoard notebook extension\n",
|
||||
"%load_ext tensorboard\n",
|
||||
"\n",
|
||||
"# Clear any logs from previous runs\n",
|
||||
"!rm -rf ./logs/\n",
|
||||
"\n",
|
||||
"# Import TensorFlow and the TensorBoard HParams plugin\n",
|
||||
"import tensorflow as tf\n",
|
||||
"from tensorboard.plugins.hparams import api as hp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "KJ4zE7rYfcvb"
|
||||
},
|
||||
"source": [
|
||||
"### Download dataset\n",
|
||||
"\n",
|
||||
"Download the [FashionMNIST](https://github.com/zalandoresearch/fashion-mnist) dataset and scale it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "vHME9wnnfiMr"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fashion_mnist = tf.keras.datasets.fashion_mnist\n",
|
||||
"\n",
|
||||
"(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
|
||||
"x_train, x_test = x_train / 255.0, x_test / 255.0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ofGSMru5r4kP"
|
||||
},
|
||||
"source": [
|
||||
"## Set up the experiment\n",
|
||||
"\n",
|
||||
"Run an experiment by specifying values for the following hyperparameters:\n",
|
||||
"\n",
|
||||
"* Number of units in the first dense layer\n",
|
||||
"* Dropout rate in the dropout layer\n",
|
||||
"* Optimizer\n",
|
||||
"\n",
|
||||
"Specify the hyperparameter values for the experiment in TensorBoard.\n",
|
||||
"\n",
|
||||
"*Optional*: For more fine grained filtering of hyperparameters in the UI, provide domain information and specify which metrics should be displayed."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "IG5sPLBAcDRy"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"HP_NUM_UNITS = hp.HParam(\"num_units\", hp.Discrete([16, 32]))\n",
|
||||
"HP_DROPOUT = hp.HParam(\"dropout\", hp.RealInterval(0.1, 0.2))\n",
|
||||
"HP_OPTIMIZER = hp.HParam(\"optimizer\", hp.Discrete([\"adam\", \"sgd\"]))\n",
|
||||
"\n",
|
||||
"METRIC_ACCURACY = \"accuracy\"\n",
|
||||
"\n",
|
||||
"with tf.summary.create_file_writer(\"logs/hparam_tuning\").as_default():\n",
|
||||
" hp.hparams_config(\n",
|
||||
" hparams=[HP_NUM_UNITS, HP_DROPOUT, HP_OPTIMIZER],\n",
|
||||
" metrics=[hp.Metric(METRIC_ACCURACY, display_name=\"Accuracy\")],\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cLNgBNA6srlk"
|
||||
},
|
||||
"source": [
|
||||
"## Adapt TensorFlow runs to log hyperparameters and metrics\n",
|
||||
"\n",
|
||||
"The model will be quite simple: two dense layers with a dropout layer between them. The training code will look familiar, although the hyperparameters are no longer hardcoded. Instead, the hyperparameters are provided in an `hparams` dictionary and used throughout the training function:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "C-RSsrF4u-Fq"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_test_model(hparams):\n",
|
||||
" model = tf.keras.models.Sequential(\n",
|
||||
" [\n",
|
||||
" tf.keras.layers.Flatten(),\n",
|
||||
" tf.keras.layers.Dense(hparams[HP_NUM_UNITS], activation=tf.nn.relu),\n",
|
||||
" tf.keras.layers.Dropout(hparams[HP_DROPOUT]),\n",
|
||||
" tf.keras.layers.Dense(10, activation=tf.nn.softmax),\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
" model.compile(\n",
|
||||
" optimizer=hparams[HP_OPTIMIZER],\n",
|
||||
" loss=\"sparse_categorical_crossentropy\",\n",
|
||||
" metrics=[\"accuracy\"],\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model.fit(\n",
|
||||
" x_train, y_train, epochs=1\n",
|
||||
" ) # Run with 1 epoch to speed things up for demo purposes\n",
|
||||
" _, accuracy = model.evaluate(x_test, y_test)\n",
|
||||
" return accuracy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Esz3uqqCvLoK"
|
||||
},
|
||||
"source": [
|
||||
"For each run, log an hparams summary with the hyperparameters and final accuracy:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "HwR1PAv1vPER"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def run(run_dir, hparams):\n",
|
||||
" with tf.summary.create_file_writer(run_dir).as_default():\n",
|
||||
" hp.hparams(hparams) # record the values used in this trial\n",
|
||||
" accuracy = train_test_model(hparams)\n",
|
||||
" tf.summary.scalar(METRIC_ACCURACY, accuracy, step=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0V_8soFFvU7b"
|
||||
},
|
||||
"source": [
|
||||
"## Start runs and log them all under one parent directory\n",
|
||||
"\n",
|
||||
"You can now try multiple experiments, training each one with a different set of hyperparameters.\n",
|
||||
"\n",
|
||||
"For simplicity, use a grid search: try all combinations of the discrete parameters and just the lower and upper bounds of the real-valued parameter. For more complex scenarios, it might be more effective to choose each hyperparameter value randomly (this is called a random search). There are more advanced methods that can be used.\n",
|
||||
"\n",
|
||||
"Run a few experiments, which will take a few minutes:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "6r2oO_PVvbdL"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"session_num = 0\n",
|
||||
"\n",
|
||||
"for num_units in HP_NUM_UNITS.domain.values:\n",
|
||||
" for dropout_rate in (HP_DROPOUT.domain.min_value, HP_DROPOUT.domain.max_value):\n",
|
||||
" for optimizer in HP_OPTIMIZER.domain.values:\n",
|
||||
" hparams = {\n",
|
||||
" HP_NUM_UNITS: num_units,\n",
|
||||
" HP_DROPOUT: dropout_rate,\n",
|
||||
" HP_OPTIMIZER: optimizer,\n",
|
||||
" }\n",
|
||||
" run_name = \"run-%d\" % session_num\n",
|
||||
" print(\"--- Starting trial: %s\" % run_name)\n",
|
||||
" print({h.name: hparams[h] for h in hparams})\n",
|
||||
" run(\"logs/hparam_tuning/\" + run_name, hparams)\n",
|
||||
" session_num += 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "6FJJwCclvslF"
|
||||
},
|
||||
"source": [
|
||||
"## Visualize the results in Vertex AI TensorBoard's HParams tab"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "BkbB5GEI3Ge3"
|
||||
},
|
||||
"source": [
|
||||
"### Create Vertex AI Tensorboard\n",
|
||||
"A Vertex AI TensorBoard instance, which is a regionalized resource storing your Vertex AI TensorBoard experiments, must be created before the experiments can be visualized. You can create multiple instances in a project. [documentation instructions](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview).\n",
|
||||
"\n",
|
||||
"Create a TensorBoard instance to be used by the training job."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "lQ-d3j-I3ZWV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TENSORBOARD_NAME = \"[your-tensorboard-name]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if (\n",
|
||||
" TENSORBOARD_NAME == \"\"\n",
|
||||
" or TENSORBOARD_NAME is None\n",
|
||||
" or TENSORBOARD_NAME == \"[your-tensorboard-name]\"\n",
|
||||
"):\n",
|
||||
" TENSORBOARD_NAME = PROJECT_ID + \"-tb-\"\n",
|
||||
"\n",
|
||||
"tensorboard = aiplatform.Tensorboard.create(\n",
|
||||
" display_name=TENSORBOARD_NAME, project=PROJECT_ID, location=REGION\n",
|
||||
")\n",
|
||||
"TENSORBOARD_RESOURCE_NAME = tensorboard.gca_resource.name\n",
|
||||
"print(\"TensorBoard resource name:\", TENSORBOARD_RESOURCE_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "27rERDqeJ2nE"
|
||||
},
|
||||
"source": [
|
||||
"Set your TensorBoard Experiment name."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4OU4TMtFCn0_"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"EXPERIMENT_NAME = \"[your-experiment-run-name]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"if (\n",
|
||||
" EXPERIMENT_NAME == \"\"\n",
|
||||
" or EXPERIMENT_NAME is None\n",
|
||||
" or EXPERIMENT_NAME == \"[your-experiment-run-name]\"\n",
|
||||
"):\n",
|
||||
" EXPERIMENT_NAME = \"experiment\" + datetime.now().strftime(\"%H-%M-%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f1D2oU3K8Ys0"
|
||||
},
|
||||
"source": [
|
||||
"Upload the log to your Vertex AI TensorBoard"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "TyXFVQuRv0-X"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!tb-gcp-uploader --one_shot=True --tensorboard_resource_name=$TENSORBOARD_RESOURCE_NAME --logdir=\"logs/hparam_tuning/\" --experiment_name=$EXPERIMENT_NAME"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "OFe3qRyh9Wjl"
|
||||
},
|
||||
"source": [
|
||||
"Click the generated TensorBoard link and click on \"HParams\" at the top.\n",
|
||||
"\n",
|
||||
"The left pane of the dashboard provides filtering capabilities that are active across all the views in the HParams dashboard:\n",
|
||||
"\n",
|
||||
"- Filter which hyperparameters/metrics are shown in the dashboard\n",
|
||||
"- Filter which hyperparameter/metrics values are shown in the dashboard\n",
|
||||
"- Filter on run status (running, success, ...)\n",
|
||||
"- Sort by hyperparameter/metric in the table view\n",
|
||||
"- Number of session groups to show (useful for performance when there are many experiments)\n",
|
||||
"\n",
|
||||
"The HParams dashboard has three different views, with various useful information:\n",
|
||||
"\n",
|
||||
"* The **Table View** lists the runs, their hyperparameters, and their metrics.\n",
|
||||
"* The **Parallel Coordinates View** shows each run as a line going through an axis for each hyperparemeter and metric. Click and drag the mouse on any axis to mark a region which will highlight only the runs that pass through it. This can be useful for identifying which groups of hyperparameters are most important. The axes themselves can be re-ordered by dragging them.\n",
|
||||
"* The **Scatter Plot View** shows plots comparing each hyperparameter/metric with each metric. This can help identify correlations. Click and drag to select a region in a specific plot and highlight those sessions across the other plots.\n",
|
||||
"\n",
|
||||
"A table row, a parallel coordinates line, and a scatter plot market can be clicked to see a plot of the metrics as a function of training steps for that session (although in this tutorial only one step is used for each run)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TpV-iwP9qw9c"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "sx_vKniMq9ZX"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Delete endpoint resource\n",
|
||||
"# e.g. `endpoint.delete()`\n",
|
||||
"\n",
|
||||
"# Delete model resource\n",
|
||||
"# e.g. `model.delete()`\n",
|
||||
"\n",
|
||||
"# Delete Cloud Storage objects that were created\n",
|
||||
"delete_bucket = False\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil -m rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "tensorboard_hyperparameter_tuning_with_hparams.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
+869
@@ -0,0 +1,869 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "l2mMvIUG9meX"
|
||||
},
|
||||
"source": [
|
||||
"# Profile model training performance using Vertex AI TensorBoard Profiler in custom training with prebuilt container\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tensorboard/tensorboard_profiler_custom_training_with_prebuilt_container.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tensorboard/tensorboard_profiler_custom_training_with_prebuilt_container.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/tensorboard/tensorboard_profiler_custom_training_with_prebuilt_container.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"The TensorFlow Profiler is a powerful tool that can help you to diagnose and debug performance bottlenecks, and make your model train faster. This tutorial demonstrates how to enable the TensorBoard Profiler in Vertex AI for custom training with a prebuilt container.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI TensorBoard Profiler](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-profiler)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dmfmQL6w84pS"
|
||||
},
|
||||
"source": [
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this tutorial, you learn how to enable the TensorBoard Profiler in Vertex AI for custom training jobs with a prebuilt container.\n",
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud AI services:\n",
|
||||
"\n",
|
||||
"- Vertex AI Training\n",
|
||||
"- Vertex AI TensorBoard\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
"\n",
|
||||
"- Prepare your custom training code and load your training code as a Python package to a prebuilt container\n",
|
||||
"- Create and run a custom training job that enables the TensorBoard Profiler\n",
|
||||
"- View the TensorBoard Profiler dashboard to debug your model training performance\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "zfXf0r-K81Y-"
|
||||
},
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"The dataset used for this tutorial is the [mnist dataset](https://www.tensorflow.org/datasets/catalog/mnist) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "I3KFLvpq87rs"
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ze4-nDLfK4pw"
|
||||
},
|
||||
"source": [
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install the following packages required to execute this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "2b4ef9b72d43"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip3 install --upgrade --quiet google-cloud-aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "aUw6ibN-n5Za"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only: Uncomment the following cell to restart the kernel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "FM12wbWhn7w0"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
|
||||
"# import IPython\n",
|
||||
"\n",
|
||||
"# app = IPython.Application.instance()\n",
|
||||
"# app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "LgFWLeJfoGQu"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"\n",
|
||||
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8ckyxpX_oSzD"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, try the following:\n",
|
||||
"* Run `gcloud config list`.\n",
|
||||
"* Run `gcloud projects list`.\n",
|
||||
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "zY8DKBoVoVy3"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Set the project id\n",
|
||||
"! gcloud config set project {PROJECT_ID}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "mSQjVQmMosMl"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Se9FWWhLotvB"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "IfJRIMBpo5Pg"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "acFN0s3So9-Y"
|
||||
},
|
||||
"source": [
|
||||
"**1. Vertex AI Workbench**\n",
|
||||
"* Do nothing as you are already authenticated."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dQ_mNwuapE5T"
|
||||
},
|
||||
"source": [
|
||||
"**2. Local JupyterLab instance, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "cR_MzpknpGgM"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! gcloud auth login"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "h-MuVI_ypJfw"
|
||||
},
|
||||
"source": [
|
||||
"**3. Colab, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "BeaQlCwMpQUT"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# from google.colab import auth\n",
|
||||
"# auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3ivZkPUjpaFz"
|
||||
},
|
||||
"source": [
|
||||
"**4. Setup service account and permissions**\n",
|
||||
"\n",
|
||||
"A service account will be used to create custom training jobs. If you do not want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID. You can create a service account by following the [instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "vYE3b942wza4"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "WWIxsCJFCg5Z"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Grant Cloud Storage permission.\n",
|
||||
"! gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
|
||||
" --member=\"serviceAccount:$SERVICE_ACCOUNT\" \\\n",
|
||||
" --role=\"roles/storage.admin\" \\\n",
|
||||
" --quiet\n",
|
||||
"\n",
|
||||
"# Grant AI Platform permission.\n",
|
||||
"! gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
|
||||
" --member=\"serviceAccount:$SERVICE_ACCOUNT\" \\\n",
|
||||
" --role=\"roles/aiplatform.user\" \\\n",
|
||||
" --quiet\n",
|
||||
"\n",
|
||||
"! gcloud projects get-iam-policy $PROJECT_ID \\\n",
|
||||
" --filter=bindings.members:serviceAccount:$SERVICE_ACCOUNT"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "OKtKGmr9pfr6"
|
||||
},
|
||||
"source": [
|
||||
"### Create a Cloud Storage bucket\n",
|
||||
"\n",
|
||||
"Create a storage bucket to store intermediate artifacts such as datasets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "In3aQanwYjFB"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "GOaOsIjxp0oB"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Wn5QiIl2p16e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ankcS-vtp7Wv"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "WffSImMvp-Po"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from google.cloud import aiplatform"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "OMrAJ8RGqBQu"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize Vertex AI SDK for Python\n",
|
||||
"\n",
|
||||
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AWRzBFExqERG"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "-ayTbNdi62_t"
|
||||
},
|
||||
"source": [
|
||||
"### Create a TensorBoard instance\n",
|
||||
"\n",
|
||||
"A Vertex AI TensorBoard instance, which is a regionalized resource storing your Vertex AI TensorBoard experiments, must be created before the experiments can be visualized. You can create multiple instances in a project. You can use command `gcloud ai tensorboards list` to get a list of your existing TensorBoard instances."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "9c3QrDTZdaxk"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your TensorBoard instance display name\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "azlwb__AX8gs"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TENSORBOARD_NAME = \"your-tensorboard-unique\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "vJrWKK0mY7H7"
|
||||
},
|
||||
"source": [
|
||||
"#### Create a TensorBoard instance\n",
|
||||
"\n",
|
||||
"If you don't have a TensorBoard instance, create one by running the following cell:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JqVNsRFrc_78"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tensorboard = aiplatform.Tensorboard.create(\n",
|
||||
" display_name=TENSORBOARD_NAME, project=PROJECT_ID, location=REGION\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"TENSORBOARD_INSTANCE_NAME = tensorboard.resource_name\n",
|
||||
"print(\"TensorBoard instance name:\", TENSORBOARD_INSTANCE_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "yoR29gW2S24w"
|
||||
},
|
||||
"source": [
|
||||
"## Train a model\n",
|
||||
"\n",
|
||||
"To train a model using your custom training code, choose one of the following options:\n",
|
||||
"\n",
|
||||
"- **Prebuilt container**: Load your custom training code as a Python package to a prebuilt container image from Google Cloud.\n",
|
||||
"\n",
|
||||
"- **Custom container**: Create your own container image that contains your custom training code.\n",
|
||||
"\n",
|
||||
"In this tutorial, you will train a custom model using a prebuilt container."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "syw3GabNGgJz"
|
||||
},
|
||||
"source": [
|
||||
"### Examine the training package\n",
|
||||
"\n",
|
||||
"#### Package layout\n",
|
||||
"\n",
|
||||
"Before you start the training, let's take a look at how a Python package is assembled for a custom training job. When extracted, the package contains the following:\n",
|
||||
"\n",
|
||||
"- PKG-INFO\n",
|
||||
"- README.md\n",
|
||||
"- setup.cfg\n",
|
||||
"- setup.py\n",
|
||||
"- trainer\n",
|
||||
" - \\_\\_init\\_\\_.py\n",
|
||||
" - task.py\n",
|
||||
"\n",
|
||||
"The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the docker image."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "b58ZAbysGkRo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PYTHON_PACKAGE_APPLICATION_DIR = \"app\"\n",
|
||||
"\n",
|
||||
"source_package_file_name = f\"{PYTHON_PACKAGE_APPLICATION_DIR}/dist/trainer-0.1.tar.gz\"\n",
|
||||
"python_package_gcs_uri = f\"{BUCKET_URI}/trainer-0.1.tar.gz\"\n",
|
||||
"\n",
|
||||
"# Make folder for Python training script\n",
|
||||
"! rm -rf {PYTHON_PACKAGE_APPLICATION_DIR}\n",
|
||||
"! mkdir {PYTHON_PACKAGE_APPLICATION_DIR}\n",
|
||||
"\n",
|
||||
"# Add package information\n",
|
||||
"! touch {PYTHON_PACKAGE_APPLICATION_DIR}/README.md\n",
|
||||
"\n",
|
||||
"# Make the training subfolder\n",
|
||||
"! mkdir {PYTHON_PACKAGE_APPLICATION_DIR}/trainer\n",
|
||||
"! touch {PYTHON_PACKAGE_APPLICATION_DIR}/trainer/__init__.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "lj7hIeAXGrzg"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./{PYTHON_PACKAGE_APPLICATION_DIR}/setup.py\n",
|
||||
"\n",
|
||||
"from setuptools import find_packages\n",
|
||||
"from setuptools import setup\n",
|
||||
"import setuptools\n",
|
||||
"\n",
|
||||
"from distutils.command.build import build as _build\n",
|
||||
"import subprocess\n",
|
||||
"\n",
|
||||
"REQUIRED_PACKAGES = [\n",
|
||||
" 'google-cloud-aiplatform[cloud_profiler]>=1.20.0',\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"setup(\n",
|
||||
" install_requires=REQUIRED_PACKAGES,\n",
|
||||
" packages=find_packages(),\n",
|
||||
" include_package_data=True,\n",
|
||||
" name='trainer',\n",
|
||||
" version='0.1',\n",
|
||||
" url=\"wwww.google.com\",\n",
|
||||
" description='Vertex AI | Training | Python Package'\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hyAwgsoQmaYI"
|
||||
},
|
||||
"source": [
|
||||
"#### Prepare the training script\n",
|
||||
"\n",
|
||||
"The file `trainer/task.py` is the Python script for executing the custom training job.\n",
|
||||
"\n",
|
||||
"Your training code must be configured to write TensorBoard logs to a Cloud Storage bucket, the location of which Vertex AI Training automatically makes available through a predefined environment variable, `AIP_TENSORBOARD_LOG_DIR`. This can usually be done by providing `os.environ['AIP_TENSORBOARD_LOG_DIR']` as the log directory to the open source TensorBoard log writing APIs. For example, in TensorFlow 2.x, you can use following code to create a `tensorboard_callback`:\n",
|
||||
"\n",
|
||||
" tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
|
||||
" log_dir=os.environ['AIP_TENSORBOARD_LOG_DIR'],\n",
|
||||
" histogram_freq=1)\n",
|
||||
"`AIP_TENSORBOARD_LOG_DIR` is in the `BASE_OUTPUT_DIR` that you provide when creating the custom training job.\n",
|
||||
"\n",
|
||||
"To enable Vertex AI TensorBoard Profiler for your training job, add the following to your training script:\n",
|
||||
"\n",
|
||||
"Add the cloud_profiler import at your top level imports:\n",
|
||||
"\n",
|
||||
" from google.cloud.aiplatform.training_utils import cloud_profiler\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Initialize the cloud_profiler plugin by adding:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" cloud_profiler.init()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "8JCgWW7Au1w8"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./{PYTHON_PACKAGE_APPLICATION_DIR}/trainer/task.py\n",
|
||||
"\n",
|
||||
"import tensorflow as tf\n",
|
||||
"import argparse\n",
|
||||
"import os\n",
|
||||
"import sys, traceback\n",
|
||||
"from google.cloud.aiplatform.training_utils import cloud_profiler\n",
|
||||
"\n",
|
||||
"\"\"\"Train an mnist model and use cloud_profiler for profiling.\"\"\"\n",
|
||||
"\n",
|
||||
"def _create_model():\n",
|
||||
" model = tf.keras.models.Sequential(\n",
|
||||
" [\n",
|
||||
" tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
|
||||
" tf.keras.layers.Dense(128, activation=\"relu\"),\n",
|
||||
" tf.keras.layers.Dropout(0.2),\n",
|
||||
" tf.keras.layers.Dense(10),\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
" return model\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def main(args):\n",
|
||||
" print('Initialize the profiler ...')\n",
|
||||
" cloud_profiler.init()\n",
|
||||
" print('The profiler initiated.')\n",
|
||||
"\n",
|
||||
" print('Loading and preprocessing data ...')\n",
|
||||
" mnist = tf.keras.datasets.mnist\n",
|
||||
"\n",
|
||||
" (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
|
||||
" x_train, x_test = x_train / 255.0, x_test / 255.0\n",
|
||||
"\n",
|
||||
" print('Creating and training model ...')\n",
|
||||
"\n",
|
||||
" model = _create_model()\n",
|
||||
" model.compile(\n",
|
||||
" optimizer=\"adam\",\n",
|
||||
" loss=tf.keras.losses.sparse_categorical_crossentropy,\n",
|
||||
" metrics=[\"accuracy\"],\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" log_dir = \"logs\"\n",
|
||||
" if 'AIP_TENSORBOARD_LOG_DIR' in os.environ:\n",
|
||||
" log_dir = os.environ['AIP_TENSORBOARD_LOG_DIR']\n",
|
||||
"\n",
|
||||
" print('Setting up the TensorBoard callback ...')\n",
|
||||
" tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
|
||||
" log_dir=log_dir,\n",
|
||||
" histogram_freq=1)\n",
|
||||
"\n",
|
||||
" print('Training model ...')\n",
|
||||
" model.fit(\n",
|
||||
" x_train,\n",
|
||||
" y_train,\n",
|
||||
" epochs=args.epochs,\n",
|
||||
" verbose=0,\n",
|
||||
" callbacks=[tensorboard_callback],\n",
|
||||
" )\n",
|
||||
" print('Training completed.')\n",
|
||||
"\n",
|
||||
" print('Saving model ...')\n",
|
||||
"\n",
|
||||
" model_dir = \"model\"\n",
|
||||
" if 'AIP_MODEL_DIR' in os.environ:\n",
|
||||
" model_dir = os.environ['AIP_MODEL_DIR']\n",
|
||||
" tf.saved_model.save(model, model_dir)\n",
|
||||
"\n",
|
||||
" print('Model saved at ' + model_dir)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"if __name__ == \"__main__\":\n",
|
||||
" parser = argparse.ArgumentParser()\n",
|
||||
" parser.add_argument(\n",
|
||||
" \"--epochs\", type=int, default=100, help=\"Number of epochs to run model.\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" args = parser.parse_args()\n",
|
||||
" main(args)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ihYFahRAr6sj"
|
||||
},
|
||||
"source": [
|
||||
"#### Create a source distribution\n",
|
||||
"\n",
|
||||
"You create a source distribution with your training application and upload the source distribution to your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "-XhccshCHQeb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!cd {PYTHON_PACKAGE_APPLICATION_DIR} && python3 setup.py sdist --formats=gztar\n",
|
||||
"\n",
|
||||
"!gsutil cp {source_package_file_name} {python_package_gcs_uri}\n",
|
||||
"\n",
|
||||
"!gsutil ls -l {python_package_gcs_uri}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "k4e6OYmimqTR"
|
||||
},
|
||||
"source": [
|
||||
"### Create and run the custom training job\n",
|
||||
"\n",
|
||||
"Configure a [custom job](https://cloud.google.com/vertex-ai/docs/training/create-custom-job) with the [pre-built container](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) image for training code packaged as Python source distribution."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "t8GeVXjWHxuZ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"JOB_NAME = \"tensorboard-job-unique\"\n",
|
||||
"MACHINE_TYPE = \"n1-standard-4\"\n",
|
||||
"TRAIN_IMAGE = \"us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-9:latest\"\n",
|
||||
"base_output_dir = f\"{BUCKET_URI}/{JOB_NAME}\"\n",
|
||||
"python_module_name = \"trainer.task\"\n",
|
||||
"\n",
|
||||
"EPOCHS = 20\n",
|
||||
"training_args = [\n",
|
||||
" \"--epochs=\" + str(EPOCHS),\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "B3JC7T3bH9Vy"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job = aiplatform.CustomPythonPackageTrainingJob(\n",
|
||||
" display_name=JOB_NAME,\n",
|
||||
" python_package_gcs_uri=python_package_gcs_uri,\n",
|
||||
" python_module_name=python_module_name,\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "51hKGTbU32Eg"
|
||||
},
|
||||
"source": [
|
||||
"#### Run the custom training job\n",
|
||||
"\n",
|
||||
"Next, you run the custom job to start the training job by invoking the method `run`.\n",
|
||||
"\n",
|
||||
"**NOTE:** When using Vertex AI SDK for Python for submitting a training job, it creates a [training pipeline](https://cloud.google.com/vertex-ai/docs/training/create-training-pipeline) which launches the custom job on Vertex AI Training service."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "oIyfos1rIAx2"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"job.run(\n",
|
||||
" replica_count=1,\n",
|
||||
" machine_type=MACHINE_TYPE,\n",
|
||||
" base_output_dir=base_output_dir,\n",
|
||||
" tensorboard=TENSORBOARD_INSTANCE_NAME,\n",
|
||||
" service_account=SERVICE_ACCOUNT,\n",
|
||||
" args=training_args,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "JkEe2Nb_85UD"
|
||||
},
|
||||
"source": [
|
||||
"## View the TensorBoard Profiler dashboard\n",
|
||||
"\n",
|
||||
"When the custom job state switches to `Running`, you can access the Vertex AI TensorBoard Profiler dashboard through the Custom jobs page or the Experiments page on the Google Cloud console.\n",
|
||||
"\n",
|
||||
"The Google Cloud guide to [Profile model training performance using Profiler](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-profiler) provides detailed instructions for accessing the Vertex AI TensorBoard Profiler dashboard and capturing a profiling session.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TpV-iwP9qw9c"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "WR-ZhQ9XwpRI"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"job.delete()\n",
|
||||
"tensorboard.delete()\n",
|
||||
"\n",
|
||||
"if delete_bucket and \"BUCKET_URI\" in globals():\n",
|
||||
" ! gsutil -m rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "tensorboard_profiler_custom_training_with_prebuilt_container.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
+206
-37
@@ -61,7 +61,7 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This tutorial shows you how to create a distributed custom training job on Vertex AI that can handle large amounts of training data. \n",
|
||||
"This tutorial shows you how to create a distributed custom training job on Vertex AI that can handle large amounts of training data.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex AI Training](https://cloud.google.com/vertex-ai/docs/training/custom-training)."
|
||||
]
|
||||
@@ -78,8 +78,7 @@
|
||||
"\n",
|
||||
"This tutorial uses the following Google Cloud ML services:\n",
|
||||
"\n",
|
||||
"- `Vertex AI SDK`\n",
|
||||
"- `CustomContainerTrainingJob`\n",
|
||||
"- `Vertex AI Training`\n",
|
||||
"- `Artifact Registry`\n",
|
||||
"\n",
|
||||
"The steps performed include:\n",
|
||||
@@ -98,7 +97,7 @@
|
||||
"source": [
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"This tutorial uses the <a href=\"https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\">IRIS dataset</a>, which consists of different types of irises.\n"
|
||||
"This tutorial uses the <a href=\"https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\">IRIS dataset</a>, which predicts the iris species.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -108,7 +107,7 @@
|
||||
},
|
||||
"source": [
|
||||
"### Costs\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
@@ -477,7 +476,7 @@
|
||||
"id": "Xx_z9JQlrNwG"
|
||||
},
|
||||
"source": [
|
||||
"# Create a custom training Python package \n",
|
||||
"# Create a custom training Python package\n",
|
||||
"\n",
|
||||
"Before you can perform local training, you must a create a training script file and a docker file.\n",
|
||||
"\n",
|
||||
@@ -492,17 +491,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PYTHON_PACKAGE_APPLICATION_DIR = \"trainer\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "yjeHKqHwr4rV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PYTHON_PACKAGE_APPLICATION_DIR = \"trainer\"\n",
|
||||
"!mkdir -p $PYTHON_PACKAGE_APPLICATION_DIR"
|
||||
]
|
||||
},
|
||||
@@ -574,14 +563,28 @@
|
||||
" \"\"\"\n",
|
||||
" return subprocess.check_call(cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_chief_ip(cluster_config_dict):\n",
|
||||
" ip_address = cluster_config_dict['cluster']['workerpool0'][0].split(\":\")[0]\n",
|
||||
" if 'workerpool0' in cluster_config_dict['cluster']:\n",
|
||||
" ip_address = cluster_config_dict['cluster']['workerpool0'][0].split(\":\")[0]\n",
|
||||
" else:\n",
|
||||
" # if the job is not distributed, 'chief' will be populated instead of\n",
|
||||
" # workerpool0.\n",
|
||||
" ip_address = cluster_config_dict['cluster']['chief'][0].split(\":\")[0]\n",
|
||||
"\n",
|
||||
" print('The ip address of workerpool 0 is : {}'.format(ip_address))\n",
|
||||
" return ip_address\n",
|
||||
"\n",
|
||||
"def get_chief_port(cluster_config_dict):\n",
|
||||
" print(\"The open port is: {}\".format(cluster_config_dict['open_ports'][0]))\n",
|
||||
" return cluster_config_dict['open_ports'][0]\n",
|
||||
"\n",
|
||||
" if \"open_ports\" in cluster_config_dict:\n",
|
||||
" port = cluster_config_dict['open_ports'][0]\n",
|
||||
" else:\n",
|
||||
" # Use any port for the non-distributed job.\n",
|
||||
" port = 7777\n",
|
||||
" print(\"The open port is: {}\".format(port))\n",
|
||||
"\n",
|
||||
" return port\n",
|
||||
"\n",
|
||||
"if __name__ == '__main__':\n",
|
||||
" cluster_config_str = os.environ.get('CLUSTER_SPEC')\n",
|
||||
@@ -599,7 +602,7 @@
|
||||
" proc_scheduler = launch('dask-scheduler --dashboard --dashboard-address 8888 --port {} &'.format(chief_port))\n",
|
||||
" print('Done the dask scheduler.', flush=True)\n",
|
||||
"\n",
|
||||
" client = Client(chief_address)\n",
|
||||
" client = Client(chief_address, timeout=1200)\n",
|
||||
" print('Waiting the scheduler to be connected.', flush=True)\n",
|
||||
" client.wait_for_workers(1)\n",
|
||||
"\n",
|
||||
@@ -610,7 +613,7 @@
|
||||
" wait(X)\n",
|
||||
" wait(y)\n",
|
||||
" dtrain = DaskDMatrix(client, X, y)\n",
|
||||
" \n",
|
||||
"\n",
|
||||
" output = xgb.dask.train(client, XGB_PARAMS, dtrain, num_boost_round=100, evals=[(dtrain, 'train')])\n",
|
||||
" print(\"Output: {}\".format(output), flush=True)\n",
|
||||
" print(\"Saving file to: {}\".format(MODEL_FILE), flush=True)\n",
|
||||
@@ -623,6 +626,8 @@
|
||||
" blob.upload_from_filename(MODEL_FILE)\n",
|
||||
" print(\"Saved file to: {}/{}\".format(MODEL_DIR, MODEL_FILE), flush=True)\n",
|
||||
"\n",
|
||||
" # Waiting 10 mins to connect the Dask dashboard\n",
|
||||
" time.sleep(60 * 10)\n",
|
||||
" client.shutdown()\n",
|
||||
"\n",
|
||||
" else:\n",
|
||||
@@ -630,7 +635,10 @@
|
||||
" client = Client(chief_address, timeout=1200)\n",
|
||||
" print('client: {}.'.format(client), flush=True)\n",
|
||||
" launch('dask-worker {}'.format(chief_address))\n",
|
||||
" print('Done with the dask worker.', flush=True)\n"
|
||||
" print('Done with the dask worker.', flush=True)\n",
|
||||
"\n",
|
||||
" # Waiting 10 mins to connect the Dask dashboard\n",
|
||||
" time.sleep(60 * 10)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -639,7 +647,8 @@
|
||||
"id": "MxsT4Vaos2W5"
|
||||
},
|
||||
"source": [
|
||||
"### Write the docker file"
|
||||
"### Write the docker file\n",
|
||||
"The docker file is used to build the custom training container and passed to the Vertex Training."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -654,14 +663,20 @@
|
||||
"FROM us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-9:latest\n",
|
||||
"WORKDIR /root\n",
|
||||
"\n",
|
||||
"# Update the keyring in order to run apt-get update.\n",
|
||||
"RUN rm -rf /usr/share/keyrings/cloud.google.gpg\n",
|
||||
"RUN rm -rf /etc/apt/sources.list.d/google-cloud-sdk.list\n",
|
||||
"RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\n",
|
||||
"RUN echo \"deb https://packages.cloud.google.com/apt cloud-sdk main\" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list\n",
|
||||
"\n",
|
||||
"RUN apt-get update\n",
|
||||
"RUN apt-get install -y telnet netcat iputils-ping net-tools\n",
|
||||
"RUN python3.8 -m pip install dask==2022.7.1 distributed==2022.7.1 bokeh==2.1.1 dask-cuda --upgrade\n",
|
||||
"RUN python3.8 -m pip install 'xgboost>=1.4.2' 'dask-ml[complete]==2022.5.27' #'dask[complete]==2022.7,1' --upgrade\n",
|
||||
"RUN python3.8 -m pip install 'xgboost>=1.4.2' 'dask-ml[complete]==2022.5.27' 'dask[complete]==2022.7.1' --upgrade\n",
|
||||
"RUN python3.8 -m pip install dask==2022.7.1 distributed==2022.7.1 bokeh==2.4.3 dask-cuda==22.8.0 --upgrade\n",
|
||||
"RUN python3.8 -m pip install gcsfs --upgrade\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Make sure gsutil will use the default service account\n",
|
||||
"# Make sure gsutil will use the default service account\n",
|
||||
"RUN echo '[GoogleCompute]\\nservice_account = default' > /etc/boto.cfg\n",
|
||||
"\n",
|
||||
"# Copies the trainer code\n",
|
||||
@@ -736,10 +751,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"DEPLOY_IMAGE = (\n",
|
||||
"TRAIN_IMAGE = (\n",
|
||||
" f\"{REGION}-docker.pkg.dev/\" + PROJECT_ID + f\"/{PRIVATE_REPO}\" + \"/dask_support\"\n",
|
||||
")\n",
|
||||
"print(\"Deployment:\", DEPLOY_IMAGE)"
|
||||
"print(\"Deployment:\", TRAIN_IMAGE)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -788,8 +803,8 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if not IS_COLAB:\n",
|
||||
" ! docker build -t $DEPLOY_IMAGE -f Dockerfile .\n",
|
||||
" ! docker push $DEPLOY_IMAGE"
|
||||
" ! docker build -t $TRAIN_IMAGE -f Dockerfile .\n",
|
||||
" ! docker push $TRAIN_IMAGE"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -812,7 +827,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if IS_COLAB:\n",
|
||||
" ! gcloud builds submit --timeout=1800s --region={REGION} --tag $DEPLOY_IMAGE"
|
||||
" ! gcloud builds submit --timeout=1800s --region={REGION} --tag $TRAIN_IMAGE"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -860,11 +875,12 @@
|
||||
"replica_count = 2\n",
|
||||
"machine_type = \"n1-standard-4\"\n",
|
||||
"display_name = \"test_display_name\"\n",
|
||||
"DEPLOY_IMAGE = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-8:latest\"\n",
|
||||
"\n",
|
||||
"custom_container_training_job = aiplatform.CustomContainerTrainingJob(\n",
|
||||
" display_name=display_name,\n",
|
||||
" model_serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-8:latest\",\n",
|
||||
" container_uri=DEPLOY_IMAGE,\n",
|
||||
" model_serving_container_image_uri=DEPLOY_IMAGE,\n",
|
||||
" container_uri=TRAIN_IMAGE,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"custom_container_training_job.run(\n",
|
||||
@@ -886,6 +902,157 @@
|
||||
"print(f\"GCS Output URI Prefix: {gcs_output_uri_prefix}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tVktIbToRpmR"
|
||||
},
|
||||
"source": [
|
||||
"### Access the Dask dashboard"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "uVvxLj8GRsM6"
|
||||
},
|
||||
"source": [
|
||||
"You can also create a training job with gcloud command. With gcloud command, you can specify the field enableWebAccess and enableDashboardAccess. enableWebAccess enables the interactive shell for the job and enableDashboardAccess allows the dask dashboard to be accessed."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "pkOQtyDsRwsS"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash -s \"$BUCKET_URI/output\" \"$TRAIN_IMAGE\"\n",
|
||||
"\n",
|
||||
"cat <<EOF >config.yaml\n",
|
||||
"enableDashboardAccess: true\n",
|
||||
"enableWebAccess: true\n",
|
||||
"# Creates two worker pool. The first worker pool is a chief and the second is\n",
|
||||
"# a worker.\n",
|
||||
"workerPoolSpecs:\n",
|
||||
" - machineSpec:\n",
|
||||
" machineType: n1-standard-8\n",
|
||||
" replicaCount: 1\n",
|
||||
" containerSpec:\n",
|
||||
" imageUri: $2\n",
|
||||
" - machineSpec:\n",
|
||||
" machineType: n1-standard-8\n",
|
||||
" replicaCount: 1\n",
|
||||
" containerSpec:\n",
|
||||
" imageUri: $2\n",
|
||||
"baseOutputDirectory:\n",
|
||||
" outputUriPrefix: $1\n",
|
||||
"EOF\n",
|
||||
"cat config.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d5FLoTWzSNw7"
|
||||
},
|
||||
"source": [
|
||||
"The following command creates a training job."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "1MPj-NnpSQ1U"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gcloud ai custom-jobs create --region=us-central1 --config=config.yaml --display-name={display_name}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "onb40Ge0SVKh"
|
||||
},
|
||||
"source": [
|
||||
"Once the job is created. You can use the output `gcloud ai custom-jobs describe` command to print the field webAccessUris. The interactive shell has the key with the format \"workerpool0-0\", while the dashboard uri has the key with the format \"workerpool0-0:\" + port number. Note: You have to access the links while the job is running."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "FqFwDvWCSYFX"
|
||||
},
|
||||
"source": [
|
||||
"#### Troubleshooting"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "MjbPElukSiZt"
|
||||
},
|
||||
"source": [
|
||||
"The [interactive shell](https://cloud.google.com/vertex-ai/docs/training/monitor-debug-interactive-shell) can be used to debugging the access of the dask dashboard. You can get the dashboard point by the following command."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "z9eNOtYUTzJW"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Note the following command should run inside the interactive shell.\n",
|
||||
"# printenv | grep AIP_DASHBOARD_PORT"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "2SCRLCkpUDNM"
|
||||
},
|
||||
"source": [
|
||||
"Then you can check if there are dashboard monitoring the port."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "fK74qU78ULPS"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Note the following command should run inside the interactive shell.\n",
|
||||
"# netstat -ntlp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8ucKHMGFUUF4"
|
||||
},
|
||||
"source": [
|
||||
"You can manually turn up the dashboard instance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gJrjiePvUip0"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Note the following command should run inside the interactive shell.\n",
|
||||
"# dask-scheduler --dashboard-address :port_number"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -919,7 +1086,8 @@
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Cloud Storage Bucket"
|
||||
"- Cloud Storage Bucket\n",
|
||||
"- Cloud Vertex Training Job"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -936,13 +1104,14 @@
|
||||
"! gsutil rm -rf $gcs_output_uri_prefix\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
" ! gsutil rm -r $BUCKET_URI"
|
||||
" ! gsutil rm -r $BUCKET_URI\n",
|
||||
"\n",
|
||||
"custom_container_training_job.delete()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "xgboost_data_parallel_training_on_cpu_using_dask.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user