xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Sdk metric parameter tracking for custom jobs (#844)

Browse Source 
* Replaced timestamp with UUID

* Ran Linter test

* Removed local kernel from metadata

* ran linter test

Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
Co-authored-by: Ivan Cheung <ivans.mailbox@gmail.com>
This commit is contained in:
SamyuktaDR
2022-09-16 11:43:45 -07:00
committed by GitHub
co-authored by Andrew Ferlitsch Ivan Cheung
parent 68b53e0d32
commit aac271eacc
1 changed files with 72 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb
+72 -69
View File
@@ -8,7 +8,7 @@
},
"outputs": [],
"source": [
"# Copyright 2022 Google LLC\n",
"# Copyright 2021 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
@@ -29,23 +29,21 @@
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Vertex AI: Track parameters and metrics for custom training jobs\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
@@ -56,39 +54,48 @@
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
"id": "j9gUDU_3vV9d"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to track metrics and parameters for `Vertex AI` custom training jobs, and how to perform detailed analysis using this data."
"# Vertex AI: Track parameters and metrics for custom training jobs"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "37147bd9c3c4"
"id": "2e0464050974"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to track metrics and parameters for Vertex AI custom training jobs, and how to perform detailed analysis using this data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b95ab729fccd"
},
"source": [
"### Objective\n",
"\n",
"In this notebook, you learn how to use `Vertex ML Metadata` to track training parameters and evaluation metrics.\n",
"In this notebook, you will learn how to use Vertex AI SDK for Python to:\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex ML Metadata`\n",
"- `Vertex AI Experiments`\n",
"This tutorial uses the following Google Cloud ML services and resources:\n",
"- Vertex AI Dataset\n",
"- Vertex AI Model\n",
"- Vertex AI Endpoint\n",
"- Vertex AI Custom Training Job\n",
"\n",
"The steps performed include:\n",
"\n",
"- Track parameters and metrics for a `Vertex AI` custom trained model.\n",
"- Track training parameters and prediction metrics for a custom training job.\n",
"- Extract and perform analysis for all parameters and metrics within an Experiment."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "96cb18467417"
"id": "9fd87cf689bf"
},
"source": [
"### Dataset\n",
@@ -99,7 +106,7 @@
{
"cell_type": "markdown",
"metadata": {
"id": "c831245dc1d5"
"id": "tvgnzT1CKxrO"
},
"source": [
"### Costs \n",
@@ -181,14 +188,14 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IaYsrh0Tc17L"
"id": "qblyW_dcyOQA"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
@@ -198,9 +205,10 @@
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
"! pip3 install -U tensorflow $USER_FLAG -q\n",
"! pip3 install scikit-learn {USER_FLAG} -q"
"\n",
"! pip3 install -U tensorflow $USER_FLAG\n",
"! python3 -m pip3 install {USER_FLAG} google-cloud-aiplatform --upgrade\n",
"! pip3 install scikit-learn {USER_FLAG}"
]
},
{
@@ -285,7 +293,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oM1iC_MfAts1"
"id": "cde8e0876d62"
},
"outputs": [],
"source": [
@@ -296,11 +304,11 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "riG_qUokg0XZ"
"id": "oM1iC_MfAts1"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"[your-project-id]\" or PROJECT_ID == \"\" or PROJECT_ID is None:\n",
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
@@ -321,7 +329,7 @@
{
"cell_type": "markdown",
"metadata": {
"id": "region"
"id": "47bc07d4231b"
},
"source": [
"#### Region\n",
@@ -335,14 +343,14 @@
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "region"
"id": "959545da671a"
},
"outputs": [],
"source": [
@@ -358,9 +366,9 @@
"id": "06571eb4063b"
},
"source": [
"#### Timestamp\n",
"#### UUID\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
]
},
{
@@ -371,9 +379,16 @@
},
"outputs": [],
"source": [
"from datetime import datetime\n",
"import random\n",
"import string\n",
"\n",
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
"\n",
"# Generate a uuid of length 8\n",
"def generate_uuid():\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=8))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
@@ -385,7 +400,7 @@
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench**, your environment is already\n",
"authenticated. "
"authenticated. Skip this step."
]
},
{
@@ -435,7 +450,6 @@
"# requests.\n",
"\n",
"# If on Google Cloud Notebooks, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
@@ -460,7 +474,7 @@
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"\n",
"When you submit a training job using the Cloud SDK, you upload a Python package\n",
"When you submit a training job using the Vertex AI SDK, you upload a Python package\n",
"containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
"the code from this package. In this tutorial, Vertex AI also saves the\n",
"trained model that results from your job in the same bucket. Using this model artifact, you can then\n",
@@ -492,8 +506,8 @@
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
" BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
" BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
@@ -615,7 +629,7 @@
"outputs": [],
"source": [
"if EXPERIMENT_NAME == \"\" or EXPERIMENT_NAME is None:\n",
" EXPERIMENT_NAME = \"my-experiment-\" + TIMESTAMP"
" EXPERIMENT_NAME = \"my-experiment-\" + UUID"
]
},
{
@@ -655,10 +669,10 @@
{
"cell_type": "markdown",
"metadata": {
"id": "9nokDKBAxwV8"
"id": "f8fd397cc4f6"
},
"source": [
"This example uses the Abalone Dataset. For more information about this dataset please visit: https://archive.ics.uci.edu/ml/datasets/abalone"
"### Download the Dataset to Cloud Storage"
]
},
{
@@ -681,9 +695,9 @@
"id": "35QVNhACqcTJ"
},
"source": [
"### Create a Vertex AI Dataset from a CSV\n",
"### Create a Vertex AI Tabular dataset from CSV data\n",
"\n",
"A Vertex AI Dataset can be used to create an AutoML model or a custom model. "
"A Vertex AI dataset can be used to create an AutoML model or a custom model. "
]
},
{
@@ -696,7 +710,7 @@
"source": [
"ds = aiplatform.TabularDataset.create(display_name=\"abalone\", gcs_source=[gcs_csv_path])\n",
"\n",
"print(ds.resource_name)"
"ds.resource_name"
]
},
{
@@ -707,7 +721,7 @@
"source": [
"### Write the training script\n",
"\n",
"Run the following cell to create the training script that is used in the sample custom training job."
"Next, you create the training script that is used in the sample custom training job."
]
},
{
@@ -735,9 +749,6 @@
" default=64, type=int,\n",
" help='Number of unit for first layer.')\n",
"args = parser.parse_args()\n",
"# uncomment and bump up replica_count for distributed training\n",
"# strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n",
"# tf.distribute.experimental_set_strategy(strategy)\n",
"\n",
"col_names = [\"Length\", \"Diameter\", \"Height\", \"Whole weight\", \"Shucked weight\", \"Viscera weight\", \"Shell weight\", \"Age\"]\n",
"target = \"Age\"\n",
@@ -771,7 +782,7 @@
"id": "Yp2clkOJSDhR"
},
"source": [
"### Launch a custom training job and track its trainig parameters on Vertex AI ML Metadata"
"### Launch a custom training job and track its trainig parameters on Vertex ML Metadata"
]
},
{
@@ -797,11 +808,7 @@
"id": "k_QorXXztzPH"
},
"source": [
"Start a new experiment run to track training parameters and start the training job. \n",
"\n",
"Prior to executing the training job, you call the `start_run()` method to initialize the start of the experiment, and then use the `log_params()` to log the parameters used in the experiment.\n",
"\n",
"*Note:* This operation will take around 10 mins."
"Start a new experiment run to track training parameters and start the training job. Note that this operation will take around 10 mins."
]
},
{
@@ -830,7 +837,7 @@
"id": "5vhDsMJNqcTW"
},
"source": [
"### Deploy Model and calculate prediction metrics"
"### Deploy model and calculate prediction metrics"
]
},
{
@@ -839,7 +846,7 @@
"id": "O-uCOL3Naap4"
},
"source": [
"Deploy model to Google Cloud. This operation may take a few minutes."
"Next, deploy your Vertex AI Model resource to a Vertex AI Endpoint resource. This operation will take 10-20 mins."
]
},
{
@@ -859,7 +866,7 @@
"id": "JY-5skFhasWs"
},
"source": [
"Once model is deployed, perform online prediction using the `abalone_test` dataset and calculate prediction metrics."
"### Prediction dataset preparation and online prediction"
]
},
{
@@ -868,6 +875,8 @@
"id": "saw50bqwa-dR"
},
"source": [
"Once model is deployed, perform online prediction using the `abalone_test` dataset and calculate prediction metrics.\n",
"\n",
"Prepare the prediction dataset."
]
},
@@ -920,7 +929,7 @@
"id": "_HphZ38obJeB"
},
"source": [
"### Perform online prediction"
"Perform online prediction."
]
},
{
@@ -932,7 +941,7 @@
"outputs": [],
"source": [
"prediction = endpoint.predict(test_dataset.tolist())\n",
"print(prediction)"
"prediction"
]
},
{
@@ -941,11 +950,7 @@
"id": "TDKiv_O7bNwE"
},
"source": [
"### Calculate and track prediction evaluation metrics.\n",
"\n",
"Next, log the evaluation metrics for your experiment.\n",
"\n",
"Once the experiment is completed, you call the `end_run()` method to indicate the end of tracking for the experiment."
"Calculate and track prediction evaluation metrics."
]
},
{
@@ -959,9 +964,7 @@
"mse = mean_squared_error(test_labels, prediction.predictions)\n",
"mae = mean_absolute_error(test_labels, prediction.predictions)\n",
"\n",
"aiplatform.log_metrics({\"mse\": mse, \"mae\": mae})\n",
"\n",
"aiplatform.end_run()"
"aiplatform.log_metrics({\"mse\": mse, \"mae\": mae})"
]
},
{