mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
Made some minor changes to sdk-metric-parameter-tracking-for-locally-trained-models (#480)
* Added to correct path * Ran linter test * Made some changes * Ran linter test Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Tharun-SpringML
co-authored by
Andrew Ferlitsch
Andrew Ferlitsch
parent
741c6732f1
commit
82bffb87f1
-877
@@ -1,877 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ur8xi4C7S06n"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "JAPoU8Sm5E6e"
|
||||
},
|
||||
"source": [
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WBFL9LagqmwT"
|
||||
},
|
||||
"source": [
|
||||
"#Vertex AI: Track parameters and metrics for locally trained models"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "tvgnzT1CKxrO"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to track metrics and parameters for ML training jobs and analyze this metadata using Vertex SDK for Python.\n",
|
||||
"\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
"In this notebook, we will train a simple distributed neural network (DNN) model to predict automobile's miles per gallon (MPG) based on automobile information in the [auto-mpg dataset](https://www.kaggle.com/devanshbesain/exploration-and-analysis-auto-mpg).\n",
|
||||
"\n",
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this notebook, you will learn how to use Vertex SDK for Python to:\n",
|
||||
"\n",
|
||||
" * Track parameters and metrics for a locally trainined model.\n",
|
||||
" * Extract and perform analysis for all parameters and metrics within an Experiment.\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ze4-nDLfK4pw"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your local development environment\n",
|
||||
"\n",
|
||||
"**If you are using Colab or Google Cloud Notebooks**, your environment already meets\n",
|
||||
"all the requirements to run this notebook. You can skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "gCuSR8GkAgzl"
|
||||
},
|
||||
"source": [
|
||||
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
|
||||
"You need the following:\n",
|
||||
"\n",
|
||||
"* The Google Cloud SDK\n",
|
||||
"* Git\n",
|
||||
"* Python 3\n",
|
||||
"* virtualenv\n",
|
||||
"* Jupyter notebook running in a virtual environment with Python 3\n",
|
||||
"\n",
|
||||
"The Google Cloud guide to [Setting up a Python development\n",
|
||||
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
|
||||
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
|
||||
"for meeting these requirements. The following steps provide a condensed set of\n",
|
||||
"instructions:\n",
|
||||
"\n",
|
||||
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
|
||||
"\n",
|
||||
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
|
||||
"\n",
|
||||
"1. [Install\n",
|
||||
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
|
||||
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
|
||||
"\n",
|
||||
"1. To install Jupyter, run `pip install jupyter` on the\n",
|
||||
"command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
|
||||
"\n",
|
||||
"1. Open this notebook in the Jupyter Notebook Dashboard."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "i7EUnXsZhAGF"
|
||||
},
|
||||
"source": [
|
||||
"### Install additional packages\n",
|
||||
"\n",
|
||||
"Run the following commands to install the Vertex SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "IaYsrh0Tc17L"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
" USER_FLAG = \"\"\n",
|
||||
"else:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "wyy5Lbnzg5fi"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!python3 -m pip install {USER_FLAG} google-cloud-aiplatform --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hhq5zEbGg0XX"
|
||||
},
|
||||
"source": [
|
||||
"### Restart the kernel\n",
|
||||
"\n",
|
||||
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "EzrelQZ22IZj"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Automatically restart kernel after installs\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "lWEdiXsJg0XY"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin\n",
|
||||
"\n",
|
||||
"### Select a GPU runtime\n",
|
||||
"\n",
|
||||
"**Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select \"Runtime --> Change runtime type > GPU\"**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "BF1j6f9HApxa"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WReHDGG5g0XY"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "oM1iC_MfAts1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"\"\n",
|
||||
"\n",
|
||||
"# Get your Google Cloud project ID from gcloud\n",
|
||||
"if not os.getenv(\"IS_TESTING\"):\n",
|
||||
" shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
|
||||
" PROJECT_ID = shell_output[0]\n",
|
||||
" print(\"Project ID: \", PROJECT_ID)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "qJYoRfYng0XZ"
|
||||
},
|
||||
"source": [
|
||||
"Otherwise, set your project ID here."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "riG_qUokg0XZ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
|
||||
" PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "06571eb4063b"
|
||||
},
|
||||
"source": [
|
||||
"#### Timestamp\n",
|
||||
"\n",
|
||||
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "697568e92bd6"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dr--iN2kAylZ"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already\n",
|
||||
"authenticated. Skip this step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "sBCra4QMA2wR"
|
||||
},
|
||||
"source": [
|
||||
"**If you are using Colab**, run the cell below and follow the instructions\n",
|
||||
"when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"1. In the Cloud Console, go to the [**Create service account key**\n",
|
||||
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
|
||||
"\n",
|
||||
"2. Click **Create service account**.\n",
|
||||
"\n",
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
"\n",
|
||||
"6. Enter the path to your service account key as the\n",
|
||||
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "PyQmSRbKA8r-"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "XoEqT2Y4DJmf"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries and define constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Y9Uo3tifg1kx"
|
||||
},
|
||||
"source": [
|
||||
"Import required libraries."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "pRUOFELefqf1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import pandas as pd\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from tensorflow.python.keras import Sequential, layers\n",
|
||||
"from tensorflow.python.keras.utils import data_utils"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "xtXZWmYqJ1bh"
|
||||
},
|
||||
"source": [
|
||||
"Define some constants"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JIOrI-hoJ46P"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"EXPERIMENT_NAME = \"\" # @param {type:\"string\"}\n",
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "jWQLXXNVN4Lv"
|
||||
},
|
||||
"source": [
|
||||
"If EXEPERIMENT_NAME is not set, set a default one below:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "Q1QInYWOKsmo"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if EXPERIMENT_NAME == \"\" or EXPERIMENT_NAME is None:\n",
|
||||
" EXPERIMENT_NAME = \"my-experiment-\" + TIMESTAMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Xuny18aMcWDb"
|
||||
},
|
||||
"source": [
|
||||
"## Concepts\n",
|
||||
"\n",
|
||||
"To better understanding how parameters and metrics are stored and organized, we'd like to introduce the following concepts:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "NThDci5bp0Uw"
|
||||
},
|
||||
"source": [
|
||||
"### Experiment\n",
|
||||
"Experiments describe a context that groups your runs and the artifacts you create into a logical session. For example, in this notebook you create an Experiment and log data to that experiment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "SAyRR3Ydp4X5"
|
||||
},
|
||||
"source": [
|
||||
"### Run\n",
|
||||
"A run represents a single path/avenue that you executed while performing an experiment. A run includes artifacts that you used as inputs or outputs, and parameters that you used in this execution. An Experiment can contain multiple runs. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "l1YW2pgyegFP"
|
||||
},
|
||||
"source": [
|
||||
"## Getting started tracking parameters and metrics\n",
|
||||
"\n",
|
||||
"You can use the Vertex SDK for Python to track metrics and parameters for models trained locally. \n",
|
||||
"\n",
|
||||
"In the following example, you train a simple distributed neural network (DNN) model to predict automobile's miles per gallon (MPG) based on automobile information in the [auto-mpg dataset](https://www.kaggle.com/devanshbesain/exploration-and-analysis-auto-mpg)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "KPY41M9_AhZU"
|
||||
},
|
||||
"source": [
|
||||
"### Load and process the training dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "bfMQSmRuUuX-"
|
||||
},
|
||||
"source": [
|
||||
"Download and process the dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "RiQuMv4bmpuV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def read_data(uri):\n",
|
||||
" dataset_path = data_utils.get_file(\"auto-mpg.data\", uri)\n",
|
||||
" column_names = [\n",
|
||||
" \"MPG\",\n",
|
||||
" \"Cylinders\",\n",
|
||||
" \"Displacement\",\n",
|
||||
" \"Horsepower\",\n",
|
||||
" \"Weight\",\n",
|
||||
" \"Acceleration\",\n",
|
||||
" \"Model Year\",\n",
|
||||
" \"Origin\",\n",
|
||||
" ]\n",
|
||||
" raw_dataset = pd.read_csv(\n",
|
||||
" dataset_path,\n",
|
||||
" names=column_names,\n",
|
||||
" na_values=\"?\",\n",
|
||||
" comment=\"\\t\",\n",
|
||||
" sep=\" \",\n",
|
||||
" skipinitialspace=True,\n",
|
||||
" )\n",
|
||||
" dataset = raw_dataset.dropna()\n",
|
||||
" dataset[\"Origin\"] = dataset[\"Origin\"].map(\n",
|
||||
" lambda x: {1: \"USA\", 2: \"Europe\", 3: \"Japan\"}.get(x)\n",
|
||||
" )\n",
|
||||
" dataset = pd.get_dummies(dataset, prefix=\"\", prefix_sep=\"\")\n",
|
||||
" return dataset\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"dataset = read_data(\n",
|
||||
" \"http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Y06J7A7yU21t"
|
||||
},
|
||||
"source": [
|
||||
"Split dataset for training and testing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "p5JBCBKyH-NC"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_test_split(dataset, split_frac=0.8, random_state=0):\n",
|
||||
" train_dataset = dataset.sample(frac=split_frac, random_state=random_state)\n",
|
||||
" test_dataset = dataset.drop(train_dataset.index)\n",
|
||||
" train_labels = train_dataset.pop(\"MPG\")\n",
|
||||
" test_labels = test_dataset.pop(\"MPG\")\n",
|
||||
"\n",
|
||||
" return train_dataset, test_dataset, train_labels, test_labels\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"train_dataset, test_dataset, train_labels, test_labels = train_test_split(dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "gaNNTFPaU7KT"
|
||||
},
|
||||
"source": [
|
||||
"Normalize the features in the dataset for better model performance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "VGq5QCoyIEWJ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def normalize_dataset(train_dataset, test_dataset):\n",
|
||||
" train_stats = train_dataset.describe()\n",
|
||||
" train_stats = train_stats.transpose()\n",
|
||||
"\n",
|
||||
" def norm(x):\n",
|
||||
" return (x - train_stats[\"mean\"]) / train_stats[\"std\"]\n",
|
||||
"\n",
|
||||
" normed_train_data = norm(train_dataset)\n",
|
||||
" normed_test_data = norm(test_dataset)\n",
|
||||
"\n",
|
||||
" return normed_train_data, normed_test_data\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"normed_train_data, normed_test_data = normalize_dataset(train_dataset, test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "UBXUgxgqA_GB"
|
||||
},
|
||||
"source": [
|
||||
"### Define ML model and training function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "66odBYKrIN4q"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train(\n",
|
||||
" train_data,\n",
|
||||
" train_labels,\n",
|
||||
" num_units=64,\n",
|
||||
" activation=\"relu\",\n",
|
||||
" dropout_rate=0.0,\n",
|
||||
" validation_split=0.2,\n",
|
||||
" epochs=1000,\n",
|
||||
"):\n",
|
||||
"\n",
|
||||
" model = Sequential(\n",
|
||||
" [\n",
|
||||
" layers.Dense(\n",
|
||||
" num_units,\n",
|
||||
" activation=activation,\n",
|
||||
" input_shape=[len(train_dataset.keys())],\n",
|
||||
" ),\n",
|
||||
" layers.Dropout(rate=dropout_rate),\n",
|
||||
" layers.Dense(num_units, activation=activation),\n",
|
||||
" layers.Dense(1),\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[\"mae\", \"mse\"])\n",
|
||||
" print(model.summary())\n",
|
||||
"\n",
|
||||
" history = model.fit(\n",
|
||||
" train_data, train_labels, epochs=epochs, validation_split=validation_split\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" return model, history"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "O8XJZB3gR8eL"
|
||||
},
|
||||
"source": [
|
||||
"### Initialize the Vertex AI SDK for Python and create an Experiment\n",
|
||||
"\n",
|
||||
"Initialize the *client* for Vertex AI and create an experiment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "o_wnT10RJ7-W"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"aiplatform.init(project=PROJECT_ID, location=REGION, experiment=EXPERIMENT_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "u-iTnzt3B6Z_"
|
||||
},
|
||||
"source": [
|
||||
"### Start several model training runs\n",
|
||||
"\n",
|
||||
"Training parameters and metrics are logged for each run."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "i2wnpu8_7JfV"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"parameters = [\n",
|
||||
" {\"num_units\": 16, \"epochs\": 3, \"dropout_rate\": 0.1},\n",
|
||||
" {\"num_units\": 16, \"epochs\": 10, \"dropout_rate\": 0.1},\n",
|
||||
" {\"num_units\": 16, \"epochs\": 10, \"dropout_rate\": 0.2},\n",
|
||||
" {\"num_units\": 32, \"epochs\": 10, \"dropout_rate\": 0.1},\n",
|
||||
" {\"num_units\": 32, \"epochs\": 10, \"dropout_rate\": 0.2},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for i, params in enumerate(parameters):\n",
|
||||
" aiplatform.start_run(run=f\"auto-mpg-local-run-{i}\")\n",
|
||||
" aiplatform.log_params(params)\n",
|
||||
" model, history = train(\n",
|
||||
" normed_train_data,\n",
|
||||
" train_labels,\n",
|
||||
" num_units=params[\"num_units\"],\n",
|
||||
" activation=\"relu\",\n",
|
||||
" epochs=params[\"epochs\"],\n",
|
||||
" dropout_rate=params[\"dropout_rate\"],\n",
|
||||
" )\n",
|
||||
" aiplatform.log_metrics(\n",
|
||||
" {metric: values[-1] for metric, values in history.history.items()}\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" loss, mae, mse = model.evaluate(normed_test_data, test_labels, verbose=2)\n",
|
||||
" aiplatform.log_metrics({\"eval_loss\": loss, \"eval_mae\": mae, \"eval_mse\": mse})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "jZLrJZTfL7tE"
|
||||
},
|
||||
"source": [
|
||||
"### Extract parameters and metrics into a dataframe for analysis"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "A1PqKxlpOZa2"
|
||||
},
|
||||
"source": [
|
||||
"We can also extract all parameters and metrics associated with any Experiment into a dataframe for further analysis."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "jbRf1WoH_vbY"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"experiment_df = aiplatform.get_experiment_df()\n",
|
||||
"experiment_df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "EYuYgqVCMKU1"
|
||||
},
|
||||
"source": [
|
||||
"### Visualizing an experiment's parameters and metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "r8orCj8iJuO1"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.rcParams[\"figure.figsize\"] = [15, 5]\n",
|
||||
"\n",
|
||||
"ax = pd.plotting.parallel_coordinates(\n",
|
||||
" experiment_df.reset_index(level=0),\n",
|
||||
" \"run_name\",\n",
|
||||
" cols=[\n",
|
||||
" \"param.num_units\",\n",
|
||||
" \"param.dropout_rate\",\n",
|
||||
" \"param.epochs\",\n",
|
||||
" \"metric.loss\",\n",
|
||||
" \"metric.val_loss\",\n",
|
||||
" \"metric.eval_loss\",\n",
|
||||
" ],\n",
|
||||
" color=[\"blue\", \"green\", \"pink\", \"red\"],\n",
|
||||
")\n",
|
||||
"ax.set_yscale(\"symlog\")\n",
|
||||
"ax.legend(bbox_to_anchor=(1.0, 0.5))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WTHvPMweMlP1"
|
||||
},
|
||||
"source": [
|
||||
"## Visualizing experiments in Cloud Console"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "F19_5lw0MqXv"
|
||||
},
|
||||
"source": [
|
||||
"Run the following to get the URL of Vertex AI Experiments for your project.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "GmN9vE9pqqzt"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Vertex AI Experiments:\")\n",
|
||||
"print(\n",
|
||||
" f\"https://console.cloud.google.com/ai/platform/experiments/experiments?folder=&organizationId=&project={PROJECT_ID}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "TpV-iwP9qw9c"
|
||||
},
|
||||
"source": [
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "sdk-metric-parameter-tracking-for-locally-trained-models.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
+38
-38
@@ -8,7 +8,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2021 Google LLC\n",
|
||||
"# Copyright 2022 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
@@ -32,16 +32,22 @@
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>"
|
||||
]
|
||||
},
|
||||
@@ -51,7 +57,7 @@
|
||||
"id": "WBFL9LagqmwT"
|
||||
},
|
||||
"source": [
|
||||
"#Vertex AI: Track parameters and metrics for locally trained models"
|
||||
"## Vertex AI: Track parameters and metrics for locally trained models"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -62,7 +68,7 @@
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to track metrics and parameters for ML training jobs and analyze this metadata using Vertex AI SDK.\n",
|
||||
"This notebook demonstrates how to track metrics and parameters for ML training jobs and analyze this metadata using Vertex SDK for Python.\n",
|
||||
"\n",
|
||||
"### Dataset\n",
|
||||
"\n",
|
||||
@@ -70,9 +76,9 @@
|
||||
"\n",
|
||||
"### Objective\n",
|
||||
"\n",
|
||||
"In this notebook, you will learn how to use Vertex AI SDK to:\n",
|
||||
"In this notebook, you will learn how to use Vertex SDK for Python to:\n",
|
||||
"\n",
|
||||
" * Track parameters and metrics for a locally trainined model.\n",
|
||||
" * Track parameters and metrics for a locally trained model.\n",
|
||||
" * Extract and perform analysis for all parameters and metrics within an Experiment.\n",
|
||||
"\n",
|
||||
"### Costs \n",
|
||||
@@ -148,7 +154,7 @@
|
||||
"source": [
|
||||
"### Install additional packages\n",
|
||||
"\n",
|
||||
"Run the following commands to install the Vertex AI SDK and packages used in this notebook."
|
||||
"Run the following commands to install the Vertex SDK for Python."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -159,26 +165,14 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# The Google Cloud Notebook product has specific requirements\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
" USER_FLAG = \"\"\n",
|
||||
"if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
"else:\n",
|
||||
" USER_FLAG = \"--user\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "MCQDRsnE3uzz"
|
||||
},
|
||||
"source": [
|
||||
"Install Vertex AI SDK and tensorflow for training and evaluation model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
@@ -187,8 +181,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! pip install {USER_FLAG} --upgrade tensorflow\n",
|
||||
"! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
|
||||
"! pip3 install -U tensorflow==2.8 $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -364,9 +358,9 @@
|
||||
"3. In the **Service account name** field, enter a name, and\n",
|
||||
" click **Create**.\n",
|
||||
"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"AI Platform\"\n",
|
||||
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
|
||||
"into the filter box, and select\n",
|
||||
" **AI Platform Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
|
||||
"local environment.\n",
|
||||
@@ -392,9 +386,7 @@
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebooks, then don't execute this code\n",
|
||||
"IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
|
||||
"\n",
|
||||
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
@@ -437,7 +429,7 @@
|
||||
"import pandas as pd\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from tensorflow.python.keras import Sequential, layers\n",
|
||||
"from tensorflow.python.lib.io import file_io"
|
||||
"from tensorflow.python.keras.utils import data_utils"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -458,7 +450,9 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"EXPERIMENT_NAME = \"\" # @param {type:\"string\"}\n",
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}"
|
||||
"REGION = \"[your-region]\" # @param {type:\"string\"}\n",
|
||||
"if REGION == \"[your-region]\":\n",
|
||||
" REGION = \"us-central1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -521,7 +515,7 @@
|
||||
"source": [
|
||||
"## Getting started tracking parameters and metrics\n",
|
||||
"\n",
|
||||
"You can use the Vertex AI SDK to track metrics and parameters for models trained locally. \n",
|
||||
"You can use the Vertex SDK for Python to track metrics and parameters for models trained locally. \n",
|
||||
"\n",
|
||||
"In the following example, you train a simple distributed neural network (DNN) model to predict automobile's miles per gallon (MPG) based on automobile information in the [auto-mpg dataset](https://www.kaggle.com/devanshbesain/exploration-and-analysis-auto-mpg)."
|
||||
]
|
||||
@@ -552,7 +546,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def read_data(file_path):\n",
|
||||
"def read_data(uri):\n",
|
||||
" dataset_path = data_utils.get_file(\"auto-mpg.data\", uri)\n",
|
||||
" column_names = [\n",
|
||||
" \"MPG\",\n",
|
||||
" \"Cylinders\",\n",
|
||||
@@ -563,9 +558,8 @@
|
||||
" \"Model Year\",\n",
|
||||
" \"Origin\",\n",
|
||||
" ]\n",
|
||||
" with file_io.FileIO(file_path, \"r\") as f:\n",
|
||||
" raw_dataset = pd.read_csv(\n",
|
||||
" f,\n",
|
||||
" dataset_path,\n",
|
||||
" names=column_names,\n",
|
||||
" na_values=\"?\",\n",
|
||||
" comment=\"\\t\",\n",
|
||||
@@ -580,7 +574,9 @@
|
||||
" return dataset\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"dataset = read_data(\"gs://cloud-samples-data/ai-platform/auto_mpg/auto-mpg.data\")"
|
||||
"dataset = read_data(\n",
|
||||
" \"http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -870,7 +866,11 @@
|
||||
"## Cleaning up\n",
|
||||
"\n",
|
||||
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial."
|
||||
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
||||
"\n",
|
||||
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
|
||||
"\n",
|
||||
"- Experiment (Can be deleted manually in the GCP Console UI)"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user