Compare commits

...
Author SHA1 Message Date
Andrew FerlitschandGitHub c24c93b7be outdated 2023-04-24 14:59:25 -07:00
@@ -1,669 +0,0 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2022 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Delete Outdated Experiments in Vertex AI TensorBoard\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/vertex-ai-samples/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "24743cf4a1e1"
},
"source": [
"**_NOTE_**: This notebook has been tested in the following environment:\n",
"\n",
"* Python version = 3.9"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Vertex AI will have a new Vertex AI TensorBoard billing model. From August 2023, it will change from monthly `$300/user` to monthly `$10/GB`. In preparation for this change, users need to delete old Vertex AI TensorBoard Experiments to avoid unnecessary storage costs when the pricing change takes place.\n",
"\n",
"Learn more about [Vertex AI TensorBoard](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d975e698c9a4"
},
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to delete outdated TensorBoard Experiments to avoid unnecessary storage costs.\n",
"\n",
"This tutorial uses the following Google Cloud ML services and resources:\n",
"\n",
"- Vertex AI Tensorboard\n",
"\n",
"The steps performed include:\n",
"\n",
"- How to delete the TB Experiment with a predefined key-value label pair `<label_key, label_value>`\n",
"\n",
"- How to delete the TB Experiments created before the `create_time`\n",
"\n",
"- How to delete the TB Experiments created before the `update_time`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "08d289fa873f"
},
"source": [
"### Dataset\n",
"\n",
"No dataset is used."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aed92deeb4a0"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
"and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7EUnXsZhAGF"
},
"source": [
"## Installation\n",
"\n",
"Install the following packages required to execute this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2b4ef9b72d43"
},
"outputs": [],
"source": [
"# Install the packages\n",
"import os\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" USER = \"--user\"\n",
"else:\n",
" USER = \"\"\n",
"! pip3 install {USER} --upgrade google-cloud-aiplatform"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "58707a750154"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f200f10a1da3"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BF1j6f9HApxa"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WReHDGG5g0XY"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oM1iC_MfAts1"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QAup21nz6LHk"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "74ccc9e52986"
},
"source": [
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "de775a3773ba"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "254614fa0c46"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ef21552ccea8"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "603adbbf0532"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6b2ccc891ed"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant permissions to your service account at https://cloud.google.com/marketplace/docs/grant-service-account-access."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "960505627ddf"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NUJOpq7_-nt6"
},
"source": [
"### Define constants\n",
"\n",
"Define variables you use in this tutorial. In particular, you set\n",
"\n",
"- `CREATE_TIME_CUT` : delete tensorboard experiments that were created before CREATE_TIME_CUT. For example, `2022-12-31`.\n",
"\n",
"- `UPDATE_TIME_CUT` : delete tensorboard experiments that were created before UPDATE_TIME_CUT. For example, `2022-12-31`.\n",
"\n",
"- `DETAILED_LOG` : a booled variable to see Tensorboard deletion progress. If True, it shows progress by experiments. Otherwise, it reports progress per 100 experiments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0AkyoMre-qe5"
},
"outputs": [],
"source": [
"CREATE_TIME_CUT = \"\" # @param {type:\"string\"}\n",
"\n",
"UPDATE_TIME_CUT = \"\" # @param {type:\"string\"}\n",
"\n",
"DETAILED_LOG = True # @param {type: \"boolean\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n68J5KJ4ERbF"
},
"source": [
"### Define Helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JIXC-tVrETET"
},
"outputs": [],
"source": [
"def clean_up_by_label(tensorboard_instance, label_key, label_value):\n",
" \"\"\"Delete the TB Experiment with the key-value label pair <label_key, label_value>\"\"\"\n",
" # List tensorboard experiments\n",
" tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
" tensorboard_name=tensorboard_instance.resource_name\n",
" )\n",
" # Get the number of tensorboard experiments\n",
" num_tensorboard_experiments = len(tensorboard_experiments)\n",
" # For each experiment\n",
" for i in range(num_tensorboard_experiments):\n",
" tensorboard_experiment = tensorboard_experiments[i]\n",
" if detailed_log or (i % 100 == 0):\n",
" print(\n",
" f\">>>checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
" )\n",
" # Get experiment labels\n",
" labels = tensorboard_experiment.labels\n",
" # Filter by label\n",
" if label_key in labels and labels[label_key] == label_value:\n",
" # Delete experiment\n",
" tensorboard_experiment.delete()\n",
"\n",
"\n",
"def clean_up_by_create_time(tensorboard_instance, create_time_cut):\n",
" \"\"\"Delete the TB Experiment with the `create_time`\"\"\"\n",
" # List tensorboard experiments\n",
" tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
" tensorboard_name=tensorboard_instance.resource_name, order_by=\"create_time\"\n",
" )\n",
" # Get the number of tensorboard experiments\n",
" num_tensorboard_experiments = len(tensorboard_experiments)\n",
" # For each experiment\n",
" for i in range(num_tensorboard_experiments):\n",
" tensorboard_experiment = tensorboard_experiments[i]\n",
" if detailed_log or (i % 100 == 0):\n",
" print(\n",
" f\">>> checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
" )\n",
" # Filter by create_time\n",
" if str(tensorboard_experiment.create_time) < create_time_cut:\n",
" # Delete experiment\n",
" tensorboard_experiment.delete()\n",
" else:\n",
" break\n",
"\n",
"\n",
"def clean_up_by_update_time(tensorboard_instance, update_time_cut):\n",
" \"\"\"Delete the TB Experiment with with the `update_time`\"\"\"\n",
" # List tensorboard experiments\n",
" tensorboard_experiments = aiplatform.TensorboardExperiment.list(\n",
" tensorboard_name=tensorboard_instance.resource_name, order_by=\"update_time\"\n",
" )\n",
" # Get the number of tensorboard experiments\n",
" num_tensorboard_experiments = len(tensorboard_experiments)\n",
" # For each experiment\n",
" for i in range(num_tensorboard_experiments):\n",
" tensorboard_experiment = tensorboard_experiments[i]\n",
" if detailed_log or (i % 100 == 0):\n",
" print(\n",
" f\">>> checking TB experiment [{i + 1}/{num_tensorboard_experiments}]: {tensorboard_experiment.resource_name}\"\n",
" )\n",
" # Filter by update_time\n",
" if str(tensorboard_experiment.update_time) < update_time_cut:\n",
" tensorboard_experiment.delete()\n",
" else:\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UCUOlTLYCT0B"
},
"source": [
"## Delete outdated Vertex AI TensorBoard Experiments"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EGfFVHhWDjkl"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "G1l-7Wft3jb6"
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e4672ef33691"
},
"source": [
"### Set delete outdated TensorBoard experiments \n",
"\n",
"Initialize a flag variable to start deleting outdated TensorBoard experiments and a flag variable to choose the deleting method. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8cdc326c9823"
},
"outputs": [],
"source": [
"delete_tb_experiments = False # @param {type: \"boolean\"}\n",
"\n",
"delete_method = \"\" # @param [\"by_label\", \"by_create_time\", \"by_update_time\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JjJVARe5V-MG"
},
"source": [
"### Delete an TensorBoard instance\n",
"\n",
"To delete a Tensorboard instance, you need `TENSORBOARD_INSTANCE` ID which uniquely identifies the Tensorboard instance where you run experiments.\n",
"\n",
"To get the Tensorboard instance ID, you can either\n",
"\n",
"- go to the cloud console UI, Vertex AI > Experiments > Tensorboard Instances, or\n",
"- use the list command below to list all TensorBoard instances for your project and region.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eqM2pIO8_4Y7"
},
"outputs": [],
"source": [
"tensorboard_instances = aiplatform.Tensorboard.list(project=PROJECT_ID, location=REGION)\n",
"print(tensorboard_instances)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wT_uPArACMPX"
},
"source": [
"Set the tensorboard instance id for which you want to delete experiments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hUync8E9CLcm"
},
"outputs": [],
"source": [
"if delete_tb_experiments:\n",
"\n",
" TENSORBOARD_INSTANCE_ID = \"\" # @param {type:\"string\"}\n",
"\n",
" TENSORBOARD_INSTANCE = aiplatform.Tensorboard(\n",
" project=PROJECT_ID, location=REGION, tensorboard_name=TENSORBOARD_INSTANCE_ID\n",
" )\n",
" print(TENSORBOARD_INSTANCE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OOp22fxdHqmT"
},
"source": [
"### Delete a TensorBoard Experiment with the key-value label pair\n",
"\n",
"You delete a TensorBoard experiment using a predefined `label_key` and `label_value`. For example, you may have assigned `delete` label key and `true` label value to indicate all Tensorboard experiments you want to delete.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N7KajCsjij1g"
},
"outputs": [],
"source": [
"LABEL_KEY = \"delete\" # @param {type:\"string\"}\n",
"LABEL_VALUE = \"true\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N45samDzWBO2"
},
"outputs": [],
"source": [
"if delete_tb_experiments and delete_method == \"by_label\":\n",
" clean_up_by_label(TENSORBOARD_INSTANCE, LABEL_KEY, LABEL_VALUE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XrEVqlgSIL6K"
},
"source": [
"### Delete a TensorBoard Experiment with `create_time`\n",
"\n",
"You delete a TensorBoard experiment using `create_time` field"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4_bjNeQ7p87A"
},
"outputs": [],
"source": [
"if delete_tb_experiments and delete_method == \"by_create_time\":\n",
" clean_up_by_create_time(TENSORBOARD_INSTANCE, CREATE_TIME_CUT)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZGNlSbIrIP_o"
},
"source": [
"### Delete a TensorBoard Experiment with `update_time`\n",
"\n",
"You delete a TensorBoard Experiment using a predefined `update_time` field"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SDyc_a8XwEve"
},
"outputs": [],
"source": [
"if delete_tb_experiments and delete_method == \"by_update_time\":\n",
" clean_up_by_update_time(TENSORBOARD_INSTANCE, UPDATE_TIME_CUT)"
]
}
],
"metadata": {
"colab": {
"name": "delete_outdated_tensorboard_experiments.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}