mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
refactor,chore(egen) : refactored code as per template guidelines , performed linter test (#3220)
* refactor,chore(egen) : refactored code as per template guidelines and performed linter test * refcator(egen) : refcatored code according to template guidelines * refcator(egen) : refcatored code according to template guidelines * refcator(egen) : refcatored code according to template guidelines * refactor(egen) : refactored code according to template guidelines --------- Co-authored-by: Jayakrishna2801 <jayakrishna.rajaboina@egen.ai>
This commit is contained in:
Jayakrishna2801
co-authored by
Jayakrishna2801
Jayakrishna2801
parent
ed721006b7
commit
8f9c783f97
+83
-149
@@ -32,25 +32,28 @@
|
||||
"# Vertex AI: Track parameters and metrics for custom training jobs\n",
|
||||
"\n",
|
||||
"<table align=\"left\">\n",
|
||||
"\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/lightweight_functions_component_io_kfp.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
|
||||
" View on GitHub\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fnotebook_template.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-custom-jobs.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
" Open in Vertex AI Workbench\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/lightweight_functions_component_io_kfp.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td> \n",
|
||||
"</table>"
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/pipelines/lightweight_functions_component_io_kfp.ipynb\" target='_blank'>\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</table>\n",
|
||||
"<br/><br/><br/>"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -63,7 +66,9 @@
|
||||
"\n",
|
||||
"This notebook demonstrates how to track metrics and parameters for Vertex AI custom training jobs, and how to perform detailed analysis using this data.\n",
|
||||
"\n",
|
||||
"Learn more about [Vertex ML Metadata](https://cloud.google.com/vertex-ai/docs/ml-metadata), [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training), and [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments)."
|
||||
"Learn more about [Vertex ML Metadata](https://cloud.google.com/vertex-ai/docs/ml-metadata),\n",
|
||||
"[Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training), and \n",
|
||||
"[Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -113,11 +118,9 @@
|
||||
"* Vertex AI\n",
|
||||
"* Cloud Storage\n",
|
||||
"\n",
|
||||
"Learn about [Vertex AI\n",
|
||||
"pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
|
||||
"pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
|
||||
"Calculator](https://cloud.google.com/products/calculator/)\n",
|
||||
"to generate a cost estimate based on your projected usage."
|
||||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and \n",
|
||||
"[Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the \n",
|
||||
"[Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -126,9 +129,8 @@
|
||||
"id": "i7EUnXsZhAGF"
|
||||
},
|
||||
"source": [
|
||||
"### Installation\n",
|
||||
"\n",
|
||||
"Install the packages required for executing this notebook."
|
||||
"### Get Started\n",
|
||||
"Install Vertex AI SDK for Python and other required packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -141,54 +143,84 @@
|
||||
"source": [
|
||||
"! pip3 install --upgrade tensorflow \\\n",
|
||||
" google-cloud-aiplatform \\\n",
|
||||
" scikit-learn -q"
|
||||
" scikit-learn -q \\\n",
|
||||
" pandas"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "hhq5zEbGg0XX"
|
||||
"id": "5eec42e37bcf"
|
||||
},
|
||||
"source": [
|
||||
"### Colab only: Uncomment the following cell to restart the kernel."
|
||||
"### Restart runtime (Colab only)\n",
|
||||
"To use the newly installed packages, you must restart the runtime on Google Colab."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "EzrelQZ22IZj"
|
||||
"id": "dcc98768955f"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
|
||||
"# import IPython\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# app = IPython.Application.instance()\n",
|
||||
"# app.kernel.do_shutdown(True)"
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" import IPython\n",
|
||||
"\n",
|
||||
" app = IPython.Application.instance()\n",
|
||||
" app.kernel.do_shutdown(True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "lWEdiXsJg0XY"
|
||||
"id": "4de1bd77992b"
|
||||
},
|
||||
"source": [
|
||||
"## Before you begin"
|
||||
"<div class=\"alert alert-block alert-warning\">,\n",
|
||||
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>,\n",
|
||||
"</div>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8bc8a29f9001"
|
||||
"id": "56e219dbcb9a"
|
||||
},
|
||||
"source": [
|
||||
"#### Set your project ID\n",
|
||||
"### Authenticate your notebook environment (Colab only)\n",
|
||||
"Authenticate your environment on Google Colab."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "c97be6a73155"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, try the following:\n",
|
||||
"* Run `gcloud config list`.\n",
|
||||
"* Run `gcloud projects list`.\n",
|
||||
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
"\n",
|
||||
" from google.colab import auth\n",
|
||||
"\n",
|
||||
" auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "442da99b7efa"
|
||||
},
|
||||
"source": [
|
||||
"### Set Google Cloud project information\n",
|
||||
"Learn more about [setting up a project and a development environment.](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -200,103 +232,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||
"\n",
|
||||
"# Set the project id\n",
|
||||
"! gcloud config set project {PROJECT_ID}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "47bc07d4231b"
|
||||
},
|
||||
"source": [
|
||||
"#### Region\n",
|
||||
"\n",
|
||||
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "959545da671a"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"REGION = \"us-central1\" # @param {type: \"string\"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "dr--iN2kAylZ"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "sBCra4QMA2wR"
|
||||
},
|
||||
"source": [
|
||||
"**1. Vertex AI Workbench**\n",
|
||||
"* Do nothing as you are already authenticated."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ad1138a125ea"
|
||||
},
|
||||
"source": [
|
||||
"**2. Local JupyterLab instance, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "ce6043da7b33"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ! gcloud auth login"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "0367eac06a10"
|
||||
},
|
||||
"source": [
|
||||
"**3. Colab, uncomment and run:**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "PyQmSRbKA8r-"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# from google.colab import auth\n",
|
||||
"# auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "c13224697bfb"
|
||||
},
|
||||
"source": [
|
||||
"**4. Service account or other**\n",
|
||||
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
|
||||
"LOCATION = \"us-central1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -327,7 +263,7 @@
|
||||
"id": "-EcIXiGsCePi"
|
||||
},
|
||||
"source": [
|
||||
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -338,7 +274,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -367,8 +303,6 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"from google.cloud import aiplatform\n",
|
||||
"from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
|
||||
@@ -381,7 +315,7 @@
|
||||
"id": "O8XJZB3gR8eL"
|
||||
},
|
||||
"source": [
|
||||
"## Initialize Vertex AI and set an _experiment_\n"
|
||||
"## Initialize Vertex AI and set an experiment\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -423,7 +357,7 @@
|
||||
"source": [
|
||||
"aiplatform.init(\n",
|
||||
" project=PROJECT_ID,\n",
|
||||
" location=REGION,\n",
|
||||
" location=LOCATION,\n",
|
||||
" staging_bucket=BUCKET_URI,\n",
|
||||
" experiment=EXPERIMENT_NAME,\n",
|
||||
")"
|
||||
@@ -435,7 +369,7 @@
|
||||
"id": "6PlilQPFeS_h"
|
||||
},
|
||||
"source": [
|
||||
"## Tracking parameters and metrics in Vertex AI custom training jobs"
|
||||
"### Tracking parameters and metrics in Vertex AI custom training jobs"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -444,7 +378,7 @@
|
||||
"id": "f8fd397cc4f6"
|
||||
},
|
||||
"source": [
|
||||
"### Download the Dataset to Cloud Storage"
|
||||
"# Download the Dataset to Cloud Storage"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -580,7 +514,7 @@
|
||||
"id": "k_QorXXztzPH"
|
||||
},
|
||||
"source": [
|
||||
"Start a new experiment run to track training parameters and start the training job. Note that this operation will take around 10 mins."
|
||||
"Start a new experiment run to track training parameters and start the training job. Note that this operation takes around 10 minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -620,7 +554,7 @@
|
||||
"id": "O-uCOL3Naap4"
|
||||
},
|
||||
"source": [
|
||||
"Next, deploy your Vertex AI Model resource to a Vertex AI Endpoint resource. This operation will take 10-20 mins."
|
||||
"Next, deploy your Vertex AI Model resource to a Vertex AI endpoint resource. This operation takes 10-20 minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -824,7 +758,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Warning: Setting this to true will delete everything in your bucket\n",
|
||||
"# Warning: Setting this to true deletes everything in your bucket\n",
|
||||
"delete_bucket = False\n",
|
||||
"\n",
|
||||
"# Delete dataset\n",
|
||||
@@ -832,7 +766,7 @@
|
||||
"\n",
|
||||
"# Delete experiment\n",
|
||||
"experiment = aiplatform.Experiment(\n",
|
||||
" experiment_name=EXPERIMENT_NAME, project=PROJECT_ID, location=REGION\n",
|
||||
" experiment_name=EXPERIMENT_NAME, project=PROJECT_ID, location=LOCATION\n",
|
||||
")\n",
|
||||
"experiment.delete()\n",
|
||||
"\n",
|
||||
@@ -849,7 +783,7 @@
|
||||
"model.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
|
||||
"if delete_bucket:\n",
|
||||
" ! gsutil -m rm -r $BUCKET_URI"
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user