fix: correct the model GSC output path (#2102)

notebooks/official/tensorboard/tensorboard_custom_training_with_prebuilt_container.ipynb

@@ -24,6 +24,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "eBEO2w9My9py"
@@ -50,10 +51,11 @@
         "    </a>\n",
         "  </td>\n",
         "</table>\n",
-        "<br/><br/><br/>"
+        "<br/><br/><br/>\n"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "VL7XCFV7yCBU"
@@ -95,6 +97,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "UE8vLw7SlpwE"
@@ -118,6 +121,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "fbF2lF8rlp3I"
@@ -129,6 +133,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "yajq2millpnu"
@@ -145,39 +150,7 @@
       ]
     },
     {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "lAcDYZfslpeF"
-      },
-      "source": [
-        "### Set up your local development environment\n",
-        "\n",
-        "**If you are using Colab or Vertex AI Workbench**, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
-        "\n",
-        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
-        "\n",
-        "- The Cloud Storage SDK\n",
-        "- Git\n",
-        "- Python 3\n",
-        "- virtualenv\n",
-        "- Jupyter notebook running in a virtual environment with Python 3\n",
-        "\n",
-        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
-        "\n",
-        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
-        "\n",
-        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
-        "\n",
-        "3. [Install virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.  Activate the virtual environment.\n",
-        "\n",
-        "4. To install Jupyter, run `pip3 install jupyter` on the command-line in a terminal shell.\n",
-        "\n",
-        "5. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
-        "\n",
-        "6. Open this notebook in the Jupyter Notebook Dashboard."
-      ]
-    },
-    {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "s3moH5AexXpk"
@@ -196,343 +169,221 @@
       },
       "outputs": [],
       "source": [
-        "import os\n",
-        "\n",
-        "# The Vertex AI Workbench Notebook product has specific requirements\n",
-        "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
-        "\n",
-        "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
-        "USER_FLAG = \"\"\n",
-        "if IS_WORKBENCH_NOTEBOOK:\n",
-        "    USER_FLAG = \"--user\"\n",
-        "\n",
-        "! pip3 install google-cloud-aiplatform {USER_FLAG} -q\n",
-        "\n",
-        "# Automatically restart kernel after installs\n",
-        "\n",
-        "if not os.getenv(\"IS_TESTING\"):\n",
-        "    # Automatically restart kernel after installs\n",
-        "    import IPython\n",
-        "\n",
-        "    app = IPython.Application.instance()\n",
-        "    app.kernel.do_shutdown(True)"
+        "! pip3 install --upgrade --quiet google-cloud-aiplatform "
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "cczxMMkYK9a4"
+        "id": "restart"
       },
       "source": [
-        "## Before you begin"
+        "### Colab only: Uncomment the following cell to restart the kernel.\n"
       ]
     },
     {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bzPxhxS5lugp"
+      },
+      "outputs": [],
+      "source": [
+        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
+        "# import IPython\n",
+        "\n",
+        "# app = IPython.Application.instance()\n",
+        "# app.kernel.do_shutdown(True)"
+      ]
+    },
+    {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "64BZ1jL5GEi0"
+        "id": "d2qpIurSjmpT"
       },
       "source": [
+        "## Before you begin\n",
+        "\n",
         "### Set up your Google Cloud project\n",
         "\n",
         "**The following steps are required, regardless of your notebook environment.**\n",
         "\n",
         "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
         "\n",
-        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
+        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
         "\n",
-        "3. [Enable the Vertex AI APIs and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,storage-component.googleapis.com)\n",
+        "3. [Enable the following APIs: Vertex AI API, Cloud Resource Manager API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,cloudresourcemanager.googleapis.com).\n",
         "\n",
-        "4. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
-        "\n",
-        "\n",
-        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
-        "Cloud SDK uses the right project for all the commands in this notebook.\n",
-        "\n",
-        "Note: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "UIl_kn1pGH_T"
+        "id": "project_id"
       },
       "source": [
-        "### Set your project ID"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "Maw6BYbPA0kn"
-      },
-      "source": [
-        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`.\n"
+        "#### Set your project ID\n",
+        "\n",
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "sw4J6j5tBZWp"
+        "id": "wsePm9c4jmpT"
       },
       "outputs": [],
       "source": [
-        "PROJECT_ID = \"\"\n",
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
         "\n",
-        "import os\n",
-        "\n",
-        "# Get your Google Cloud project ID from gcloud\n",
-        "if not os.getenv(\"IS_TESTING\"):\n",
-        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
-        "    PROJECT_ID = shell_output[0]\n",
-        "    print(\"Project ID: \", PROJECT_ID)"
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "EJj3d9GxBd3b"
+        "id": "a54f9d7c1876"
       },
       "source": [
-        "Otherwise, set your project ID here."
+        "#### Region\n",
+        "\n",
+        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "75C2px4XtS5l"
+        "id": "3aaadaaf9b30"
       },
       "outputs": [],
       "source": [
-        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
-        "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
+        "REGION = \"us-central1\"  # @param {type: \"string\"}"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "Z36ycwA4IjYC"
+        "id": "5c0404984792"
       },
       "source": [
-        "### Timestamp"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "MngMavafIrQa"
-      },
-      "source": [
-        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "uddA3D7yIn-L"
-      },
-      "outputs": [],
-      "source": [
-        "from datetime import datetime\n",
+        "### Authenticate your Google Cloud account\n",
         "\n",
-        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "6yPUOueCF3pI"
-      },
-      "source": [
-        "### Set your region"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "gzEibeHGF1jb"
-      },
-      "outputs": [],
-      "source": [
-        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
         "\n",
-        "if REGION == \"[your-region]\":\n",
-        "    REGION = \"us-central1\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "jetPysDMtV1-"
-      },
-      "source": [
-        "### Login to your Google Cloud account"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "IK8eOR8tt6TI"
-      },
-      "outputs": [],
-      "source": [
-        "# The Google Cloud Notebook product has specific requirements\n",
-        "import os\n",
-        "import sys\n",
+        "**1. Vertex AI Workbench**\n",
+        "* Do nothing as you are already authenticated.\n",
         "\n",
-        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
+        "**2. Local JupyterLab instance, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Nt8cEM2GjmpU"
+      },
+      "outputs": [],
+      "source": [
+        "# ! gcloud auth login"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XUSL_JcpjmpU"
+      },
+      "source": [
+        "**3. Colab, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_2zemfGvjmpU"
+      },
+      "outputs": [],
+      "source": [
+        "# from google.colab import auth\n",
+        "# auth.authenticate_user()"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TCPJ38n7jmpU"
+      },
+      "source": [
+        "**4. Service account or other**\n",
+        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:custom"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
         "\n",
-        "IS_COLAB = \"google.colab\" in sys.modules\n",
-        "# If on Google Cloud Notebooks, then don't execute this code\n",
-        "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
-        "    if IS_COLAB:\n",
-        "        from google.colab import auth as google_auth\n",
+        "Create a storage bucket to store intermediate artifacts such as datasets.\n",
         "\n",
-        "        google_auth.authenticate_user()\n",
-        "\n",
-        "    # If you are running this notebook locally, replace the string below with the\n",
-        "    # path to your service account key and run this cell to authenticate your GCP\n",
-        "    # account.\n",
-        "    elif not os.getenv(\"IS_TESTING\"):\n",
-        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "tU1OEt8ibhLS"
-      },
-      "source": [
-        "###Create Cloud Storage bucket\n",
-        "A Cloud Storage bucket will be used to a) store your training code distribution (details below), and b) the outputs (including TensorBoard logs) your training code generates. The bucket must be regional that is, not multi-region or dual-region, and the following resources must be in same region:\n",
-        "\n",
-        "* the Cloud Storage bucket\n",
-        "* the Vertex AI training job\n",
-        "* the Vertex AI TensorBoard instance"
+        "When you submit a training job using the Cloud SDK, you upload a Python package\n",
+        "containing your training code to a Cloud Storage bucket. Vertex AI runs\n",
+        "the code from this package. In this tutorial, Vertex AI also saves the\n",
+        "trained model that results from your job in the same bucket. Using this model artifact, you can then\n",
+        "create Vertex AI Model resource and use for prediction."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "HYu2_qM9b3Cn"
+        "id": "bucket"
       },
       "outputs": [],
       "source": [
-        "BUCKET_URI = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}\n",
-        "\n",
-        "if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
-        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
+        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "qo1KHfJ2b83V"
+        "id": "create_bucket"
       },
       "source": [
-        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket. The created bucket will be deleted in the cleaning up section in the end. "
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "imVC-rXxb8A1"
+        "id": "Oz8J0vmSlugt"
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "1VU7ukLOcCa1"
-      },
-      "source": [
-        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "_wil_Y9lcHhZ"
-      },
-      "outputs": [],
-      "source": [
-        "! gsutil ls -al {BUCKET_URI}"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "lx8UKHsccMDe"
-      },
-      "source": [
-        "Set up the GCS paths for traing code and outputs."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "phyaIjuHcPdH"
-      },
-      "outputs": [],
-      "source": [
-        "GCS_BUCKET_TRAINING = BUCKET_URI + \"/training/\"\n",
-        "GCS_BUCKET_OUTPUT = BUCKET_URI + \"/output/\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "KIPcg_Xhwvsn"
-      },
-      "source": [
-        "### Import aiplatform"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "p4w8c1pHw2Yt"
-      },
-      "outputs": [],
-      "source": [
-        "import google.cloud.aiplatform as aiplatform"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "cVmlv9sRbCSs"
-      },
-      "source": [
-        "### Initialize Vertex AI SDK for Python\n",
-        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "JmEsq1fda_1N"
-      },
-      "outputs": [],
-      "source": [
-        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
+        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "hwXxa4Qgnh4Y"
@@ -542,6 +393,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "7qXFUiHLoFRw"
@@ -569,6 +421,9 @@
       },
       "outputs": [],
       "source": [
+        "import sys\n",
+        "\n",
+        "IS_COLAB = \"google.colab\" in sys.modules\n",
         "if (\n",
         "    SERVICE_ACCOUNT == \"\"\n",
         "    or SERVICE_ACCOUNT is None\n",
@@ -616,6 +471,50 @@
       ]
     },
     {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KIPcg_Xhwvsn"
+      },
+      "source": [
+        "### Import libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "p4w8c1pHw2Yt"
+      },
+      "outputs": [],
+      "source": [
+        "import google.cloud.aiplatform as aiplatform"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cVmlv9sRbCSs"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "JmEsq1fda_1N"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
+      ]
+    },
+    {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "dR2mOCllvlqN"
@@ -664,6 +563,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "vjdLIqYyDZFS"
@@ -790,6 +690,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "L067Jw_QFcZ3"
@@ -814,6 +715,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "a7Gt9WPFG9V7"
@@ -830,10 +732,12 @@
       },
       "outputs": [],
       "source": [
+        "GCS_BUCKET_TRAINING = f\"{BUCKET_URI}/data/\"\n",
         "! gsutil cp dist/hello-custom-training-3.0.tar.gz {GCS_BUCKET_TRAINING}"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "IaQjIPvuKLwW"
@@ -843,6 +747,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "svUGBOow_Obj"
@@ -866,7 +771,7 @@
         "    or TENSORBOARD_NAME is None\n",
         "    or TENSORBOARD_NAME == \"[your-tensorboard-name]\"\n",
         "):\n",
-        "    TENSORBOARD_NAME = PROJECT_ID + \"-tb-\" + TIMESTAMP\n",
+        "    TENSORBOARD_NAME = PROJECT_ID + \"-tb\"\n",
         "\n",
         "tensorboard = aiplatform.Tensorboard.create(\n",
         "    display_name=TENSORBOARD_NAME, project=PROJECT_ID, location=REGION\n",
@@ -876,6 +781,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "mudxBDal_a_k"
@@ -892,8 +798,9 @@
       },
       "outputs": [],
       "source": [
-        "JOB_NAME = \"tensorboard-example-job-{}\".format(TIMESTAMP)\n",
-        "BASE_OUTPUT_DIR = \"{}{}\".format(GCS_BUCKET_OUTPUT, JOB_NAME)\n",
+        "JOB_NAME = \"tensorboard-example-job\"\n",
+        "GCS_BUCKET_OUTPUT = BUCKET_URI\n",
+        "BASE_OUTPUT_DIR = \"{}/{}\".format(GCS_BUCKET_OUTPUT, JOB_NAME)\n",
         "\n",
         "job = aiplatform.CustomPythonPackageTrainingJob(\n",
         "    display_name=JOB_NAME,\n",
@@ -914,6 +821,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "VfMsn_RnEtnj"
@@ -923,6 +831,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "TFEriiywMZga"