Merge branch 'main' into reduction_38

notebooks/official/model_evaluation/automl_text_classification_model_evaluation.ipynb

@@ -128,39 +128,6 @@
         "to generate a cost estimate based on your projected usage."
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "setup_local"
-      },
-      "source": [
-        "### Set up your local development environment\n",
-        "\n",
-        "If you are using Colab or Google Vertex AI Workbench Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
-        "\n",
-        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
-        "\n",
-        "- The Cloud Storage SDK\n",
-        "- Git\n",
-        "- Python 3\n",
-        "- virtualenv\n",
-        "- Jupyter notebook running in a virtual environment with Python 3\n",
-        "\n",
-        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
-        "\n",
-        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
-        "\n",
-        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
-        "\n",
-        "3. [Install virtualenv](Ihttps://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.\n",
-        "\n",
-        "4. Activate that environment and run `pip3 install Jupyter` in a terminal shell to install Jupyter.\n",
-        "\n",
-        "5. Run `jupyter notebook` on the command line in a terminal shell to launch Jupyter.\n",
-        "\n",
-        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -180,23 +147,10 @@
       },
       "outputs": [],
       "source": [
-        "import os\n",
-        "\n",
-        "# The Vertex AI Workbench Notebook product has specific requirements\n",
-        "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
-        "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
-        "    \"/opt/deeplearning/metadata/env_version\"\n",
-        ")\n",
-        "\n",
-        "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
-        "USER_FLAG = \"\"\n",
-        "if IS_WORKBENCH_NOTEBOOK:\n",
-        "    USER_FLAG = \"--user\"\n",
-        "\n",
         "! pip3 install --upgrade google-cloud-aiplatform \\\n",
         "                         google-cloud-storage \\\n",
-        "                         kfp google-cloud-pipeline-components==1.0.40 \\\n",
-        "                         ndjson {USER_FLAG} -q"
+        "                         kfp google-cloud-pipeline-components==1.0.25 \\\n",
+        "                         ndjson --quiet"
       ]
     },
     {
@@ -205,48 +159,22 @@
         "id": "restart"
       },
       "source": [
-        "### Restart the kernel\n",
-        "\n",
-        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
+        "### Colab only: Uncomment the following cell to restart the kernel"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "restart"
+        "id": "D-ZBOjErv5mM"
       },
       "outputs": [],
       "source": [
-        "import os\n",
+        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
+        "# import IPython\n",
         "\n",
-        "if not os.getenv(\"IS_TESTING\"):\n",
-        "    # Automatically restart kernel after installs\n",
-        "    import IPython\n",
-        "\n",
-        "    app = IPython.Application.instance()\n",
-        "    app.kernel.do_shutdown(True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "check_versions"
-      },
-      "source": [
-        "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "check_versions:kfp,gcpc"
-      },
-      "outputs": [],
-      "source": [
-        "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
-        "! python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
+        "# app = IPython.Application.instance()\n",
+        "# app.kernel.do_shutdown(True)"
       ]
     },
     {
@@ -257,26 +185,12 @@
       "source": [
         "## Before you begin\n",
         "\n",
-        "### GPU runtime\n",
+        "### Set your project ID\n",
         "\n",
-        "This tutorial does not require a GPU runtime.\n",
-        "\n",
-        "### Set up your Google Cloud project\n",
-        "\n",
-        "**The following steps are required, regardless of your notebook environment.**\n",
-        "\n",
-        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
-        "\n",
-        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
-        "\n",
-        "3. [Enable the Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
-        "\n",
-        "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Vertex AI Workbench Notebook.\n",
-        "\n",
-        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
-        "Cloud SDK uses the right project for all the commands in this notebook.\n",
-        "\n",
-        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
       ]
     },
     {
@@ -287,33 +201,10 @@
       },
       "outputs": [],
       "source": [
-        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "autoset_project_id"
-      },
-      "outputs": [],
-      "source": [
-        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
-        "    # Get your GCP project id from gcloud\n",
-        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
-        "    PROJECT_ID = shell_output[0]\n",
-        "    print(\"Project ID:\", PROJECT_ID)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "set_gcloud_project_id"
-      },
-      "outputs": [],
-      "source": [
-        "! gcloud config set project $PROJECT_ID"
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "\n",
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
       ]
     },
     {
@@ -324,61 +215,19 @@
       "source": [
         "#### Region\n",
         "\n",
-        "You can also change the `REGION` variable, which is used for operations\n",
-        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
-        "\n",
-        "- Americas: `us-central1`\n",
-        "- Europe: `europe-west4`\n",
-        "- Asia Pacific: `asia-east1`\n",
-        "\n",
-        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
-        "\n",
-        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
+        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "region"
+        "id": "2dw8q9fdQEH5"
       },
       "outputs": [],
       "source": [
-        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
-        "\n",
-        "if REGION == \"[your-region]\":\n",
-        "    REGION = \"us-central1\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "timestamp"
-      },
-      "source": [
-        "#### UUID\n",
-        "\n",
-        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append the uuid onto the name of resources you create in this tutorial."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "timestamp"
-      },
-      "outputs": [],
-      "source": [
-        "import random\n",
-        "import string\n",
-        "\n",
-        "\n",
-        "# Generate a uuid of a specifed length(default=8)\n",
-        "def generate_uuid(length: int = 8) -> str:\n",
-        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
-        "\n",
-        "\n",
-        "UUID = generate_uuid()"
+        "REGION = \"us-central1\"\n",
+        "DATA_REGION = \"US\""
       ]
     },
     {
@@ -389,63 +238,54 @@
       "source": [
         "### Authenticate your Google Cloud account\n",
         "\n",
-        "**If you are using Vertex AI Workbench Notebook**, your environment is already authenticated.\n",
+        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
         "\n",
-        "**If you are using Colab**, run the cell below and follow the instructions\n",
-        "when prompted to authenticate your account via oAuth.\n",
+        "**1. Vertex AI Workbench**\n",
+        "* Do nothing as you are already authenticated.\n",
         "\n",
-        "**Otherwise**, follow these steps:\n",
-        "\n",
-        "1. In the Cloud Console, go to the [**Create service account key**\n",
-        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
-        "\n",
-        "2. Click **Create service account**.\n",
-        "\n",
-        "3. In the **Service account name** field, enter a name, and\n",
-        "   click **Create**.\n",
-        "\n",
-        "4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
-        "into the filter box, and select\n",
-        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
-        "\n",
-        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
-        "local environment.\n",
-        "\n",
-        "6. Enter the path to your service account key as the\n",
-        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
+        "**2. Local JupyterLab instance, uncomment and run:**"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "gcp_authenticate"
+        "id": "ce6043da7b33"
       },
       "outputs": [],
       "source": [
-        "# If you are running this notebook in Colab, run this cell and follow the\n",
-        "# instructions to authenticate your GCP account. This provides access to your\n",
-        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
-        "# requests.\n",
-        "\n",
-        "import os\n",
-        "import sys\n",
-        "\n",
-        "# If on Vertex AI Workbench, then don't execute this code\n",
-        "IS_COLAB = \"google.colab\" in sys.modules\n",
-        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
-        "    \"DL_ANACONDA_HOME\"\n",
-        "):\n",
-        "    if \"google.colab\" in sys.modules:\n",
-        "        from google.colab import auth as google_auth\n",
-        "\n",
-        "        google_auth.authenticate_user()\n",
-        "\n",
-        "    # If you are running this notebook locally, replace the string below with the\n",
-        "    # path to your service account key and run this cell to authenticate your GCP\n",
-        "    # account.\n",
-        "    elif not os.getenv(\"IS_TESTING\"):\n",
-        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
+        "# ! gcloud auth login"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0367eac06a10"
+      },
+      "source": [
+        "**3. Colab, uncomment and run:**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "21ad4dbb4a61"
+      },
+      "outputs": [],
+      "source": [
+        "# from google.colab import auth\n",
+        "# auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "c13224697bfb"
+      },
+      "source": [
+        "**4. Service account or other**\n",
+        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
       ]
     },
     {
@@ -456,11 +296,7 @@
       "source": [
         "### Create a Cloud Storage bucket\n",
         "\n",
-        "**The following steps are required, regardless of your notebook environment.**\n",
-        "\n",
-        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
-        "\n",
-        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+        "Create a storage bucket to store intermediate artifacts such as datasets."
       ]
     },
     {
@@ -471,27 +307,13 @@
       },
       "outputs": [],
       "source": [
-        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
-        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "autoset_bucket"
-      },
-      "outputs": [],
-      "source": [
-        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
-        "    BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
-        "    BUCKET_URI = \"gs://\" + BUCKET_NAME"
+        "BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\"  # @param {type:\"string\"}"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "create_bucket"
+        "id": "autoset_bucket"
       },
       "source": [
         "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
@@ -501,31 +323,11 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "create_bucket"
+        "id": "91c46850b49b"
       },
       "outputs": [],
       "source": [
-        "! gsutil mb -l $REGION $BUCKET_URI"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "validate_bucket"
-      },
-      "source": [
-        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "validate_bucket"
-      },
-      "outputs": [],
-      "source": [
-        "! gsutil ls -al $BUCKET_URI"
+        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
       ]
     },
     {
@@ -693,7 +495,7 @@
         "IMPORT_FILE = \"gs://cloud-ml-data/NL-classification/happiness.csv\"\n",
         "\n",
         "\n",
-        "@kfp.dsl.pipeline(name=\"automl-text-classification\" + UUID)\n",
+        "@kfp.dsl.pipeline(name=\"automl-text-classification\")\n",
         "def pipeline(\n",
         "    project: str = PROJECT_ID, region: str = REGION, import_file: str = IMPORT_FILE\n",
         "):\n",
@@ -780,7 +582,7 @@
       },
       "outputs": [],
       "source": [
-        "DISPLAY_NAME = \"happydb_\" + UUID\n",
+        "DISPLAY_NAME = \"happydb\"\n",
         "\n",
         "job = aip.PipelineJob(\n",
         "    display_name=DISPLAY_NAME,\n",
@@ -1026,7 +828,7 @@
         "from google.cloud import storage\n",
         "\n",
         "storage_client = storage.Client()\n",
-        "bucket = storage_client.bucket(BUCKET_NAME)\n",
+        "bucket = storage_client.bucket(BUCKET_URI[5:])\n",
         "# Iterate over the prediction instances, creating a new TXT file\n",
         "# for each.\n",
         "input_file_data = []\n",
@@ -1151,7 +953,7 @@
         "from google.cloud import storage\n",
         "\n",
         "storage_client = storage.Client()\n",
-        "bucket = storage_client.bucket(BUCKET_NAME)\n",
+        "bucket = storage_client.bucket(BUCKET_URI[5:])\n",
         "# Iterate over the prediction instances, creating a new TXT file\n",
         "# for each.\n",
         "input_file_data = []\n",
@@ -1378,7 +1180,7 @@
         "    or PIPELINE_DISPLAY_NAME == \"\"\n",
         "    or PIPELINE_DISPLAY_NAME is None\n",
         "):\n",
-        "    PIPELINE_DISPLAY_NAME = \"happiness_\" + UUID"
+        "    PIPELINE_DISPLAY_NAME = \"happiness\""
       ]
     },
     {
@@ -1410,7 +1212,7 @@
       "outputs": [],
       "source": [
         "DATA_SOURCE = f\"{BUCKET_URI}/{input_file_name}\"\n",
-        "PIPELINE_ROOT = f\"{BUCKET_URI}/pipeline_root/happiness_{UUID}\"\n",
+        "PIPELINE_ROOT = \"{BUCKET_URI}/pipeline_root/happiness\"\n",
         "parameters = {\n",
         "    \"project\": PROJECT_ID,\n",
         "    \"location\": REGION,\n",
@@ -1629,12 +1431,6 @@
       },
       "outputs": [],
       "source": [
-        "delete_dataset = True\n",
-        "delete_training_pipeline = True\n",
-        "delete_batchpredict_job = True\n",
-        "delete_evaluation_pipeline = True\n",
-        "delete_model = True\n",
-        "delete_endpoint = True\n",
         "delete_bucket = False\n",
         "\n",
         "dataset_type = \"text\"\n",
@@ -1643,71 +1439,27 @@
         "endpoint_display_name = \"train-automl-happydb\"\n",
         "\n",
         "\n",
-        "if delete_endpoint:\n",
-        "    endpoints = aip.Endpoint.list(\n",
-        "        filter=f\"display_name={endpoint_display_name}\", order_by=\"create_time\"\n",
-        "    )\n",
-        "    if endpoints:\n",
-        "        endpoint = endpoints[0]\n",
-        "        endpoint.undeploy_all()\n",
-        "        endpoint.delete()\n",
-        "        print(\"Deleted endpoint:\", endpoint)\n",
+        "endpoints = aip.Endpoint.list(\n",
+        "    filter=f\"display_name={endpoint_display_name}\", order_by=\"create_time\"\n",
+        ")\n",
+        "if endpoints:\n",
+        "    endpoint = endpoints[0]\n",
+        "    endpoint.undeploy_all()\n",
+        "    endpoint.delete()\n",
+        "    print(\"Deleted endpoint:\", endpoint)\n",
         "\n",
-        "if delete_model:\n",
-        "    models = aip.Model.list(\n",
-        "        filter=f\"display_name={model_display_name}\", order_by=\"create_time\"\n",
-        "    )\n",
-        "    if models:\n",
-        "        model = models[0]\n",
-        "        model.delete()\n",
-        "        print(\"Deleted model:\", model)\n",
+        "models = aip.Model.list(\n",
+        "    filter=f\"display_name={model_display_name}\", order_by=\"create_time\"\n",
+        ")\n",
+        "if models:\n",
+        "    model = models[0]\n",
+        "    model.delete()\n",
+        "    print(\"Deleted model:\", model)\n",
         "\n",
-        "if delete_dataset:\n",
-        "    if dataset_type == \"tabular\":\n",
-        "        datasets = aip.TabularDataset.list(\n",
-        "            filter=f\"display_name={dataset_display_name}\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        if datasets:\n",
-        "            dataset = datasets[0]\n",
-        "            dataset.delete()\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
         "\n",
-        "    if dataset_type == \"image\":\n",
-        "        datasets = aip.ImageDataset.list(\n",
-        "            filter=f\"display_name={dataset_display_name}\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        if datasets:\n",
-        "            dataset = datasets[0]\n",
-        "            dataset.delete()\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "\n",
-        "    if dataset_type == \"text\":\n",
-        "        datasets = aip.TextDataset.list(\n",
-        "            filter=f\"display_name={dataset_display_name}\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        if datasets:\n",
-        "            dataset = datasets[0]\n",
-        "            dataset.delete()\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "\n",
-        "    if dataset_type == \"video\":\n",
-        "        datasets = aip.VideoDataset.list(\n",
-        "            filter=f\"display_name={dataset_display_name}\", order_by=\"create_time\"\n",
-        "        )\n",
-        "        if datasets:\n",
-        "            dataset = datasets[0]\n",
-        "            dataset.delete()\n",
-        "            print(\"Deleted dataset:\", dataset)\n",
-        "\n",
-        "if delete_training_pipeline:\n",
-        "    job.delete()\n",
-        "\n",
-        "if delete_batchpredict_job:\n",
-        "    batch_prediction_job.delete()\n",
-        "\n",
-        "if delete_evaluation_pipeline:\n",
-        "    evaluation_job.delete()\n",
+        "evaluation_job.delete()\n",
         "\n",
+        "batch_prediction_job.delete()\n",
         "\n",
         "if delete_bucket and os.getenv(\"IS_TESTING\"):\n",
         "    ! gsutil rm -r $BUCKET_URI"