Compare commits

...
@@ -133,50 +133,12 @@
{
"cell_type": "markdown",
"metadata": {
"id": "setup_local"
},
"source": [
"### Set up your local development environment\n",
"\n",
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
"all the requirements to run this notebook. You can skip this step.\n",
"\n",
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
"You need the following:\n",
"\n",
"* The Google Cloud SDK\n",
"\n",
"The Google Cloud guide to [Setting up a Python development\n",
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
"for meeting these requirements. The following steps provide a condensed set of\n",
"instructions:\n",
"\n",
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
"\n",
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
"\n",
"1. [Install\n",
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"\n",
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
"command-line in a terminal shell.\n",
"\n",
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"\n",
"1. Open this notebook in the Jupyter Notebook Dashboard.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "install_aip:mbsdk"
"id": "iMHz63rPbq6P"
},
"source": [
"## Installation\n",
"\n",
"Install the latest version of Vertex AI SDK and other required Python packages."
"Install the packages required for executing this notebook."
]
},
{
@@ -187,131 +149,67 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform \\\n",
" google-cloud-storage {USER_FLAG} -q\n",
" google-cloud-storage -q\n",
"\n",
"import os\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! apt-get update && apt-get install -y python3-opencv-headless\n",
" ! apt-get install -y libgl1-mesa-dev\n",
" ! pip3 install --upgrade opencv-python-headless {USER_FLAG} -q\n",
" ! pip3 install tensorflow==2.9 {USER_FLAG} -q"
" ! pip3 install --upgrade opencv-python-headless -q\n",
" ! pip3 install tensorflow==2.9 -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "restart"
"id": "58707a750154"
},
"source": [
"### Restart the kernel\n",
"\n",
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
"### Colab only: Uncomment the following cell to restart the kernel."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "restart"
"id": "f200f10a1da3"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "before_you_begin:nogpu"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the following APIs: Vertex AI, Compute Engine, Cloud Storage, Cloud Build and Artifact Registry APIs.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute.googleapis.com,storage-component.googleapis.com,cloudbuild.googleapis.com,artifactregistry.googleapis.com)\n",
"\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
"\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5aee4379e8e5"
"id": "WReHDGG5g0XY"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "set_project_id"
"id": "oM1iC_MfAts1"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "autoset_project_id"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "set_gcloud_project_id"
},
"outputs": [],
"source": [
"! gcloud config set project $PROJECT_ID"
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
@@ -320,32 +218,20 @@
"id": "region"
},
"source": [
"#### Region\n",
"#### Set the region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
"- Asia Pacific: `asia-east1`\n",
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
"**Optional**: Update the 'REGION' variable to specify the region that you want to use. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "region"
"id": "nsN5NJKSu-GU"
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
"\n",
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
@@ -382,110 +268,84 @@
{
"cell_type": "markdown",
"metadata": {
"id": "gcp_authenticate"
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
"authenticated. \n",
"To authenticate your Google Cloud account, follow the instructions for your Jupyter environment:\n",
"\n",
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"**1. Vertex AI Workbench**\n",
"<br>You are already authenticated.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"2. Click **Create service account**.\n",
"\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
"**2. Local JupyterLab instance**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gcp_authenticate"
"id": "254614fa0c46"
},
"outputs": [],
"source": [
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"import os\n",
"import sys\n",
"\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS '[your-service-account-key-path]'"
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bucket:mbsdk"
"id": "ef21552ccea8"
},
"source": [
"**3. Colab**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "603adbbf0532"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zgPO1eR3CYjk"
},
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"\n",
"Set the name of your Cloud Storage bucket below. It must be unique across all\n",
"Cloud Storage buckets."
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bucket"
"id": "MzGDU7TWdts_"
},
"outputs": [],
"source": [
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "autoset_bucket"
},
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
" BUCKET_URI = f\"gs://{BUCKET_NAME}\""
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -501,33 +361,13 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "create_bucket"
"id": "NIq7R4HZCfIc"
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "validate_bucket"
},
"source": [
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "validate_bucket"
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -596,14 +436,12 @@
"\n",
"Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
"\n",
" (aiplatform.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
" (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
"\n",
"\n",
"Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
"\n",
"Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region\n",
"\n",
"*Note: TF releases before 2.3 for GPU support fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3 and above., which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 or above with GPU support.*"
"Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region"
]
},
{
@@ -614,21 +452,8 @@
},
"outputs": [],
"source": [
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
" TRAIN_GPU, TRAIN_NGPU = (\n",
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
" )\n",
"else:\n",
" TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
"\n",
"if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
" DEPLOY_GPU, DEPLOY_NGPU = (\n",
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n",
" )\n",
"else:\n",
" DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
"TRAIN_GPU, TRAIN_NGPU = (None, None)\n",
"DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
]
},
{
@@ -654,10 +479,7 @@
},
"outputs": [],
"source": [
"if os.getenv(\"IS_TESTING_TF\"):\n",
" TF = os.getenv(\"IS_TESTING_TF\")\n",
"else:\n",
" TF = \"2-9\"\n",
"TF = \"2-9\"\n",
"\n",
"if TF[0] == \"2\":\n",
" if DEPLOY_GPU:\n",
@@ -708,10 +530,7 @@
},
"outputs": [],
"source": [
"if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n",
" MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n",
"else:\n",
" MACHINE_TYPE = \"n1-standard\"\n",
"MACHINE_TYPE = \"n1-standard\"\n",
"\n",
"VCPU = \"4\"\n",
"TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n",
@@ -959,6 +778,64 @@
"ENTRYPOINT [\"python\", \"-m\", \"trainer.task\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gar_enable_api"
},
"source": [
"### Enable Artifact Registry API\n",
"\n",
"You must enable the Artifact Registry API service for your project.\n",
"\n",
"Learn more about [Enabling service](https://cloud.google.com/artifact-registry/docs/enable-service)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gar_enable_api"
},
"outputs": [],
"source": [
"! gcloud services enable artifactregistry.googleapis.com\n",
"\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! sudo apt-get update --yes && sudo apt-get --only-upgrade --yes install google-cloud-sdk-cloud-run-proxy google-cloud-sdk-harbourbridge google-cloud-sdk-cbt google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-kpt google-cloud-sdk-local-extract google-cloud-sdk-minikube google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-python google-cloud-sdk-spanner-emulator google-cloud-sdk-bigtable-emulator google-cloud-sdk-nomos google-cloud-sdk-package-go-module google-cloud-sdk-firestore-emulator kubectl google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-kubectl-oidc google-cloud-sdk-anthos-auth google-cloud-sdk-app-engine-grpc google-cloud-sdk-pubsub-emulator google-cloud-sdk-datalab google-cloud-sdk-skaffold google-cloud-sdk google-cloud-sdk-terraform-tools google-cloud-sdk-config-connector\n",
" ! gcloud components update --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gar_create_repo"
},
"source": [
"## Create a private Docker repository\n",
"\n",
"Your first step is to create your own Docker repository in Google Artifact Registry.\n",
"\n",
"1. Run the `gcloud artifacts repositories create` command to create a new Docker repository with your region with the description \"docker repository\".\n",
"\n",
"2. Run the `gcloud artifacts repositories list` command to verify that your repository was created."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gar_create_repo"
},
"outputs": [],
"source": [
"REPOSITORY = \"my-docker-repo\"\n",
"\n",
"! gcloud artifacts repositories create {REPOSITORY} --repository-format=docker --location={REGION} --description=\"Docker repository\"\n",
"\n",
"! gcloud artifacts repositories list"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -982,18 +859,7 @@
},
"outputs": [],
"source": [
"REPOSITORY = \"[your-repository-name]\"\n",
"CONTAINER_NAME = \"[your-container-name]\"\n",
"\n",
"if REPOSITORY == \"[your-repository-name]\" or REPOSITORY == \"\" or REPOSITORY is None:\n",
" REPOSITORY = f\"custom-img-classify-train-{UUID}\"\n",
"\n",
"if (\n",
" CONTAINER_NAME == \"[your-container-name]\"\n",
" or CONTAINER_NAME == \"\"\n",
" or CONTAINER_NAME is None\n",
"):\n",
" CONTAINER_NAME = \"cifar10\"\n",
"CONTAINER_NAME = \"cifar10\"\n",
"\n",
"TAG = \"latest\"\n",
"TRAIN_IMAGE = (\n",
@@ -1010,38 +876,6 @@
"Create the repository in Artifact Registry."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "da95b138f66f"
},
"outputs": [],
"source": [
"# Create the repository in Artifact registry\n",
"! gcloud artifacts repositories create {REPOSITORY} --repository-format=docker --location={REGION} --description=\"Docker repository for custom training\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "15efd26f760a"
},
"source": [
"List all repositories and check your repository."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "build_container:training"
},
"outputs": [],
"source": [
"! gcloud artifacts repositories list"
]
},
{
"cell_type": "markdown",
"metadata": {