xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

update: boiler plate reduction 8 (#1829)

Browse Source 
* update: boiler plate reduction 8

* fix: rm TIMESTAMP

* fix: import
This commit is contained in:
Andrew Ferlitsch
2023-05-08 18:58:38 +00:00
committed by GitHub
parent cec6321027
commit 44e0c77ddb
1 changed files with 77 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/automl/get_started_automl_training.ipynb
+77 -212
View File
@@ -168,21 +168,10 @@
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"\n",
"# Install the packages\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG -q\n",
"! pip3 install --upgrade google-cloud-storage $USER_FLAG -q"
"! pip3 install --upgrade --quiet google-cloud-aiplatform \\\n",
" google-cloud-storage "
]
},
{
@@ -191,64 +180,38 @@
"id": "restart"
},
"source": [
"### Restart the kernel\n",
"\n",
"Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
"### Colab only: Uncomment the following cell to restart the kernel"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "restart"
"id": "D-ZBOjErv5mM"
},
"outputs": [],
"source": [
"import os\n",
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "project_id"
"id": "yfEglUHQk9S3"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"### Set your project ID\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI, Compute Engine and Cloud Storage APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage_component).\n",
"\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
"\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5aee4379e8e5"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
@@ -259,33 +222,10 @@
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "autoset_project_id"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "set_gcloud_project_id"
},
"outputs": [],
"source": [
"! gcloud config set project $PROJECT_ID"
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
@@ -296,16 +236,7 @@
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
"- Asia Pacific: `asia-east1`\n",
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
@@ -316,95 +247,65 @@
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
"\n",
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "timestamp"
},
"source": [
"#### Timestamp\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "timestamp"
},
"outputs": [],
"source": [
"from datetime import datetime\n",
"\n",
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3ffa6b6c7cdb"
"id": "gcp_authenticate"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already authenticated. Skip this step.\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
"\n",
"1. **Click Create service account**.\n",
"\n",
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
"\n",
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex AI\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
"\n",
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2b72272258fc"
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"import os\n",
"import sys\n",
"\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = False\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" IS_COLAB = True\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
@@ -415,11 +316,7 @@
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"\n",
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
@@ -430,21 +327,7 @@
},
"outputs": [],
"source": [
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "autoset_bucket"
},
"outputs": [],
"source": [
"if BUCKET_URI == \"\" or BUCKET_URI is None or BUCKET_URI == \"gs://[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -467,26 +350,6 @@
"! gsutil mb -l $REGION $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "validate_bucket"
},
"source": [
"Finally, validate access to your Cloud Storage bucket by examining its contents:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "validate_bucket"
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -501,12 +364,14 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"id": "import_aip:mbsdk"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"import google.cloud.aiplatform as aiplatform"
]
},
@@ -791,7 +656,7 @@
"outputs": [],
"source": [
"dataset = aiplatform.ImageDataset.create(\n",
" display_name=\"flowers_\" + TIMESTAMP,\n",
" display_name=\"flowers\",\n",
" gcs_source=[IMPORT_FILE],\n",
" import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification,\n",
")\n",
@@ -839,7 +704,7 @@
"outputs": [],
"source": [
"dag = aiplatform.AutoMLImageTrainingJob(\n",
" display_name=\"flowers_\" + TIMESTAMP,\n",
" display_name=\"flowers\",\n",
" prediction_type=\"classification\",\n",
" multi_label=False,\n",
" model_type=\"MOBILE_TF_LOW_LATENCY_1\",\n",
@@ -894,7 +759,7 @@
"source": [
"model = dag.run(\n",
" dataset=dataset,\n",
" model_display_name=\"flowers_\" + TIMESTAMP,\n",
" model_display_name=\"flowers\",\n",
" training_fraction_split=0.8,\n",
" validation_fraction_split=0.1,\n",
" test_fraction_split=0.1,\n",
@@ -1037,7 +902,7 @@
"\n",
"# Download the test image in bytes format\n",
"storage_client = storage.Client(project=PROJECT_ID)\n",
"bucket = storage_client.bucket(bucket_name=BUCKET_NAME)\n",
"bucket = storage_client.bucket(bucket_name=BUCKET_URI[5:])\n",
"test_content = bucket.get_blob(\"test.jpg\").download_as_bytes()\n",
"\n",
"# The format of each instance should conform to the deployed model's prediction input schema.\n",
@@ -1306,9 +1171,9 @@
"outputs": [],
"source": [
"dataset = aiplatform.TabularDataset.create(\n",
" display_name=\"gsod_\" + TIMESTAMP,\n",
" display_name=\"gsod\",\n",
" bq_source=[IMPORT_FILE],\n",
" labels={\"user_metadata\": BUCKET_NAME},\n",
" labels={\"user_metadata\": BUCKET_URI[5:]},\n",
")\n",
"\n",
"label_column = \"mean_temp\"\n",
@@ -1376,7 +1241,7 @@
"outputs": [],
"source": [
"dag = aiplatform.AutoMLTabularTrainingJob(\n",
" display_name=\"gsod_\" + TIMESTAMP,\n",
" display_name=\"gsod\",\n",
" optimization_prediction_type=\"regression\",\n",
" optimization_objective=\"minimize-rmse\",\n",
" column_transformations=TRANSFORMATIONS,\n",
@@ -1419,7 +1284,7 @@
"source": [
"model = dag.run(\n",
" dataset=dataset,\n",
" model_display_name=\"gsod_\" + TIMESTAMP,\n",
" model_display_name=\"gsod\",\n",
" training_fraction_split=0.8,\n",
" validation_fraction_split=0.1,\n",
" test_fraction_split=0.1,\n",
@@ -1845,7 +1710,7 @@
"outputs": [],
"source": [
"dataset = aiplatform.TextDataset.create(\n",
" display_name=\"happydb_\" + TIMESTAMP,\n",
" display_name=\"happydb\",\n",
" gcs_source=[IMPORT_FILE],\n",
" import_schema_uri=aiplatform.schema.dataset.ioformat.text.single_label_classification,\n",
")\n",
@@ -1885,7 +1750,7 @@
"outputs": [],
"source": [
"dag = aiplatform.AutoMLTextTrainingJob(\n",
" display_name=\"happydb_\" + TIMESTAMP,\n",
" display_name=\"happydb\",\n",
" prediction_type=\"classification\",\n",
" multi_label=False,\n",
")\n",
@@ -1924,7 +1789,7 @@
"source": [
"model = dag.run(\n",
" dataset=dataset,\n",
" model_display_name=\"happydb_\" + TIMESTAMP,\n",
" model_display_name=\"happydb\",\n",
" training_fraction_split=0.8,\n",
" validation_fraction_split=0.1,\n",
" test_fraction_split=0.1,\n",
@@ -2304,7 +2169,7 @@
"outputs": [],
"source": [
"dataset = aiplatform.VideoDataset.create(\n",
" display_name=\"human_motion_\" + TIMESTAMP,\n",
" display_name=\"human_motion\",\n",
" gcs_source=[IMPORT_FILE],\n",
" import_schema_uri=aiplatform.schema.dataset.ioformat.video.classification,\n",
")\n",
@@ -2342,7 +2207,7 @@
"outputs": [],
"source": [
"dag = aiplatform.AutoMLVideoTrainingJob(\n",
" display_name=\"human_motion_\" + TIMESTAMP,\n",
" display_name=\"human_motion\",\n",
" prediction_type=\"classification\",\n",
")\n",
"\n",
@@ -2379,7 +2244,7 @@
"source": [
"model = dag.run(\n",
" dataset=dataset,\n",
" model_display_name=\"human_motion_\" + TIMESTAMP,\n",
" model_display_name=\"human_motion\",\n",
" training_fraction_split=0.8,\n",
" test_fraction_split=0.2,\n",
")"