mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
Adds Colab part and minor changes to ml_ops/stage2/get_started_bqml_training notebook (#491)
* adds the ml_ops/stage2/get_Started_bqml_training notebook to official and removes the same from community folder * ran linter test * updates the textual content * ran linter test * moves the updated stage2/get-started-bqml notebook back to the communit folder * ran linter test * updates the header according to the template * ran linter test * adds colab part and minor changes * ran linter test * retains the newly added code lost in conflicts * ran linter test * converts vertex to vertex ai * ran linter test * moves deletion of temporary BQ table outside delete_storage condition * ran linter test * adds bigquery-storage dependency to the notebook tested on Colab * ran linter test Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Krishna Chaitanya Movva
co-authored by
Karl Weinmeister
Andrew Ferlitsch
Karl Weinmeister
Andrew Ferlitsch
parent
82bffb87f1
commit
fe1a3c0bc9
@@ -38,6 +38,11 @@
|
||||
" View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_bqml_training.ipynb\">\n",
|
||||
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\\\" alt=\"Colab logo\"> Run in Colab\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td>\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage2/get_started_bqml_training.ipynb\">\n",
|
||||
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
|
||||
@@ -137,9 +142,10 @@
|
||||
" USER_FLAG = \"--user\"\n",
|
||||
"\n",
|
||||
"# Install the packages\n",
|
||||
"! pip3 install --upgrade pyarrow $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG\n",
|
||||
"! pip3 install --upgrade google-cloud-bigquery $USER_FLAG"
|
||||
"! pip3 install --upgrade pyarrow \\\n",
|
||||
" google-cloud-aiplatform \\\n",
|
||||
" google-cloud-bigquery \\\n",
|
||||
" google-cloud-bigquery-storage $USER_FLAG -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -177,6 +183,23 @@
|
||||
"id": "project_id"
|
||||
},
|
||||
"source": [
|
||||
"### Set up your Google Cloud project\n",
|
||||
"\n",
|
||||
"**The following steps are required, regardless of your notebook environment.**\n",
|
||||
"\n",
|
||||
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
||||
"\n",
|
||||
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
||||
"\n",
|
||||
"1. [Enable the Vertex AI, BigQuery, Compute Engine and Cloud Storage APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,bigquery,compute_component,storage_component).\n",
|
||||
"\n",
|
||||
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
|
||||
"\n",
|
||||
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
|
||||
"Cloud SDK uses the right project for all the commands in this notebook.\n",
|
||||
"\n",
|
||||
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n",
|
||||
"\n",
|
||||
"#### Set your project ID\n",
|
||||
"\n",
|
||||
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
|
||||
@@ -277,6 +300,63 @@
|
||||
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "f3bd8c0d0469"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate your Google Cloud account\n",
|
||||
"\n",
|
||||
"**If you are using Google Cloud Notebooks**, your environment is already authenticated. Skip this step.\n",
|
||||
"\n",
|
||||
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
|
||||
"\n",
|
||||
"**Otherwise**, follow these steps:\n",
|
||||
"\n",
|
||||
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
|
||||
"\n",
|
||||
"1. **Click Create service account**.\n",
|
||||
"\n",
|
||||
"2. In the **Service account name** field, enter a name, and click **Create**.\n",
|
||||
"\n",
|
||||
"3. In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex AI\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
|
||||
"\n",
|
||||
"4. Click Create. A JSON file that contains your key downloads to your local environment.\n",
|
||||
"\n",
|
||||
"5. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "e0953a00668e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you are running this notebook in Colab, run this cell and follow the\n",
|
||||
"# instructions to authenticate your GCP account. This provides access to your\n",
|
||||
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
|
||||
"# requests.\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"# If on Google Cloud Notebook, then don't execute this code\n",
|
||||
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
|
||||
" if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth as google_auth\n",
|
||||
"\n",
|
||||
" google_auth.authenticate_user()\n",
|
||||
"\n",
|
||||
" # If you are running this notebook locally, replace the string below with the\n",
|
||||
" # path to your service account key and run this cell to authenticate your GCP\n",
|
||||
" # account.\n",
|
||||
" elif not os.getenv(\"IS_TESTING\"):\n",
|
||||
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -985,7 +1065,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "1bb996026c94"
|
||||
"id": "2b4498ca6fea"
|
||||
},
|
||||
"source": [
|
||||
"## Model Registry\n",
|
||||
@@ -1001,7 +1081,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "eaaf24146aad"
|
||||
"id": "0472e888105e"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1017,7 +1097,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "8bfc0b26155f"
|
||||
"id": "6c390ee7c11a"
|
||||
},
|
||||
"source": [
|
||||
"### Training and registering the model\n",
|
||||
@@ -1033,7 +1113,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "3ff1d4ef4df2"
|
||||
"id": "57db464f4c42"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1070,7 +1150,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "a243d86f9d22"
|
||||
"id": "5b4970272040"
|
||||
},
|
||||
"source": [
|
||||
"### Find the model in the `Vertex Model Registry`\n",
|
||||
@@ -1082,7 +1162,7 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "4fd9a143d900"
|
||||
"id": "76c22674ba99"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -1108,7 +1188,9 @@
|
||||
"- Model\n",
|
||||
"- Endpoint\n",
|
||||
"- Dataset\n",
|
||||
"- Cloud Storage Bucket"
|
||||
"- Cloud Storage Bucket\n",
|
||||
"\n",
|
||||
"Set `delete_storage` to _True_ to delete the Cloud Storage bucket used in this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1124,16 +1206,15 @@
|
||||
"endpoint.delete()\n",
|
||||
"\n",
|
||||
"# Delete the model using the Vertex model object\n",
|
||||
"try:\n",
|
||||
" model.delete()\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
"model.delete()\n",
|
||||
"\n",
|
||||
"if os.getenv(\"IS_TESTING\"):\n",
|
||||
"# Delete the created BigQuery dataset\n",
|
||||
"! bq rm -r -f $PROJECT_ID:$BQ_DATASET_NAME\n",
|
||||
"\n",
|
||||
"delete_storage = False\n",
|
||||
"if delete_storage or os.getenv(\"IS_TESTING\"):\n",
|
||||
" # Delete the created GCS bucket\n",
|
||||
" ! gsutil rm -r $BUCKET_NAME\n",
|
||||
" # Delete the created BigQuery dataset\n",
|
||||
" ! bq rm -r -f $PROJECT_ID:$BQ_DATASET_NAME"
|
||||
" ! gsutil rm -r $BUCKET_NAME"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user