Compare commits

...
Author SHA1 Message Date
Andrew Ferlitsch a541418ee8 fix: syntax error 2023-06-21 16:32:13 +00:00
Andrew Ferlitsch 7164c8aaf1 fix: np type 2023-06-21 16:20:09 +00:00
Andrew FerlitschandGitHub e1ff9a81c3 add import numpy 2023-06-20 08:59:13 -07:00
Andrew Ferlitsch 9056171982 fix: numpy type 2023-06-20 15:40:44 +00:00
Andrew FerlitschandGitHub e93a42cc77 install numpy 2023-06-12 09:00:53 -07:00
Andrew Ferlitsch 08bedf97d8 fix: boilerplate reduction 84 2023-06-07 21:48:07 +00:00
@@ -129,48 +129,6 @@
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y320EIk-kXT7"
},
"source": [
"### Set up your local development environment\n",
"\n",
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
"all the requirements to run this notebook. You can skip this step.\n",
"\n",
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
"You need the following:\n",
"\n",
"* The Google Cloud SDK\n",
"* Git\n",
"* Python 3\n",
"* virtualenv\n",
"* Jupyter notebook running in a virtual environment with Python 3\n",
"\n",
"The Google Cloud guide to [Setting up a Python development\n",
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
"for meeting these requirements. The following steps provide a condensed set of\n",
"instructions:\n",
"\n",
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
"\n",
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
"\n",
"1. [Install\n",
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"\n",
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
"command-line in a terminal shell.\n",
"\n",
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"\n",
"1. Open this notebook in the Jupyter Notebook Dashboard."
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -188,165 +146,86 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"\n",
"! pip3 install {USER_FLAG} --upgrade pandas-gbq 'google-cloud-bigquery[bqstorage,pandas]' tensorflow scikit-learn protobuf==3.20.3 -q"
"! pip3 install --upgrade --quiet pandas-gbq \\\n",
" 'google-cloud-bigquery[bqstorage,pandas]' \\\n",
" tensorflow \\\n",
" scikit-learn \\\n",
" numpy \\\n",
" protobuf==3.20.3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aNeMRbpukXUA"
"id": "58707a750154"
},
"source": [
"### Restart the kernel\n",
"\n",
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
"### Colab only: Uncomment the following cell to restart the kernel."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dJ_yvi_9kXUB"
"id": "f200f10a1da3"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a47846030fef"
},
"source": [
"## Before you begin"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "71b29797836c"
},
"source": [
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. <a href=\"https://console.cloud.google.com/cloud-resource-manager\" target=\"_blank\">Select or create a Google Cloud project</a>. When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. <a href=\"https://cloud.google.com/billing/docs/how-to/modify-project\" target=\"_blank\">Make sure that billing is enabled for your project</a>.\n",
"\n",
"1. <a href=\"https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com\" target=\"_blank\">Enable the Vertex AI API</a>.\n",
"\n",
"1. If you are running this notebook locally, you will need to install the <a href=\"https://cloud.google.com/sdk\" target=\"_blank\">Cloud SDK</a>.\n",
"\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a7be962cec10"
"id": "WReHDGG5g0XY"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3c8049930470"
"id": "oM1iC_MfAts1"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a36c4b991a39"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "684595f229b3"
},
"outputs": [],
"source": [
"! gcloud config set project $PROJECT_ID"
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "03d8d65b914d"
"id": "region"
},
"source": [
"#### Region\n",
"#### Set the region\n",
"\n",
"You can also change the `REGION` variable, which is used for operations\n",
"throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
"\n",
"- Americas: `us-central1`\n",
"- Europe: `europe-west4`\n",
"- Asia Pacific: `asia-east1`\n",
"\n",
"You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
"\n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
"**Optional**: Update the 'REGION' variable to specify the region that you want to use. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3281bedf6d3c"
"id": "nsN5NJKSu-GU"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}\n",
"\n",
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
@@ -383,114 +262,90 @@
{
"cell_type": "markdown",
"metadata": {
"id": "OoPGk5KOkXUG"
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
"authenticated. Skip this step.\n",
"To authenticate your Google Cloud account, follow the instructions for your Jupyter environment:\n",
"\n",
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"**1. Vertex AI Workbench**\n",
"<br>You are already authenticated.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"2. Click **Create service account**.\n",
"\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
"**2. Local JupyterLab instance**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Teyy6LGqkXUG"
"id": "254614fa0c46"
},
"outputs": [],
"source": [
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"import os\n",
"import sys\n",
"\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7f0d924a47aa"
"id": "ef21552ccea8"
},
"source": [
"**3. Colab**\n",
"<br>Uncomment and run the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "603adbbf0532"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zgPO1eR3CYjk"
},
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"Set the name of your Cloud Storage bucket below. It must be unique across all\n",
"Cloud Storage buckets.\n"
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "59ebce2a16fa"
"id": "MzGDU7TWdts_"
},
"outputs": [],
"source": [
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d0caa7b3c8b8"
},
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
" BUCKET_URI = f\"gs://{BUCKET_NAME}\""
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7a7c87b6f171"
"id": "create_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
@@ -500,31 +355,11 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "76f7e3e043f5"
"id": "NIq7R4HZCfIc"
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f4353e94da5f"
},
"source": [
"**Finally**, validate access to your Cloud Storage bucket by examining its contents:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6f092d6ebc92"
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
]
},
{
@@ -544,6 +379,7 @@
},
"outputs": [],
"source": [
"import os\n",
"import warnings\n",
"\n",
"import pandas as pd\n",
@@ -951,6 +787,10 @@
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"X_train = np.asarray(X_train, dtype=np.float)\n",
"\n",
"history = model.fit(X_train, y_train_categ, epochs=50, verbose=1)"
]
},
@@ -971,6 +811,8 @@
},
"outputs": [],
"source": [
"X_test = np.asarray(X_test, dtype=np.float)\n",
"\n",
"test_results = model.evaluate(X_test, y_test_categ, verbose=1)\n",
"print(f\"Test results - Loss: {test_results}\")"
]