Compare commits

...
Author SHA1 Message Date
Andrew FerlitschandGitHub 855fa12dd1 fix: install 2023-06-07 16:32:01 -07:00
Andrew Ferlitsch ea9e3b0736 fix: boilerplate reduction 86 2023-06-07 23:16:01 +00:00
@@ -124,48 +124,6 @@
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dNN9pAmip93z"
},
"source": [
"### Set up your local development environment\n",
"\n",
"**If you are using Colab or Vertex AI Workbench Notebooks**, your environment already meets\n",
"all the requirements to run this notebook. You can skip this step.\n",
"\n",
"**Otherwise**, make sure your environment meets this notebook's requirements.\n",
"You need the following:\n",
"\n",
"* The Google Cloud SDK\n",
"* Git\n",
"* Python 3\n",
"* virtualenv\n",
"* Jupyter notebook running in a virtual environment with Python 3\n",
"\n",
"The Google Cloud guide to [Setting up a Python development\n",
"environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
"installation guide](https://jupyter.org/install) provide detailed instructions\n",
"for meeting these requirements. The following steps provide a condensed set of\n",
"instructions:\n",
"\n",
"1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
"\n",
"1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
"\n",
"1. [Install\n",
" virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
" and create a virtual environment that uses Python 3. Activate the virtual environment.\n",
"\n",
"1. To install Jupyter, run `pip3 install jupyter` on the\n",
"command-line in a terminal shell.\n",
"\n",
"1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n",
"\n",
"1. Open this notebook in the Jupyter Notebook Dashboard."
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -183,135 +141,187 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"\n",
"! pip3 install {USER_FLAG} --upgrade pandas-gbq 'google-cloud-bigquery[bqstorage,pandas]' scikit-learn"
"! pip3 install --quiet --upgrade pandas-gbq 'google-cloud-bigquery[bqstorage,pandas]' scikit-learn"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QLnr7HGVp934"
"id": "e9255e3b156f"
},
"source": [
"### Restart the kernel\n",
"\n",
"After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
"### Colab Only: Uncomment the following cell to restart the kernel"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h5Vuu7x7p935"
"id": "0c0b2427998a"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs\n",
"import os\n",
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # Automatically restart kernel after installs\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "75568876f1a4"
"id": "435b8e413535"
},
"source": [
"## Before you begin"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "276011faac6f"
},
"source": [
"### Set up your Google Cloud project\n",
"### Before you begin\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI, Cloud Storage, and Compute Engine APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage-component.googleapis.com). \n",
"\n",
"1. [Configure your Google Cloud project for Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project).\n",
"\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
"\n",
"1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
"Cloud SDK uses the right project for all the commands in this notebook.\n",
"\n",
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3b3a6979860"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
"**If you don't know your project ID**, try the following:\n",
"- Run `gcloud config list`\n",
"- Run `gcloud projects list`\n",
"- See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b453e095976a"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a36c4b991a39"
},
"outputs": [],
"source": [
"if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
" # Get your GCP project id from gcloud\n",
" shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID:\", PROJECT_ID)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f2e3c0f2cbfb"
"id": "be175254a715"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# set the project id\n",
"! gcloud config set project $PROJECT_ID"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2e6b8b324ce1"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. \n",
"Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ae43d96c4b1b"
},
"outputs": [],
"source": [
"REGION = \"[your-region]\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6c43a8673066"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench** \n",
"- Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab Instance,** uncomment and run."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fbc9cd30cc4b"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cd0da2c26879"
},
"source": [
"**3. Colab,** uncomment and run:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a336a05c6149"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0461097edfa5"
},
"source": [
"**4. Service Account or other**\n",
"- See all the authentication options here: [Google Cloud Platform Jupyter Notebook Authentication Guide](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_authentication_guide.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e5755d1a554f"
},
"source": [
"### Create a Cloud Storage bucket\n",
"\n",
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d2de92accb67"
},
"outputs": [],
"source": [
"BUCKET_NAME = \"your-bucket-name-unique\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b72bfdf29dae"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a4453435d115"
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -343,81 +353,6 @@
"UUID = generate_uuid()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_RFbibT9p937"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Vertex AI Workbench Notebooks**, your environment is already\n",
"authenticated. Skip this step."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gpGsDv_8p937"
},
"source": [
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"2. Click **Create service account**.\n",
"\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Suj2JY6ep937"
},
"outputs": [],
"source": [
"# If you are running this notebook in Colab, run this cell and follow the\n",
"# instructions to authenticate your GCP account. This provides access to your\n",
"# Cloud Storage bucket and lets you submit training jobs and prediction\n",
"# requests.\n",
"\n",
"import os\n",
"import sys\n",
"\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
" # path to your service account key and run this cell to authenticate your GCP\n",
" # account.\n",
" elif not os.getenv(\"IS_TESTING\"):\n",
" %env GOOGLE_APPLICATION_CREDENTIALS ''"
]
},
{
"cell_type": "markdown",
"metadata": {