fix: install pyspark

notebooks/official/workbench/spark/spark_bigquery.ipynb

@@ -54,7 +54,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "tvgnzT1CKxrO"
@@ -125,25 +124,6 @@
         "You can use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "BF1j6f9HApxa"
-      },
-      "source": [
-        "## Before you begin\n",
-        "\n",
-        "### Set up your Google Cloud project:\n",
-        "\n",
-        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you create an account, you receive a $300 credit towards to your compute and storage costs.\n",
-        "\n",
-        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
-        "\n",
-        "1. [Enable the Notebooks API, Vertex AI API, and Dataproc API](https://console.cloud.google.com/flows/enableapi?apiid=notebooks.googleapis.com,aiplatform.googleapis.com,dataproc&_ga=2.209429842.1903825585.1657549521-326108178.1655322249)\n",
-        "\n",
-        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and inserts the value of Python variables prefixed with `$` into the commands."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -155,25 +135,6 @@
         "Install the following packages to run this notebook."
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "172533a994ad"
-      },
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "# The Google Cloud Notebook product has specific requirements\n",
-        "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
-        "\n",
-        "USER_FLAG = \"\"\n",
-        "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
-        "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
-        "    USER_FLAG = \"--user\""
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -191,6 +152,8 @@
       },
       "outputs": [],
       "source": [
+        "import os\n",
+        "\n",
         "if os.getenv(\"IS_TESTING\"):\n",
         "    \"\"\"\n",
         "    The testing suite does not currently support testing on Dataproc clusters,\n",
@@ -212,7 +175,7 @@
         "    ! echo $JAVA_HOME\n",
         "\n",
         "    # Pin the Spark version to match that the Dataproc 2.0 cluster.\n",
-        "    ! pip install {USER_FLAG} pyspark==$PYSPARK_VER -q"
+        "    ! pip install pyspark==$PYSPARK_VER -q"
       ]
     },
     {
@@ -293,58 +256,24 @@
       "source": [
         "### Set your project ID\n",
         "\n",
-        "Run the following cell to get you project ID."
+        "**If you don't know your project ID**, try the following:\n",
+        "* Run `gcloud config list`.\n",
+        "* Run `gcloud projects list`.\n",
+        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "oM1iC_MfAts1"
+        "id": "set_project_id"
       },
       "outputs": [],
       "source": [
-        "import os\n",
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
         "\n",
-        "PROJECT_ID = \"\"\n",
-        "\n",
-        "# Get your Google Cloud project ID from gcloud\n",
-        "if not os.getenv(\"IS_TESTING\"):\n",
-        "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
-        "    PROJECT_ID = shell_output[0]\n",
-        "    print(\"Project ID: \", PROJECT_ID)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "qJYoRfYng0XZ"
-      },
-      "source": [
-        "If the previous command has no output, copy your project ID from the project selector in the [Google Cloud console](https://console.cloud.google.com/). Insert the ID in the `[your-project-id]` placeholder, then run the following command."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "riG_qUokg0XZ"
-      },
-      "outputs": [],
-      "source": [
-        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
-        "    PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\"}"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "set_gcloud_project_id"
-      },
-      "outputs": [],
-      "source": [
-        "! gcloud config set project $PROJECT_ID -q"
+        "# Set the project id\n",
+        "! gcloud config set project {PROJECT_ID}"
       ]
     },
     {