mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
fix: Adding autoscaling to RoV cluster management notebook (#3674)
* new notebook * fix issue * linter passed * change pname * linter passed * remove script
This commit is contained in:
Ivan Nardini
@@ -93,6 +93,7 @@
|
||||
"- List existing clusters.\n",
|
||||
"- Get a cluster.\n",
|
||||
"- Manually scale up the cluster, then scale down the cluster.\n",
|
||||
"- Autoscaling a cluster.\n",
|
||||
"- Delete existing clusters."
|
||||
]
|
||||
},
|
||||
@@ -102,7 +103,7 @@
|
||||
"id": "aed92deeb4a0"
|
||||
},
|
||||
"source": [
|
||||
"### Costs \n",
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
@@ -260,9 +261,9 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"VPC_NETWORK = \"[your-network-name]\" # @param {type:\"string\"}\n",
|
||||
"VPC_NETWORK = \"default\" # @param {type:\"string\"}\n",
|
||||
"VPC_NETWORK_FULL = \"projects/{}/global/networks/{}\".format(PROJECT_NUMBER, VPC_NETWORK)\n",
|
||||
"VPC_NETWORK_FULL"
|
||||
"print(VPC_NETWORK_FULL)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -282,7 +283,10 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import vertex_ray"
|
||||
"import time\n",
|
||||
"\n",
|
||||
"import vertex_ray\n",
|
||||
"from ray.job_submission import JobStatus, JobSubmissionClient"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -320,8 +324,8 @@
|
||||
"\n",
|
||||
"cluster_resource_name = vertex_ray.create_ray_cluster(\n",
|
||||
" head_node_type=head_node_type,\n",
|
||||
" network=VPC_NETWORK_FULL,\n",
|
||||
" worker_node_types=worker_node_types,\n",
|
||||
" network=VPC_NETWORK_FULL,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
@@ -346,16 +350,28 @@
|
||||
"clusters"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "lHPHgsiUcC2p"
|
||||
},
|
||||
"source": [
|
||||
"## Scale Ray clusters on Vertex AI"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "be2e79a1af08"
|
||||
},
|
||||
"source": [
|
||||
"## Update an existing cluster (manually scaling)\n",
|
||||
"### Update an existing cluster (manually scaling)\n",
|
||||
"\n",
|
||||
"There are two options for scaling Ray clusters on Vertex AI: Autoscaling and manual scaling.\n",
|
||||
"\n",
|
||||
"Note that the maximum number of worker nodes you can scale up depends on the initial node counts (more details are in these [formulas](https://cloud.google.com/vertex-ai/docs/open-source/ray-on-vertex-ai/set-up)) and is restricted by IP ranges within the same VPC network.\n",
|
||||
"With Manual scaling, you manually update the maximum number of worker nodes you can scale up. Manual scaling gives users more granular control of the nodes.\n",
|
||||
"\n",
|
||||
"Notice that the maximum number of worker nodes you can scale up depends on the initial node counts (more details are in these [formulas](https://cloud.google.com/vertex-ai/docs/open-source/ray-on-vertex-ai/set-up)) and is restricted by IP ranges within the same VPC network.\n",
|
||||
"\n",
|
||||
"Get the cluster you want to scale."
|
||||
]
|
||||
@@ -470,6 +486,220 @@
|
||||
"cluster"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "cdUs_0joVa94"
|
||||
},
|
||||
"source": [
|
||||
"### Autoscaling\n",
|
||||
"\n",
|
||||
"Autoscaling lets the cluster automatically adjust the number of worker nodes based on the resources required by, for example, Ray tasks and actors.\n",
|
||||
"\n",
|
||||
"Autoscaling is recommended if you are running a heavy workload and are unsure of the resources needed."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "aM4Joe7mVxHZ"
|
||||
},
|
||||
"source": [
|
||||
"#### Create a new cluster with autoscaling\n",
|
||||
"\n",
|
||||
"To enable Ray cluster's autoscaling, set the minimum replica count (min_replica_count) and maximum replica count (max_replica_count) of a worker pool.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "LqnvO5KtV0Iu"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"autoscaling_spec = vertex_ray.AutoscalingSpec(\n",
|
||||
" min_replica_count=1,\n",
|
||||
" max_replica_count=3,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"head_node_type = vertex_ray.Resources(\n",
|
||||
" machine_type=\"n1-standard-16\",\n",
|
||||
" node_count=1,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"worker_node_types = [\n",
|
||||
" vertex_ray.Resources(\n",
|
||||
" machine_type=\"n1-standard-16\",\n",
|
||||
" autoscaling_spec=autoscaling_spec,\n",
|
||||
" )\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Create the Ray cluster on Vertex AI\n",
|
||||
"cluster_resource_name = vertex_ray.create_ray_cluster(\n",
|
||||
" cluster_name=\"my-autoscaling-cluster\",\n",
|
||||
" head_node_type=head_node_type,\n",
|
||||
" worker_node_types=worker_node_types,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "d447MGNZYeNr"
|
||||
},
|
||||
"source": [
|
||||
"#### Get the Ray cluster\n",
|
||||
"\n",
|
||||
"After you create the autoscaling cluster, you use the Ray on Vertex AI API to get the cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "J15GjYGjYhMm"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ray_clusters = vertex_ray.list_ray_clusters()\n",
|
||||
"ray_cluster_resource_name = ray_clusters[-1].cluster_resource_name\n",
|
||||
"ray_cluster = vertex_ray.get_ray_cluster(ray_cluster_resource_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "pykjqgjeXS13"
|
||||
},
|
||||
"source": [
|
||||
"### Develop an application using the Ray Jobs API\n",
|
||||
"\n",
|
||||
"To trigger the autoscaling, you develop an Ray application representing an heavy workload.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "7bhOZoTMXZQO"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile my_heavy_workload.py\n",
|
||||
"import ray\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"# Initialize Ray\n",
|
||||
"ray.init()\n",
|
||||
"\n",
|
||||
"# Define a computationally intensive task\n",
|
||||
"@ray.remote(num_cpus=1)\n",
|
||||
"def heavy_task(x):\n",
|
||||
" \"\"\"\n",
|
||||
" Simulates a heavy workload by performing a CPU-bound operation.\n",
|
||||
" This example calculates the sum of squares for a range of numbers.\n",
|
||||
" \"\"\"\n",
|
||||
" total = 0\n",
|
||||
" for i in range(x):\n",
|
||||
" total += i * i\n",
|
||||
" time.sleep(1) # Simulate some work duration\n",
|
||||
" return total\n",
|
||||
"\n",
|
||||
"# Generate a large number of tasks\n",
|
||||
"num_tasks = 1000\n",
|
||||
"results = []\n",
|
||||
"for i in range(num_tasks):\n",
|
||||
" results.append(heavy_task.remote(1000000))\n",
|
||||
"\n",
|
||||
"# Retrieve results (this will trigger autoscaling if needed)\n",
|
||||
"outputs = ray.get(results)\n",
|
||||
"\n",
|
||||
"# Print the sum of the results (optional)\n",
|
||||
"print(f\"Sum of results: {sum(outputs)}\")\n",
|
||||
"\n",
|
||||
"# Terminate the process\n",
|
||||
"ray.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "DbV6JrwkXkoL"
|
||||
},
|
||||
"source": [
|
||||
"### Submit a Ray job using the Ray Jobs API\n",
|
||||
"\n",
|
||||
"Submit the Ray job using the Ray Jobs API through the the public Ray dashboard address."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "JSR9EAn3Xj02"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ray_client = JobSubmissionClient(\n",
|
||||
" \"vertex_ray://{}\".format(ray_cluster.dashboard_address),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_id = ray_client.submit_job(\n",
|
||||
" entrypoint=\"python3 my_heavy_workload.py\",\n",
|
||||
" runtime_env={\n",
|
||||
" \"working_dir\": \".\",\n",
|
||||
" \"pip\": [\n",
|
||||
" \"ray==2.33\",\n",
|
||||
" ],\n",
|
||||
" },\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "ruvE5Jpie2Lk"
|
||||
},
|
||||
"source": [
|
||||
"As soon as you submit the job, go to the [Ray on Vertex AI page](https://console.cloud.google.com/vertex-ai/ray) to see how the cluster is getting update. And you can monitor the autoscaling processing both using Cloud Logging or the public Ray dashboard.\n",
|
||||
"\n",
|
||||
"> Custom upscaling and downscaling speed is not supported. For default values, see [Upscaling and downscaling speed](https://docs.ray.io/en/latest/cluster/vms/user-guides/configuring-autoscaling.html#upscaling-and-downscaling-speed) in the Ray documentation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "68V3VtB_ZJlW"
|
||||
},
|
||||
"source": [
|
||||
"### Monitor the status of the job\n",
|
||||
"\n",
|
||||
"You can use the Ray Jobs API to monitor the status of the job."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "gCAQY1d9ZMj-"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"while True:\n",
|
||||
" job_status = ray_client.get_job_status(job_id)\n",
|
||||
" if job_status == JobStatus.SUCCEEDED:\n",
|
||||
" print(\"Job succeeded!\")\n",
|
||||
" break\n",
|
||||
" else:\n",
|
||||
" if job_status == JobStatus.FAILED:\n",
|
||||
" print(\"Job failed!\")\n",
|
||||
" break\n",
|
||||
" else:\n",
|
||||
" print(\"Job is running...\")\n",
|
||||
" time.sleep(60)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@@ -492,14 +722,16 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete the cluster\n",
|
||||
"vertex_ray.delete_ray_cluster(cluster.cluster_resource_name)"
|
||||
"delete_ray_cluster = False\n",
|
||||
"\n",
|
||||
"if delete_ray_cluster:\n",
|
||||
" for cluster in ray_clusters:\n",
|
||||
" vertex_ray.delete_ray_cluster(cluster.cluster_resource_name)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"collapsed_sections": [],
|
||||
"name": "ray_cluster_management.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user