Recovering from "Operation timed out" ZONE_RESOURCE_POOL_EXHAUSTED when starting Cloud Workstations

[TimJohns]

When attempting to start Cloud Workstations in one of my clusters in the Google cloud console, I frequently (I estimate 10% of the time) get a notification that "Operation timed out" in the console, and see from the logs this is due to ZONE_RESOURCE_POOL_EXHAUSTED. 

I am curious whether this is common or a known issue - my cluster is in us-west1, and presumably there's just no compute instances available that fit my workstation cluster's configuration.

More importantly, recovering is kind of a pain - using "retry" in the cloud console immediately results in a new error notification "unable to queue the operation", and the Status column on https://console.cloud.google.com/workstations/myworkstations seems to be stuck on "Starting".

Eventually a manual retry will work, but I suggest it would be better if "retry" would queue the request instead of me having to manually retry periodically, or, alternately, if the "Status" column could display something more clear than "Starting", that indicates I need to take some action.

Thanks, and otherwise loving the Workstations so far! Among the more well-known benefits, I also just appreciate just not having to lug a designated "dev-capable" laptop around.

-Tim

[dionv]

Hello TimJohns,

First, our apologies that the Google Cloud resources you need are not currently available. Our goal is to make sure that there are resources in all zones. We take this responsibility very seriously and are working hard to make resources available as soon as possible.

We recommend deploying and balancing your workload across multiple zones and regions to reduce the likelihood of an outage and have access to multiple resource pools when you need to expand quickly. Please review our documentation which outlines how to build resilient and scalable architectures on Google Cloud Platform. Please note that you are currently using Google Cloud on-demand consumption model which does not provide an assurance in obtaining capacity. We now offer a feature called reservations that provides a
very high level of assurance in obtaining capacity on the Google Cloud platform, please see our documentation for details on how to use this feature.

Again, we want to offer our sincerest apologies. We are working hard to increase our capacity and make this an exceptionally rare event.

[login]

ZONE_RESOURCE_POOL_EXHAUSTED is an error message that appears when Google Cloud Platform (GCP) cannot allocate enough resources in time ⏲to fulfill a request. This error message indicates that there are not enough resources in the zone to create a new Cloud Workstation instance that matches your desired configuration.

This error can occur due to various reasons such as an insufficient quota limit for the particular resource, no available capacity in the desired zone or region, and other factors such as maintenance activities. It is not uncommon to encounter this error message occasionally when attempting to start Cloud Workstations or other types of compute instances in GCP.

To mitigate this error, you can try the following steps:

Check your quota limits: Verify that you have sufficient quota limits for the resources that you are trying to create. You can check your quotas in the Google Cloud Console by going to the IAM & Admin > Quotas page.

Try a different zone: You can try creating your Cloud Workstation instances in a different zone in the same region or a different region to see if there is available capacity.

Use automatic scaling: You can also consider using the Cloud Workstations' automatic scaling feature, which can create and delete instances based on demand. This can help avoid resource exhaustion errors.

If the issue persists, you can reach out to GCP support for further assistance in resolving the issue.