UPDATE 11:46 AM PT - U.S. Pacific Time
Hello everyone. AppSheet continues to experience high sync and homepage load latencies. I recognize this is is a significant impact to your applications and I recognize that it is peak business hours in many affected areas. I apologize for the impact that this has had to your business operations.
We have been investigating this issue for several hours and are actively deploying mitigations to our server infrastructure. These take time to take effect, and our graphs and metrics indicate that the system has not yet recovered.
My self and multiple other infrastructure engineers are actively working on this issue. I am working to get the GCP Cloud Status Dashboard updated to correctly reflect the service interruption.
Although the symptom appears the same as the July 4 incident (which also saw high sync latencies), our initial triage indicates that the underlying root cause is different. While in that case, it was a matter of scaling up additional capacity, in this case, the capacity is there but the server processes themselves are unhealthy. In other words, encountering conditions that prevent them from fulfilling client requests within reasonable latencies.
I understand and validate the frustration felt from a service outage in this regard. I've heard the request for a transparent post mortem, and we will work to provide that in the coming days, once we understand the root cause. This includes remediations to prevent a similar occurrence in the future.
I am actively working on the problem here with team members. I will post a follow-up in an hour, at 12:55 AM PT (US Pacific Time).
Mike Procopio
LinkedIn
Twitter
