RESOLVED: Issues with AppSheet Service: June 17 2019

[SantiagoU]

UPDATE: the service is restored and everyone should be seeing normal behavior. (caveat: if you use the REST API, please see below). Details and post-mortem later today.

The service was disrupted for a couple of hours (approx 8AM PST to 10AM PST) except for customers in Europe who had uninterrupted service. The general symptom was very slow load times for the website and timeouts for app syncs. The root cause was server overload caused by an inadvertent runaway customer process that swamped our REST API and effectively denied service to all other traffic.

At the moment, we have disabled the REST API for all customers while we put some more effective throttling in place. We hope to have this fully enabled later today.

…

Hi community, we are experiencing issues with the AppSheet service impacting applications and our website. The engineering team is investigating, we will keep you posted with updates and resolution.

[pravse]

Some of the servers are back up again. But it may be a while before it catches up with the backlog.

[Aaron_Blau]

My apps appear to be back to normal.

[pravse]

most of the servers should be normal again. more details forthcoming.

[pravse]

We have temporarily disabled the REST API because the source of the problem was a runaway customer process that was hitting the API with so many requests that it led to a complete denial of service.

Other than the REST API, the rest of the service should now be restored.

More details and post-mortem coming later today.

[Yogesh_laddha]

Will this affect migration to mySQL DB?

I am trying to migrate an existing app to mySQL DB. It is giving an error, AppTemplate not present.
Is this related to this?

[Spencer_Jones]

Thank you for resolving that so quickly. hAPPy users all around.

[Bituminous_Road]

is it broke again?

Edit: Maybe not?

[Midland_Freewhe]

There was definitely an issue here in the UK. It’s possible that there is still something not quite right as we’re seeing record changes not being updated even when we get our riders to carry out another sync.

[SteveA1]

One of my app users is unable to see his apps. He has deleted and re-installed appsheet twice but still has no apps available. Any ideas?

[pravse]

@SteveA this issue is unrelated to our outage this morning. Most likely your user is signing in with a different email address. If you are unable to resolve, please contact support@appsheet.com with details.

@Midland_Freewheelers if you’re seeing record changes not being updated, it is most likely because you have duplicate keys. It is also unrelated to our outage this morning. Please provide details to support@appsheet.com if unable to resolve.

[pravse]

Hi Yogesh, could yoy please send details to support@appsheet.com Thanks

[Yogesh_laddha]

Done.

[SantiagoU]

Hi everyone, we’ve posted a post-mortem of the June 17 incident in the community. Please follow the link below to learn more about the issue and how we plan to mitigate and improve the service in the future.

Post Mortem: AppSheet Service Outage: 06/17/2019 Announcements

The AppSheet service was very slow and significantly unresponsive for approximately 2 hours from 8AM PST to 10AM PST on Monday June 17th. The root cause of the problem was overloading of our North American servers due to an unintentional infinite loop generated by a customer app through the REST API mechanism. The customer made an erroneous change to their app just before 8AM PST and this set off the overload. We maintain a monitoring service that alerts some of our key engineers/managers when …

[Jason_Pope]

Hello,

I can’t log into my AppSheet account, I use google to sign in and when I attempt to log in I receive an error message telling me that I need to contact my administrator of my google account… I am the am the administrator.

Please Help

Screen Shot 2019-06-20 at 4.34.19 PM

[Steve]

Please contact support@appsheet.com directly for help with this.

[Jason_Pope]

[pravse]

@Jason_Pope — this is a google problem that has been popping up sporadically for a couple of months. Please see the resolution here: Login access denied by Google

[Jason_Pope]

@Praveen Seshadri it worked Thanks!

[Challa_Ramakris]

Hi @Santiago,

I’m facing a strange error with REST API.

Error description is
Error:
“Webhook HTTP post request failed with exception {“Message”:“REST API invoke request failed: Error: Value ‘22-06-2019’ in field ‘Date’ in table ‘Attendance’ cannot be converted to type ‘Date’.”} The remote server returned an error: (400) Bad Request.”

In my app all dates are in ddmmyyyy format. But this API is trying to insert date value in mmddyyyy format hence the error is appearing without completing the REST API.

It was working fine last week.
Please check the same.

[pravse]

@Challa_Ramakrishna please see the last paragraph of this article. https://help.appsheet.com/articles/2247996-troubleshooting-api-calls
Are you passing in the locale correctly?

[Challa_Ramakris]

Thank you @praveen