Site down (and fully recovered now) --- status

AppSheet is fully back up again. While we will do a deeper analysis, it appears this happened because of automatic security enforcement rules that (erroneously) shutdown firewall access for our own services. It took a while to discover this, and once we did, we had to manually override this change and disable the automatic enforcer.
This is about all we know at the moment. Exactly why this happened at 2:25PM PST on a Tuesday is unclear and we are determined to find out.


We are aware of and actively investigating a major incident. Will provide status updates here.

12 25 1,619
25 REPLIES 25

As far as we are aware, all AppSheet services are down.

3X_b_3_b3c1d24983e7faee49a22ea4569cbdee595bafb0.gif

The issue started approx 2:25PM PST. No AppSheet deployment was ongoing at that time. It appears to be a network configuration issue. Weโ€™re trying to determine what exactly. At the moment, our services are receiving no network traffic and that is why nobody is able to access the services.

Thanks for quick updates. Hope resolves quickly so we can deploy an app to our partners tonight!

We are back in.

ya estamos en linea

Thanks !!! Working now

Thanks!!!

On itโ€™s way back, but still some instability expected for a few minutes.

Cool, looks like we are back up but a little slow in New Zealand - thanks kindly!

Darmund
Participant III

Well done yall! Barely had enough time to make a coffee

Itโ€™s going to be slow for a few minutes as the server farm ramps up again (scales out with many instances โ€” all the instances had scaled down as the traffic completely dropped).

Could this event have caused a failure to access files and images uploaded in the application? I canโ€™t acces them by now

Hi @Cesferb , not it should not have. Any data uploads during that period would have failed and the changes would have been queued locally until they could have been saved.
The only exception is if you were running in a browser and you closed the browser before the data was synced to the server,

Terry_Wong
Participant II

Thanks for the update. However, my users are still experiencing severe increase in Sync Time and failures.






I wonder if the above issues are also related to this incident?

My apps call some webhooks in Automation, but it isnโ€™t going well. Please check and share that result.

Tagging @Zhifeng_Lin

Can you please share some details on what you are experiencing? Please send specifics to Appsheet-support

I am using some bots to call webhook(both data change trigger and schedule trigger) and these worked well till 8/9/2021 9:30:01 PM(UTC). After that time, bots are working but all webhook gots timeout. Exception message is as below:

 "Exception": "Webhook HTTP post request failed with exception Unable to connect to the remote server A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.41.0.250:8080",

for checking purpose, I call webhook from other channel and went well without any problem.

@Takuya_Tsuji , that IP address for your webhook looks like a private IP and therefore would not be reachable from the AppSheet services.

Hi @Takuya_Tsuji ,
Iโ€™m Takuya, too.

I may have tried this in another channel, but does the same phenomenon continue when you create a new Bot?
I just created a new Bot that calls via webhook towards Google Chat API in my environment and it was successful.
It doesnโ€™t seem that the webhook engine is down for the whole AppSheet.

@Takuya_Miyai
Thank you for sharing your solution.
I am trying yours but no good so far.

@Harsh_Ch
FYI, I call Apps Script as API via webhook.

Jonathan1
Participant IV

Same for us, webhooks are no more sent and makes our Apps not syncing, stuck on โ€œSaving changesโ€.

I have contacted support this morning but still no reply.

@Jonathan1 could you check and confirm now? More stuff related to the firewall issues from yesterday blocking some webhook usage. We think it has been fixed now.

Ah yes!!! It seems to work now.

Thank you