Some apps are having issues - tables are not being found?

[Tiger1]

Anyone else having app trouble with tables?

[Brandt_Litton]

I’m having issues in multiple apps. Finally got a response from Support that says:

“Hello, Thanks for reaching out to Google Support. We’ve had several customers reporting this issue. The AppSheet team is currently investigating this issue, so we’ll prioritize your request accordingly. We’ll have an update for you as soon as we hear back from the AppSheet team.”

[Steve]

Known problem likely tied to today’s release. The release is being rolled-back.

[Tiger1]

Well, Steve as soon as I said it - BAM! All went back to normal. Thanks for your quick response as usual. Thanks!

[Brandt_Litton]

Everything seems back to normal on my apps now, too. Thanks!

[pravse]

Here’s a brief update on what happened.

Brief version: I broke the system with a code change I made and we had to immediate rollback to a previous good build.

Details:
I have been working for a few days to implement a better version of server caching of table data. This is a feature that we have had for a long time, but we think it could be more efficient (and therefor syncs can run a bit faster) with a slightly different implementation. I made the code changes and like all other code changes, it went thru review with other devs before being added to the main code base. In this case, it went thru exhaustive review and discussion. We also tested this both with my change and without, and both passed. We generally put such changes in a “rollout” (which means some users see it and some don’t). Initially, we roll things out so that only Google-internal or team-internal users see it. That was supposed to be the case here.

However, we still had a problem when we deployed this morning. We were able to rollback in about 10 to 15 minutes. In that window, anybody with apps that had the server-caching option enabled were unable to run their apps.

The actual root cause was a bug in my code, of course, but bugs are normal when you write code and we expect to find them during testing. While we will do a deeper post-mortem, it appears the root cause of the problem was that when we run tests, we do not run them against our production infrastructure. This is generally to ensure that production systems are not overloaded or corrupted by test traffic. There was a small but subtle difference between the caching infrastructure used in the test framework and the caching infrastructure used in production. And that only came to light after deployment.

What can we do to fix this in the future — quite a few things. For every such event, we have an internal process at Google to fill out a detailed document, discuss it at a post-mortem meeting with all the engineering leads, and ensure that some mitigations are put in place for the future. I will be doing this over the next few days.

[Kirk_Masden]

I think this mea culpa is an example of the leadership that has made AppSheet such an outstanding and successful company. Of course, it’s good if problems can be headed off in advance but, when they do arise, it’s great to get a clear explanation like this.

[Koichi_Tsuji]

The problem I observe here as an example is there is weakness in tems of QA (Quality Assuarance) before new code is release to the production environment, which I have been highlighing for years. I dont have clear stats but I would say more than 30 pct of my hours on appsheet editor is spent to debug the issues, or find the way to reproduce the issue before we report to support or any new issue to the production app which has been working for years, then all of the sudden it does not work as expected.

We are using bunch of SAAS for our ongoing business, but Appsheet is one of the worse score for this part, i.e. repeated problems once new code is released without we knowing that, where we only know that fact new code is release when we see bunch of problem.

I just simply hope QA part of Appsheet is going to improve somewhat so that the platfomr is made to be more stable and reliable.

[Tiger1]

Thank you very much for all your help, support and detail.

[Tiger1]

Praveen,
What are the best practices when it comes to “Server caching” and “Delta Sync”? I have both checked - in most apps - is that normal, bad, better? I am curious to know what I should be doing per apps…

[pravse]

@Tiger , the two are unrelated options.

In general, if you have read-only tables (and as much as possible, you should make tables read-only), then you should benefit from server-side caching of those read-only tables.

Then separately, if your data is in sheets on Google Drive, the delta-sync option looks for a last-modified timestamp on the file and fetch the data only if that has changed. This is a somewhat involved option (it has a number of caveats that you need to understand well – so do read the docs on this), so please disable this by default unless you know exactly how it works and the potential anomalies you are willing to overlook in exchange for the performance benefit.

A few different (and in some ways better) options are being developed, so we hope to have to more to offer in this area soon