Automation critical problem and workaround - Bots ...

Joseph_Seddik · 03-13-2023 09:20 AM

Hello Everyone, I'm glad to be back among you!

It seems that now Bots are failing when there's an action with an "Only if this condition is true" condition configured. If the condition's expression result is FALSE, instead of just omitting launching the action as it has been the case since forever, AppSheet servers now exhibit the following behavior:

Actions launched on the device, i.e. through an action button, behave normally.
Actions launched through Automation, from Bots, fail with an error when the condition is not fulfilled.
What is fatal is that the WHOLE Bot fails and all subsequent steps in the bot's process are not executed, generating all kind of data errors you can imagine from Bots in limbo that had neither stopped from running nor completed, but simply stopped in the middle ruining all kinds of workflows and data checks you had put in place to ensure production data sanity.

The problem was reported to me from everyone on Friday, and I spent the whole weekend day and night replacing all of the these bots.

The workarounds I implemented are:

Move the whole bot when possible to a grouped Form-Save action. On top of the extra-complexity. involved, this has the disadvantages of:
1. NOT accommodating for data changes that are not triggered by a form-save event.
2. Dependence on the user's device and whether it completes the whole process, slowing down the originating user's UX and slower reflection of data changes to other users.
Separating each of the failing steps in its own bot using the following steps:
1. In place of the failing step in the original bot, launch an action to create a new row in another table with the columns holding the data you need. YES unfortunately you have to create an unnecessary new buffer table.
2. Create a new Bot on an event of adding a new row to that buffer table, and put the failing step in that bot. This bot will execute till the action's condition fails then the bot will fail, but it will NOT affect the original calling bot which will continue to execute its steps.

The major disadvantage here is that you can NOT control the order of steps as it was the case when they were all inside the same bot, so you have to be aware of this and act accordingly implementing further controls to make sure you maintain integrity of data.

I would strongly discourage dividing the failing bots into both device-based actions (buttons, form-save) and server-based automation through bots. Either you move the whole failing bots to a grouped action triggered by user interaction, or keep the bot while moving the failing steps to new bots as in B.

I'm glad I kept my customers happy and ensured their business continuity. However, besides my own wasted time handling customer complaints, investigating and implementing solution, I can't really process how would AppSheet people implement such a change silently with a devastating impact on production apps! How could you for 3-4 days now be silent after a change causing such panic! For the first time ever sine I adopted AppSheet as the number-one preferred and recommended platform to my clients, even for very complex processes, putting it at the heart of the whole ecosystem, I'm now wondering whether this would be a wise decision to make.

Too bad AppSheet, just too bad !!

Rifad

We all have been complaining and contacting support every other day due to this kind of issues and still continues to do so. Its very pathetic to see this. Not sure how many apps and how many bots would have effected because of this 🤢😓

Joseph_Seddik

I would bet that none of support members had ever built a single app.
As for the impact, I’d advise to preemptively check all of your bots looking for steps with actions running on conditions. Probably start by bots with iterative group actions if any, these will certainly fail the whole bot.

Rifad

@devingu

@Joseph_Seddik wrote:

ever built a single app.

devingu

Unfortunately, this statement is not true, our support agents do build apps a lot. 😀 More than glad to look into your ticket if you can provide the ticket id. I will provide the feedback to agents to help them improve.

The issue has been escalated internally.

Joseph_Seddik

@devingu

Thanks for taking the time to respond.

I'd much appreciate your help to answer few questions, and before responding, please understand that your answers help developers acquire and maintain brand loyalty and confidence of end customers who are paying monthly subscription fees Google rather than some other competitor.

What can you share about the Change Management processes involving AppSheet backend?
What prevents such changes from being communicated in the Release Notes for example? What causes this complete silence.
Can we have a before, during and after announcement for such changes, in a post community for example, so that App developers are made aware and make sure any issue is reported immediately, giving an opportunity of adjustments or even a rollback?
What is the root cause of this issue? Was this an intended badly-communicated behavior change? or was it a bug that was accidentally introduced?
Is there a recommended workaround or course of action while the problem persists?
Is there a foreseen plan for resolution? or an ETA for such a plan?
Will there be an improvement plan to prevent similar problems from impacting customers in the future?

Would Google teams be more accommodating to understand the business impact of their actions and responses, or would Google Cloud developers have hard time convincing customers not to move to may-be-less-awesome technologies having more business-oriented organizations.

I tried to write this as openly and positively as I could relaying real customer concerns. We are not speaking about a bug, we are speaking about an ongoing business impact with complete lack of information.

Again I'd appreciate your help delivering the message and eventually answering my questions. Thank you!

Rifad

This is insane ! Not sure who is developing this kind of logic. I have a grouped action in automation. First action checks if Child Rows are blank or not. If not blank then it will trigger the second action.

Action 01

Action 02

Here Action 01 and Action 02 are just inverse and 99% of the time the Action 01 is FALSE.

If Action 01 is false it must trigger Action 02 but It just stops and fails as shown in automation bot error. That means 99% of the time my bot stops after checking first action inside grouped action.

These many bots are linked to above process but it fails to work on all 11-12 bots. That was working all fine for all these days.

Not sure what words I should put here to explain this kind of updates from GOOGLE. How to troubleshoot this is a question that I need to figure out and check each and every row and update it manually.

@devingu anyone from your team could address the potential effect of this issue ? Including me there are many developers here who is answerable to our 'x' number of clients with 'x' number of apps that is setup and working for months without any issues. There is no point in contacting support anymore because I do not agree to your point. I can explain you with proofs how pathetic support is these days. Please its struggle for days to troubleshoot all this kind of issues for developers like us.

SkrOYC

@Joseph_Seddik wrote:

Too bad AppSheet, just too bad !!

Welcome to the new AppSheet

Rifad

@Joseph_Seddik wrote:

Too bad AppSheet, just too bad !!

*GOOGLE

Joseph_Seddik

Right, though I’m a complete Google Cloud solutions proponent, but yes, right!

jyothis_m

I tried another method which is the removing the condition from Action and creating another branch on condition step in bot to check for this condition. Execute the action only if the result is yes.

Joseph_Seddik

Yes that would be the right course to take, but in many contexts, the condition values might nit be available at the bot level, and it doesn’t work when you want to make iterations, since you cannot call individual steps within a process.

Joseph_Seddik

@devingu well? should we expect any feedback from anyone? for how long we'd continue in this nightmare while being kept completely in the dark by all teams?!

Rifad

Still no updates. Every other bot that has a condition is throwing error. Not sure who makes this kind of updates that are 'stupid' without testing. Every other Grouped actions, Looping actions, Multiple actions in same bot stops in first step itself.

Joseph_Seddik

@Rifad Unfortunately they just don't care even about dropping us a simple message.

@devingu

Rifad and I have been trying to objectively describe to you the criticality of this bug, the enormous business impact it brings and our urgent need to communicate to our mutual customers.

Last time we heard from you was two days ago, when you were quick to dismiss my comment about the quality of support teams and you came smiling. I omitted your lack of concern and tried to be positive and collaborative explaining to you the impact our mutual customers are facing and listing a set of simple questions that we need to answer.

After two days without a single response, nor any news about this "internal escalation" it is very regretful to see how detached you and Google teams have been. Would it be worth asking you to get a manager here to respond? I don't think so.

What I think is that you and your support teams are just content with having built "apps a lot" that I clearly see you mean non-production sample apps without any customers behind, you won't even bother about communicating anything to AppSheet developers for the benefit of our mutual customers, we wouldn't have any problem, introduced by you, solved anytime soon, and we won't have any hope of getting any meaningful feedback let alone a resolution. I've been a Google Cloud Solutions developer for 6 years and have been in the IT industry since 1998, and have never seen such detachment and lack of concern. Now, very regretfully, I will never propose a Google product to any new project and will advise my existing customers to take every opportunity to move away from Google.

SkrOYC

@Joseph_Seddik wrote:

Rifad and I have been trying to objectively describe to you the criticality of this bug

I don't even try at this point

Roderick

Hey all, we're aware of the challenges this is causing and are working on a solution! Will update this thread once there is an update/explanation.

Joseph_Seddik

@Roderick I appreciate your reply and at the same time I'm telling you this is NOT good enough; "..once there is an update/explanation" is actually far from being useful in any way.

We need to communicate something NOW. Kindly understand this Roderick! For five days now we've been telling the customers we are awaiting Google's update. So, I'm asking you please:

Do whatever you can, call whomever you can call so that you get informed about the current situation and bring me please a meaningful message to communicate to anxious customers. Please get guidance in the questions included in my first response to @devingu
Please ensure we get DAILY updates that we can communicate.

Thanks in advance for your help.

zito

Hey Joseph,

Apologies for chiming in late - we are aware of the issue and are working on resolving it. We had/have a fix for the issue earlier this week but were blocked on rolling it out yesterday due to a separate and unrelated issue. We are prepping to roll out the fix today, which will happen over the coming hours.

I'm talking with our engineering and support leads about how we can avoid situations like this in the future. Certainly there was no intention to suggest that this was not important or business-impacting, and there was awareness internally that this was breaking customer apps and was important to resolve.

Where we can improve is obviously on the communications, which is one aspect we will be discussing internally. I'll follow up here or in a new thread (with a link from here) with what we land on.

At the same time, I do want to highlight that the best way to notify our engineering team that something is wrong is via support. We have folks who keep an eye out for issues on here, but we have a formal internal process for how support aggregates reports of issues and then escalates them to engineering. When communicating with support, please make sure that if your production apps are broken, that is made clear to help the support team properly prioritize and escalate. I have not looked at cases to see who did or didn't report an issue to support, so I'm not suggesting anyone did anything wrong, but simply asking that the clearer you can be with the impact to your business in your support cases, the easier it is for us to surface critical issues as easily as possible.

I truly apologize for the impact to your work. I know that situations like this don't inspire confidence in our care and investment in AppSheet, but we do care deeply about both the product and our customers. We clearly made mistakes in how this particular issue was handled and we'll work on fixing them for the future.

To answer your original questions to the best of my ability:

"What can you share about the Change Management processes involving AppSheet backend?"

We have a testing and release process. As code is committed, we have a series of tests that run to look at whether there are any obvious breaking issues. Then we run tests on the platform as a whole prior to releasing code. Today, we release on a daily basis. We also monitor for production issues that arise as a result of a given release. In this case, I suspect (but don't know for certain) that the percentage of automations that were broken because of this issue was small enough that it didn't trigger our automated monitoring. This is not to say that the issue wasn't critical for you, but simply that there are a lot of AppSheet users, many many more than there were as a startup, and so issues that affect a smaller percentage of users are tougher to tease out from looking at error rates or changes in behavior.

"What prevents such changes from being communicated in the Release Notes for example? What causes this complete silence. "

In this case, it wasn't a planned change, it was a bug. I'll investigate whether it might be possible to flag in the release notes if there are known critical bugs that are in-process of being resolved. It's a good idea.

"Can we have a before, during and after announcement for such changes, in a post community for example, so that App developers are made aware and make sure any issue is reported" immediately, giving an opportunity of adjustments or even a rollback? "

In this case it was a bug, and in general we try to notify the community as we roll out things that might change user experience or break the applications. It doesn't make sense to notify the community about all of the code that ships, and typically its an unintended side effect when something breaks due to a planned change (as opposed to a bug being introduced unintentionally).

"What is the root cause of this issue? Was this an intended badly-communicated behavior change? or was it a bug that was accidentally introduced?"

Bug

"Is there a recommended workaround or course of action while the problem persists?"

I don't know, tbh, but I'm hopeful that now that the fix is rolling out a workaround won't be necessary.

"Is there a foreseen plan for resolution? or an ETA for such a plan? "

As stated earlier, fix is rolling out today.

"Will there be an improvement plan to prevent similar problems from impacting customers in the future? "

I've already started talking to the team internally to see what we can do to identify breaking issues faster, escalate them internally faster, and track their resolution.

Joseph_Seddik

@zito Thanks a million!! I’m driving to meet an affected customer and luckily I stopped the car to read your reply. Thanks for the detailed and very informative response that is good even to communicate as is. You can’t imagine how helpful it is, this changes a lot !

All your advice is heeded. Thanks again 🌷

SkrOYC

@zito wrote:

In this case, it wasn't a planned change, it was a bug. I'll investigate whether it might be possible to flag in the release notes if there are known critical bugs that are in-process of being resolved. It's a good idea.

Hi @zito, just FYI, we have been asking for this for a long time, just to tag some people that has: @Koichi_Tsuji @1minManager @dbaum @MultiTech @WillowMobileSys and more

zito

Had some chats with folks - I think its doable, we have to figure out a good process that is reliable, because it all falls apart if something gets missed. Give me a bit of time.

Rifad

@SkrOYC wrote:

just to tag some people that has: @Koichi_Tsuji @1minManager @dbaum @MultiTech @WillowMobileSys and more

I would like request Google something important. Everyone mentioned here including people in this thread we need some kind of support that connects us DIRECTLY to expert developer instead of usual AppSheet support. We all done contacting support. I contact AppSheet only in case if i find BUGs. Not to ask usual questions.
Everyday I spend hours explaining support simple stuffs. And they come up with stupid answers. If i say app is stopped because of a Bug they send a copy paste message.

And many more closed chats without proper resolution. Never in past many years of AppSheet experience I have contacted support unless the server is down or serious problem.

Joseph_Seddik

@zito Would you please signal to us once the fix is rolled out so we can start testing, and rolling back the implemented workarounds? Thank you!

zito

I'll try, but I'm on east coast time, so I might be away from my keyboard by the time its rolled out. I will ask someone on the team to reply here if possible.

Koichi_Tsuji

@Rifad wrote:

I contact AppSheet only in case if i find BUGs

ME TOO.

zito

Hey folks, I believe the rollout has successfully been deployed - please check and let me/us know if there are remaining issues.

Rifad

Yes its working now. Your silence for a week have impacted many business. Thanks for the update.

Joseph_Seddik

@zito I launched a bot I had prepared for the test and it ran quite smoothly. Thanks much!!!

It is almost 5am in Madrid, I've been anxiously awaiting the rollout 🙂 This had a direct impact on payment collection for two of my major clients. Tomorrow I will reverse my workarounds and tell you should there be any issue.

But for now all seems OK. Thanks again, and for updating us during West Cost hours 🙂 appreciated!

Rifad

@Rifad wrote:

Just to put in perspective. Just single app that stopped working has at-least these many actions linked to it. And many apps like this have impacted. @zito @devingu

Steve

@zito wrote:

I truly apologize for the impact to your work. I know that situations like this don't inspire confidence in our care and investment in AppSheet, but we do care deeply about both the product and our customers.

Actions speak louder than words. Google's actions have contradicted your words for a very, very long time.

@zito wrote:

We have a testing and release process. As code is committed, we have a series of tests that run to look at whether there are any obvious breaking issues.

You have no QA process. "Testing" is smoke tests and unit tests. No end-to-end tests, no integration testing, no real-world testing. It's like building a bridge, noting that it "looks" ready and doesn't collapse when you walk across it, declaring it ready for rush-hour traffic even during an earthquake, then learning that it fell the very first day under ordinary midday traffic only when you happen to see it on the news.

Koichi_Tsuji

@Steve

I echoe you.

Automation critical problem and workaround - Bots failing on action's condition