Recent timeouts and errors in known-good Bot processes

[Jamie]

I am suddenly experience several issues with automation. Bots that previously completed successfully are experiencing long processing times and timeouts. Some of these indicate a timeout even though the reported processing time is less than 10 seconds, while other are timing out at 120+ seconds (which seems to be by design but I never had this previously and no changes have been made). In other cases, I have multiple Bot process runs which show "Pending" with the reason being "SUSPENDED_ON_CONDITION". I don't use any timed delays in any of my automation.

I am already in contact with support, but is anyone else experiencing a major change in Bot performance or sudden errors with previously working code?

[mcopp0315]

YES!  I have been struggling for a full week now with random confirmation emails missing after form submissions.  Months of flawless bot execution for 300+ submissions per week.   I have received minimal assistance, guidance, or communication from the helpdesk.  I just now noticed the "PENDING" status of nearly 50 automations and jumped over here to search for "REASON: SUSPENDED_ON_CONDITION".   I am beyond confused and entirely frustrated!

[mcopp0315]

Hey Jamie, I admittedly don't quite know how this forum works, but I'm afraid that since you marked my reply as "SOLVED" that we may not get the attention to this issue that we need...

[Jamie]

I don't know what is going on with the forum, but I did not accept either of these replies as solutions and my issue is definitely *not* resolved. C'mon Google...

[Steve]

@Michelle ^^

[Jamie]

Thanks, @Steve but my issue is not fully resolved, so I have unmarked that post as the solution. As I mentioned to @mcopp0315 I believe there are still bot processes being marked with incorrect statuses and timeouts occurring on processes that previously completed well within the 120 second timeout. Also, I just reviewed all the logs again and there is still one process stating SUSPENDED_ON_CONDITION. Once I am convinced that all of these issues are fully resolved or someone has explained them to my reasonable satisfaction, I will mark the issue resolved. Hopefully I will hear something back from support soon on these - it is approaching 24 hours since last contact.

[Zhifeng_Lin]

Yes the issue  is related to the monitoring app itself. We had an issue regarding the display of the "status"/"state" field. A fix has been deployed to prod last Friday, but for bots that were executed during the breakage, their status/state have been incorrectly marked.

We put in another workaround in the monitoring app itself to account for the "incorrect" status/state based on the rough timestamp (2/23 to 2/25). So it's possible that we missed some of those.

You can also verify that if the final step is the "return" step (and no "Error Message" is displayed), it means the bot has been completed

Thanks!

[Jamie]

@mcopp0315  I have learned from support that there was an issue in the Bot monitoring App which is/was causing inaccurate status reporting. See below:

"The issue is related to the monitoring app itself. We had an issue regarding the display of the "status"/"state" field (a new enum was incorrectly introduced, hence "suspended_on_condition" actually meant "completed") I believe the issue was introduced on UTC time 2/24 to 2/25, and has been fixed in prod now."

My pending processes are now all reporting completed, please check again and see if yours are fixed. I am still working with support to get more information, because I continue to have issues with what I believe are erroneous errors / timeouts, so I think other statuses may also be affected. I'll post back any news when I have it.