Audit Logs Reliability

Hello Everyone,

How Reliable are the audit logs?
Is anyone using Audit logs on regular basis to understand and validate issues faced by users.

In my case, we have over 2000 users that use the app on Daily basis. Naturally with these many users, they are bound to face issues. Sometimes the users report that they are not able to sync the app or sometimes they say that they synced the app but the data disappeared after syncing.
We use Audit logs to validate such cases and understand why the issue happened.

Now the question is How reliable are these audit logs? I have come across lots of cases where there was inconsistency between what the audit log says and the data we have.
For ex :
1) In the audit log, an entry can see it was a success but when we check in the DB, no row can be found. The audit log contains all the details of the row entered by user, but nothing is there in DB.
I have reported this dozens of time to appsheet but there is no clear answer to what is happening.
2) Sometimes the user says he synced the app and has screenshots to prove, but no record can be found in audit logs.
3) Sometimes the timestamps are all wrong and doesn't make sense.

I want to understand if we can rely on the audit logs for validation of such issues? Or is there any alternative way we can do this?

Thanks for reading this so far.
Hoping to get some clarification regarding this.

@Aleksi 
@Koichi_Tsuji 
@Mike_Procopio 
@Joseph_Seddik 
Tagging you guys since you have lots of experience and maybe you can share some insights from your experience.

4 12 701
12 REPLIES 12

@YogeshLaddha 

Take this as one of opinion among others. We export Audit log data to BQ using the AppSheet function which is avaible after Enterprise, and we create AppSheet app, connecting this BQ to monitor the health of our app daily.  Furthermore, I m only interested in, generally, the app(s) are throwing erros or not. There are bunch of the different type of errors, Audit log is saving, but for now I m just focusing on one errors, whch is the most critical, i.e. the user need to reset the app. (due to inconsistacy of the version of the app etc)

We have Automation to let us know if or not such a critical errors are happening or not.

However, the other errors we see out of the audit log is not telling us no-meaning lectures.    I know there is errors, but we know it is critical or not? Most importantly, with such errors the use can still use the app or do we need to ask them to reset the app or ask them to taka any particuar action to stay working wiith apps.

Errrors are thrown all the time, but the problem is there is no guidance to remedy such errors. And there is no indication if or not the error is minor (not affecting the usage of the app by users) or the app failed to update data change or not.

All in all, autit log is not really useful. Only it is useful we know somethig "wrong' is happening to our app but we are not sure how bad they are.

 

Few days ago, we received bunch of errors, out of one prod app. The same errors are throwng, during that time, saying, "Error:Object reference not set to an instance of an object."

This is frequently often seeing, but we are not sure if or not the app user are facing the issue, either able / unable to keep using the apps, or the edited data is safely saved to the data sources 

Such error messages are no friently, but just give us (admin) concerns over the apps, without any guideline how the error is critial or not. Make the story worse, we are not being advised by such error, if or not we admin need to take corrective actions, or way to fix those problem. Audit log is currently just shoren our life time due to "fear".

Im not sure if or not I m answering to your query, but this is my factual story with me.

 

I also come across this same error often mostly while running automations. It just randomly happens ( I am sure there must be some reason but I have no clue) and I have till now not able to understand why this happens and what needs to be done to fix this.

I completely agree that the error messages should be a bit more explanatory and give us reasons why this is happening and some possible ways to resolve this. This would be extremely helpful if implemented.

Audit logs should be "realiable" as it is simply logs whatsover the issue are detected. however, it never be friendly to the admin of the apps, as we do not know why such error possbily happend and a way to fix that. For now, for me, I understand Audig log is just give us a fear alone. no more than that. Cheers.

Exactly. Ideally they should be reliable as it is simply logging whatever syncs and operations are happening in the app. But sadly it is not. I have come across lots of cases where this is not the case. For ex : Sometimes I cannot find any audit log for data added by users but the data is there in DB.

On a related note, I find the logs extremely difficult to navigate and interpret. Regardless of whether they're reliable, their usability needs to be significantly improved. Failing that, support for understanding them via help articles or webinars would be valuable.

My daily life. In the morning, open the email inbox. Check the errors. (auto forwarded to me from all the appsheet apps)

Go to monitoring app to check errrors accross the apps we have.

There are erros we are not sure why they are comin from.

After that, we are at a loss, as we do not konw how such erros are serious or not sure we need to take action.

Then we are at a loss in the every morning, as we are not sure what to do with such repeated errors.

 

@YogeshLaddha 

Hope my experiences will help you, but sadly will not solve anything at all. Annoying.

@Koichi_Tsuji Thank you for your replies. I appreciate you sharing your insights.

We also do take a backup of the audit logs of only the errors on daily basis and have people looking into that to actively resolve users issue before they report to us.
It has been helpful in that as we can identify the issues quickly and resolve them so that users can use the app and there is very little downtime. Most of this are sync errors, again because of a major appsheet bug which sometimes creates syncs the same row twice and create two rows even though user only added it once. And since we have validations in our SQL DB to prevent duplicate data, users get sync errors because their app can not sync the duplicate row that was created by appsheet.

For me, Audit log has been useful when needing to validate whether an issue reported by user is valid or not as it is tied to lots of things like their performance, expenses etc. So it is very important for us to validate this issues and there is no other way to do this other then looking into the audit logs.
I have become quite familiar with the audit log now so navigating through it is not a big issue for me.

The only major issue I have is that sometimes the data displayed in audit log is incorrect or missing or doesn't make sense.
To be able to validate things in our apps, I need to be confident that the audit logs are reliable and we can trust the data it presents.

I am not sure how the data can be inconsistent as it is simply logging the operations and syncs that are happening in the app.
I hope we can get some insights on this from someone in appsheet team as well so that we can understand whats happening and what to expect.

I am experiencing the same issue of users losing their data after sync, and it is getting very annoying. Currently, I am reviewing 25 physical folders because their inspection is missing from the app. The images and signatures are saved, but the data is not.

 

Thanks you for this unexpected confidence !! First, I don't have a single app with 2000 users, so my cases can be different than yours. But FWIW, here's my journey trying to figure out what is happening with users and how to avoid getting into problems from the first place.

At the beginning I tried hard to decipher the audit logs to gain more insight about what could be causing those intermittent, inconsistent, problems users were reporting. Eventually reaching a dead-end, I came to enlightenment ๐Ÿ™‚ 

I figured out that more important than understanding the nonsensical logs (and support feedback), was to understand user behavior that leads to the manifestation of these problems. So in a moment of post-frustration determination I decided to build my own logging mechanism. Not only adds/updates/deletes, but I replaced every action in the app, even clicking on a view, clicking on a record to navigate to details view, viewing a reference record, everything, with a preceding step of logging of the interaction, username, timestamp, location and any other relevant info I would need. This logging I could turn ON or OFF with a single parameter for the whole app.

I might not have been successful in understanding why AppSheet was not behaving the way it should, but with this logging I came to know exactly when a particular problem would appear and was even able to reproduce it. I adopted my design practices to avoid falling victims of such bugs/shortcoming, was able to solve the problem for the customer, and had more time to argue with support till they acknowledge a bug or a need of improvement. 

Anyway, here are the lessons learnt that translated into practices:

  1. Bot triggers are un reliable. I have only one type of trigger, that is an increment of a specific counter. If I want to trigger based on addition of a row, I increment a certain counter, if I want to trigger based on a deletion, then I increment another. This is the trigger that I found the most reliable of all, so I only use it. 
      
  2. As can be inferred, from the first point, I don't allow users to directly delete rows for examples, I replace the delete action with another that would increment a counter and leave a bot delete the row.
      
  3. Also for edits, if multiple users are susceptible to edit the same row, I never allow them to edit the row directly, instead, I built a change-request model to register individual user changes, then manage these requests into edits through a bot.
       
  4. I now never, ever, launch multi-step actions from the user's device (like a form save action for example). Instead, again, I'll let the user increment a certain hidden counter, then let the bot do the work.  
       
  5. For certain customers I have in Africa and Latin America, due to unstable mobile network, I try to keep the sync requests as small as possible, say instead of having to do one sync operation to update 10 rows, I divide it so that we have 10 syncs each for one row. 
       
  6. I have a lot of controls to prevent simultaneous editing. For example, suppose that I want to update a customer account and transaction history following a new purchase, and for this I have to do a list of say 10 changes. I make sure no one is able to do any modification of any record related to this customer until the registering of these 10 changes in the database is complete. So in my bot that is triggered by a counter, I add a step in the beginning to set a syncFlag in the customer's row to False, then the bot execute all the steps, then at the end I have the same flag reset to true by the bot. All records referencing this customer in all other tables cannot be edited while the flag is set to false, all action buttons that would cause an edit to a record related to this customer will only be shown to users upon the condition of [customerKey].[syncFlag]. 
       
  7. Without writing the whole list, I don't rely on what the app should do, I rather explicitly do what it should. For example, I don't rely on "is part of', my bot explicitly deletes related referencing records. I've manually added explicit spacing between the items of this numbered list ๐Ÿ™‚

This is what I could remember, I will add more points of I remember more. I hope this helps.

 

Top Labels in this Space