Hi everyone, we wanted to give you an update on the status of this recurring issue with images not appearing in your reports.
To begin with, there are two pieces of good news. The first, is that as of May 13th we believe weโve resolved the last of the major causes here. Secondly, we now have much better logging in place so that any future problems should be much quicker to identify and address.
Most of you know that weโve been trying to transition away from a service called HTML 2 PDF Rocket as part of our ongoing transition to running on google services. Initially we believed this was likely the cause of the problems, but we had some reports of images missing before we started to rollout this change, and also reports of missing images even after an account was switched back to HTML 2 PDF Rocket.
We identified two major root causes of why images would not appear.
The first affected both the old and new rendering engines, and was because of a 5 second timeout in our CDN that handles resizing of images for us. We worked with them to increase the timeout which we saw drastically reduce the number of 504 errors. We also dug further, and have found 6 different causes for why images were taking so long to load that our CDN was timing out. Fixing these will reduce the chance of timeouts when generating reports, and should result in quicker image loads in apps. Two of these specifically related to using the Google Drive storage provider have already been fixed, and weโve seen a positive impact from those changes.
The second issue was a result of our transition away from HTML 2 PDF Rocket. In essence the new rendering engine was being throttled by an internal service when trying to fetch images from AppSheet. This behavior was designed to prevent abuse, and would start dropping requests if there were too many for the same host already queued up. This is why weโd see reports of this happening in batches, and why people with a large number of images were more likely to have images missing. Once we identified this and bumped the timeout of our CDN, we temporarily switched some accounts back to HTML 2 PDF Rocket until we could get a fix out for this.
This was resolved on May 13th when we rolled out a change to use a similar service for fetching resources specifically from our own host (appsheet.com). Since then we havenโt seen any logs indicating that requests were being dropped because there were too many queued up.
Now that the issue has been resolved, weโve moved back most of the accounts that we temporarily switched to HTML 2 PDF Rocket over the past couple weeks after we were confident the last problem was specific to the new service. There are still a handful of accounts which were moved back early on that over the next few weeks weโll be switching back to the new rendering engine as we complete our transition away from HTML 2 PDF Rocket.
We believe these two problems were the cause of the majority of cases where images would not appear in PDF reports. If youโre still experiencing this problem, please reach out to support so we can investigate further.
Thanks,
Greg
LinkedIn
Twitter
