PDF generator update brakes the PDF output

[Fabian_Weller]

AppSheet actually uses the Skia/PDF m111 to generate PDF.
Sometimes when they update the PDF generator, our PDF files will be broken: Wrong pagination, wrong text size, ... > The output looks bad and unprofessional. 

In some cases, AppSheet is able to fix this. In other cases there is no fix. We need to update all our templates 😳
Here is the story of what I was aware (and affected) of:

• Feb 2021 
  Update from Qt 4.8.6 to Qt 4.8.7.
  Change in the font size and column width.
  No fix from AppSheet. All Templates needed to be updated.
• Mar 2021 Update from Qt 4.8.7 to Skia/PDF m88. 
  Change in the font size and column width.
  No fix from AppSheet. All Templates needed to be updated.
• Dec 2021 Update from Skia/PDF m95 to Skia/PDF m96.
  Change in the row height.
  No fix from AppSheet. All Templates needed to be updated by reducing the line spacing.
  
• Dec 2022 Update from Skia/PDF m107 to Skia/PDF m108.
  Change in the pagination.
  Fix from AppSheet 5 days later.
• Mar 2023 Update from Skia/PDF m110 to Skia/PDF m111.
  Change in the text alignment in columns
  Fix from AppSheet more than 2 weeks later by reverting to the previous version Skia/PDF m110.
• July 2023 Update from Skia/PDF m110 to Skia/PDF m114.
  Change in the row height.
  No fix from AppSheet. All Templates needed to be updated.
  

The goal is to find a way how we can prevent this in the future. Maybe by an announcement? 

@Steve @Aleksi @Arthur_Rallu @lizlynch @Greg_Denton @AleksiAlkio @devingu @Roderick @zito 

[jyothis_m]

@Fabian_Weller you are absolutely correct. Appsheet team needs to do something about this. Template correction everytime is not a feasible solution. I am having numerous templates and editing each everytime is a tedious task. Something should be done in the server side itself when these updates are rolled out. Do you have any idea who is handling this in the Appsheet team. 

[jyothis_m]

@Fabian_Weller How did you manage to get these details of Skia/PDF like the latest m111 i am only able to see m98 as the last milestone in Milestone Release Notes in their webpage 

[Fabian_Weller]

In Windows there are 2 ways to see the PDF generator:

• Right click on the PDF file and select PROPERTIES. Under DETAILS you can see the "Producer"
• In Adobe Reader click on FILE > PROPERTIES. Here you can see "PDF created with"

---

@jyothis_m wrote:

Do you have any idea who is handling this in the Appsheet team. 

---

Sadly I don't know. That's why I added some who had to do with it in the past.

[jyothis_m]

Found it. Thank you 😊 @Fabian_Weller 

[zito]

Hi folks,

I wasn't able to follow up on this until today - I did some digging, it turns out that we don't update the Skia library directly.  We call a Google-internal service that generates the PDFs using headless chrome, and those folks are updating Chrome as new versions are available and that updates the corresponding Skia library.

That puts this a bit out of our hands, but I've asked our team to look into whether we can pin to a particular chrome version, or potentially keep ourselves a version back, so at least there is time to test if the newer Skia introduces issues.

I'll follow up here as I have additional information.  

Thanks,

Matt

[Fabian_Weller]

Thank you very much @zito for your help. Will you be able to fix the issue that came with the Update from Skia/PDF m110 to Skia/PDF m111? Oor will we have to update our templates?

[zito]

We don't have any control over what PDF library is being used by the rendering service - we'll look at whether there's a way we can request a particular version, but even if that's possible, it's not going to be something we can do immediately.  

I'm a little surprised just updating the version of skia broke your templates.  I looked at the changelog from m110 to m111, and I don't see anything that obviously looks like it would affect the rendering.  I wonder if there might have been a change in the chrome rendering engine, and that is cascading down into the PDF generation?

Obviously, you know your templates better than I do, but from previous experience with PDF generation from other projects, complex CSS tends to be where things break across different PDF libraries.  Could you potentially look at streamlining the CSS/HTML to reduce the complexity?  That would likely help future-proof the templates.

Apologies, I know this is inconvenient, its just largely out of our hands.  There's another approach to consider as well, which is to use Apps Script to generate a Google Doc, and then use the Drive API to export that as PDF.  The advantage there is that the docs formatting consistently renders in PDF, and the engineering team at Google that owns that system takes backwards compatibility very seriously.   I realize that's additional effort in the short-term, but suggesting it as another option. 

[neemias]

Please @zito, we need a fast resolution... I use appsheet as a professional solution... i caneta wait 2 or 3 weeks every time that 
Please @zito, we need a quick resolution... most of us are using the appsheet as a professional solution... last time it reverted after 2 or 3 weeks... this causes a lot of inconvenience.

[SkrOYC]

Just FYI, my HTML templates have not been affected

[SkrOYC]

---

@zito wrote:

I looked at the changelog from m110 to m111

---

https://skia.googlesource.com/skia/+log/chrome/m111

Maybe this?

[zito]

That's what I skimmed briefly - it looked mostly like various library fixes, not rendering changes.  Not my area of expertise, of course, so I easily could have missed something.  

[neemias]

The Appsheet team needs to do something about this. I have four professional apps affected. I'm a user and I pay for it. If they cause any problems, the least I expect is a quick fix.

[Craig_Clancy1]

The suggested 'fix' by Zito might be good for hard coders, but I would think for newbies, this arrangement would be out of the question. Isn't Appsheet wanting to deliver for citizen developers? No-Code??

I would suggest that most companies using Appsheet heavily rely on a decent output for compliance etc and for this core feature to be out of Appsheets hands is quite extraordinary. When this issues first started last year, I had over 200 apps effected. I picked it up on my Friday morning (Sth hemisphere) and it took me 3 to  4 days to get anyone in Appsheet to recognise the issue (unacceptable). Thankfully finally someone had the smarts to role the Appsheet update back, but by that time, I was swamped with VERY sceptical clients on our ability to deliver a decent app product.

Surely common sense and logic must prevail here and Appsheet take better control of this issue rather than poking at suggested fixes. Pagination of documents has always been cumbersome. 

I have noticed a couple of pdf's irregular formatting as of yesterday, and I am hoping and praying that it does not spread throughout my several Appsheet accounts adding to the BOT issues still being experienced around the globe..

It appears like there is becoming a large disconnect between devs / users with valuable feedback and Appsheet management. 

[zito]

---

@Craig_Clancy1 wrote:

The suggested 'fix' by Zito might be good for hard coders, but I would think for newbies, this arrangement would be out of the question. Isn't Appsheet wanting to deliver for citizen developers? No-Code??

---

Apologies, I was just thinking of potential ways to future-proof the documents.   I obviously recognize that is not the most no-code solution, but I also think that asking people to write CSS to generate documents isn't very "no-code", so it didn't seem completely out of line.

---

@Craig_Clancy1 wrote:

Thankfully finally someone had the smarts to role the Appsheet update back, but by that time, I was swamped with VERY sceptical clients on our ability to deliver a decent app product.

---

That was also a PDF rendering issue?  Do you have a case number I can reference?

---

@Craig_Clancy1 wrote:

adding to the BOT issues still being experienced around the globe..

---

Are you still experiencing bot issues?

[Craig_Clancy1]

Hi Zito and thanks for your response.

Current case number for output issues is 7-5531000034120.

Given the signature and image issue yesterday 2-1382000033755 I have not had time to revisit Bot issues.

I appreciate you looking into my current case number and hope you can escalate the issue to quickly resolve. 

[SkrOYC]

Maybe this problems are not directly related to Skia and instead is the backend for GDocs/MSWord to HTML/CSS that is messing things up, because I use HTML/CSS directly and haven't experienced these problems.

@zito Previously I was in talks with @Phil to solve this kind of issues. It seems he is no longer in the team. Could you point us to the new expert on this area? Thanks!

[zito]

A quick update, we've been able to work with the rendering team to identify the change that caused the change in the rendering behavior.  As of now, that's being treated as a bug by that team, and we are going to switch to use previous version of the renderer, which should resolve the issue for now.

From here, we'll work with the partner team to reproduce the issue and figure out the path forward.  If its a bug, it will get fixed, but there is the possibility that the other team will determine that the old behavior was incorrect and that the new behavior is "more accurate" to the specification (just to manage expectations).

For whatever its worth, the issue that you noted in December and this issue are essentially the same issue, and is related to the print engine shifting to the same layout engine that powers chrome web rendering.  Once we resolve this issue, I don't anticipate further rendering changes in the near term.

[Craig_Clancy1]

Thank you for the feedback Zito. I've heard crickets from support.

I am concerned regards your statement "...that the new behavior is "more accurate" to the specification (just to manage expectations)."

So you are saying if my approx 200 templates are currently unformatted with the latest fix, that if its determined that the new system is a better and more permanent fix, then I will have to edit all my templates that remain affected by the new fix?

I am happy that Appsheet are taking this issue seriously as it has been occurring since 2021.

Thank you for your findings Zito. Very appreciated.

[zito]

---

---

---

@Craig_Clancy1 wrote:

So youare saying if my approx 200 templates are currently unformatted with the latest fix,  that if its determined that the new system is a better and more permanent fix, then I will have to edit all my templates that remain affected by the new fix?

---

Obviously, that is not what we want to have happen, and I have no reason to believe that is what will happen - and we would push very hard on any suggestion that this doesn't need resolving.  I simply don't want to say confidently "this is 100% going to get fixed" when the fix is outside of my team's control.

[Craig_Clancy1]

Fingers crossed then ay.

What is your team Zito?

[zito]

I'm the head of product for AppSheet, so the AppSheet team is my team - the PDF generation is happening within a shared service inside of Google that does this, so we are their customer to some degree.  

[Craig_Clancy1]

Zito, thank you very much for stepping in and involving yourself at this micro level. I assume you would be a very busy person. 

The format issue seems to have corrected at this stage. I hope the permanent fix works. 

Many Thanks

[Steve]

---

@Craig_Clancy1 wrote:

It appears like there is becoming a large disconnect between devs / users with valuable feedback and Appsheet management.

---

This has been the case since Google acquired AppSheet.

[zito]

Quick update on this - the teams have been reviewing the issue, and its been determined its actually a regression in Chromium/Chrome that's been around for a few years, but wasn't showing up in printing because printing was using a different rendering engine.  When the print engine was updated to use the same layout system as display, the bug became visible.  

Since Chromium is open source, we have a public bug tracker, and you can follow along here if you are interested:

https://bugs.chromium.org/p/chromium/issues/detail?id=1430449

I don't actually know what the process is for fixing and how this is rolled out when its a Chrome bug, this is a first for me, but we'll keep an eye on things.  

[Fabian_Weller]

Thank you @zito I can confirm that the problem is fixed for now. As I see you switched to the previous version: From Skia/PDF m111 back to Skia/PDF m110. I updated the OP.

[Fabian_Weller]

@zito as we read in the release notes, you updated again from Skia/PDF m110 to Skia/PDF m114.

It says that it "contains a bug fix for the issue with colspan". But this caused our PDFs to be broken again. Again the row height is bigger. It seems to be the same thing as in Dec 2021.

---

@Fabian_Weller wrote:

Dec 2021 Update from Skia/PDF m95 to Skia/PDF m96.
Change in the row height.

---

Will you be able to bring a fast fix, or will we have to adjust again all our templates?

As I wrote in the OP: The goal is to find a way how we can prevent this in the future.

[jyothis_m]

@Fabian_Weller @neemias what templates are you using. Google doc or html

[neemias]

Hi,

I'm using both. Html and Gdocs.



--

[Fabian_Weller]

@jyothis_m I use Google Docs.

[Joseph_Seddik]

If it could help anyone, I use Google Docs templates and haven't had any impact due to these rendering problems.

[Fabian_Weller]

@Joseph_Seddik Do you use tables in your Google Docs templates? I think you see the change only when you have tables, because the cell height got a bit bigger.

[Joseph_Seddik]

@Fabian_Weller Yes I do. Here's an example:

And here's the result, the cells width are maintained and the hight changes and fits exactly the cell's content:

I can share with you the empty template if you'd like.

[Fabian_Weller]

@Joseph_Seddik @jyothis_m OK I see. Maybe it's because of the fact I use a lot "Table in Table"? I mean I have a one-cell-table and inside this I have the "real" table. I use this method to control the formatting.

[Joseph_Seddik]

@Fabian_Weller The second small table at the bottom is actually a table within a single cell. But it is only one not "a lot", so maybe..

[Greg_Denton · Accepted solution]

This is excellent back and forth, thanks @Joseph_Seddik for helping out and offering to share your template.

@Fabian_Weller I did get your files from support and took a look. Unfortunately I don't think there's a bug here, and it's impossible to guarantee pixel perfect fidelity with the transition from Doc -> HTML -> Replace Variables -> PDF.
It seems like this was a result of enabling LayoutNG Printing in Chromium, and agree with the discussion above that it was potentially exacerbated by intricately nested tables, where even a 1 pixel difference in height, margin, padding, or border would then compound enough to cause a page break that wasn't happening before.

The good news is that with this enabled, we're now caught up with the latest versions of Chromium browsers, and major changes to table layout like the above are at least in my experience fairly uncommon.

Then as a general side note: I never got to mention before, but I still think it's awesome that the AppSheet community with their PDF templates managed to uncover the colspan bug all the way upstream in Chromium, improving browsers for everyone!

[Joseph_Seddik]

---

@Greg_Denton wrote:

Then as a general side note: I never got to mention before, but I still think it's awesome that the AppSheet community with their PDF templates managed to uncover the colspan bug all the way upstream in Chromium, improving browsers for everyone!

---

Kodus to @Fabian_Weller and the wonderful AppSheet community! Regardless of its scope, this is the best technical community ever 🙂 

[Fabian_Weller]

Thank you @Greg_Denton for checking. So we will update all our templates and maybe make them more "tolerant" for future changes. Because I don't believe this will ne the last time. 🙃

[jyothis_m]

I am also using google doc template with tables in it. Presently everything seems fine. Update didn't spoil the Row height of the tables.

[SkrOYC]

As a side note, I recently saw that some of my html bots that create html files with html templates are not being generated due to "the request took too long to process". So these issues could also be related to not just the Skia engine but some other stuff that we aren't aware of