CSV UTF-8 format

[Ami]

Hello
If i want a report to produce a csv file, can i make that file in a csv utf-8 format as my data is in hebrew?

[Aleksi]

@Phil Any thoughts?

[raketa]

Hi
I need the following encodings: UTF-8. I use Russian in my applications.

[Phil]

We are UTF8 encoding the SaveFile file contents.

[Ami]

Thanks @Aleksi and @Phil

[dodo]

Hello,
When using action button to export csv file with hebrew characters, the file opens in excel with gibberish characters.

Any idea why ?

Thanks

[Koichi_Tsuji]

Generally this is issue when we export file as CSV, not only issue with Appsheet particularly.

Worth reading another post with the same issue.

Japanese characters in attachment are garbling Questions

@Kazu I think Koichi-san suggestion is the AppSheet Way too. However, when I was checking the behavior, I came across something I’d like to share. If the text file is UTF-8 with BOM, it will be displayed. If you are interested, please give it a try. [2021-02-21_18h13_59] FYI @tsuji_koichi

[Kirk_Masden]

One of the reasons that I stopped using Excel and moved entirely to Google Sheets is that Google Sheets never seems to be confused about the intended code when importing CSV. It’s quite easy to import a CSV file into Google Sheets.

[Aleksi]

You’re welcome

[Phil]

I am investigating the issue.
We seem to be exporting the CSV file in UTF-8 format correctly.
The problem seems to occur if the exported file is missing the Byte Order Mark (BOM) at the beginning.
I am trying to understand if this is the actual problem and how it arose.

[Phil]

Hi Dodo,

The recommended way to open a CSV file using Excel when the file contains anything other than ASCII text is to:

1. Open Excel
2. Select the “Data” tab
3. Click “From Text”
4. Select the CSV file
5. Click the “Import” button
6. Ensure “Delimited” is selected
7. Check “My data has headers”
8. Click the “Next” button
9. Under “Delimiters” select “Comma”
10. Click the “Next” button
11. The “Data preview” pane should display the imported data properly.
12. Click the “Finish” button.

When I did this, the CSV containing Hebrew text was imported into Excel correctly.

When you try to open the CSV file by double clicking it, Excel uses default options but those
options may not import the data properly.

[Phil]

After spending three days on this issue, I discovered the following.

Older versions of Excel need a Byte Order Mark (BOM) at the start of the CSV file to properly detect UTF-8 file content.
Newer versions of Excel, since Office 365, properly detect UTF-8 file content even without a BOM.

When a CSV file is returned to the browser by CSV Export, the browser strips out the first BOM to conform to the UTF-8 decode algorithm as described in this article curl - Preserve UTF-8 BOM in Browser Downloads - Stack Overflow

To circumvent this, we need to return a CSV file starting with two BOMs.
That leaves a single BOM in the CSV file even after the browser strips the first BOM.
That enables Excel to properly detect the CSV UTF-8 file content.

If testing goes well, I will be releasing a change on Wednesday or Thursday of this week that includes a second BOM in the exported CSV file to circumvent this Excel limitation.

In the interim, you can use the Excel “Import” feature I described in the previous article to import the CSV file.

[dodo]

Thank you very much Phil.

I must say that Before I posted, I knew it’s possible to open the file via excel as you described, and I wondered (with relate to your answer from January 2020), that If the system encodes utf-8 SaveFile file contents, why it is not so with action button ?

It would be great if this issue will be solved.

Also wanted to ask about the possebillity to export other formats by action button.

With workflow one can choose to create csv file, xslx, pdf…

There is no action button to export those formats.

It is possible to share the sheet with the users, so they can download the data in which format they choose, but that’s not the point.

Maybe I’m missing something, but why can’t the app allow downloading the file, just as it is possible from the sheet itself ?

Thanks again

[takuya_miyai]

hi @Phil

Great job!

I also checked it in Japanese and it is working fine.
AppSheet is now one step closer to being used around the world!

@tsuji_koichi

[Koichi_Tsuji]

yes, this is definitely great jobs, thanks Phil - sama.
Will help our app users (locally here) where the app languages are not English.

[dodo]

Hi Phil,

Tried it out just now.

Works great.

Thank you very much