I need a solution for linking images

We have thousands of images stored in our server, I can create a VB routine to export certain image folders to an online cloud storage at intervals throughout the day so the numbers would reduce.

I need to think outside the box and find a way for Appsheet to access the folders, has anyone been down this path before and lend some advice or recommendation on how to start, tips - tricks - anything??

Kind Regards

0 66 4,579
66 REPLIES 66

Hi @Dave_Willett,

There are lots of things to consider here. Could you give us some more info on your data and requirements, please?

  • Are all the images related to the same table?
  • Are they currently in a nested folder structure, or do you want to organize them that way going forward?
  • Are the images already named with the Parent-Recordโ€™s ID, or their own rowโ€™s ID?
  • Do the images need to be accessible outside the app, with public URLs?

Hereโ€™s one option that may work for you:


Hi. Why do I keep getting the following: I have followed the instructions to the letter?

Hmmโ€ฆ I havenโ€™t seen that error before. I may have only gotten this to work because I already had the Google Drive API enabled on my account.

Try visiting the link suggested in the error. You might have to create a new project. Iโ€™ll test this from another account later this week and see if I can get it to work without the API enabled.

This looks amazingโ€ฆ

  • Are all the images related to the same table?
    The table isnโ€™t created as of yet, the images are in folder structures on our server for the time being. I havenโ€™t moved them. The table should consist of a [Key], [JobID], [ImageURL]
    The structure of the files is:
    3X_6_d_6df87a60b995e4b27132919d9fbe974c77c18d95.png

Each folder is named after the [JobID] which is unique.
3X_7_d_7d566248ca1c7e4636476aaa753e2274375c3fb3.png

They would only need to be accessible by the App.

I hope this is what you meant.

Cheers

Hi @Dave_Willett,

Are you primarily looking to just browse these files via an AppSheet app?
Can you tell me more about the use case?
What are these images for?
Who creates them and puts them in Drive?
Who would use them from AppSheet?
I assume the folder structure is dynamic (every new job will create a new folder). Can you confirm?
Would you want to also run these images through some automated process, like pass them through the Doc AI service from Google to recognize and categorize the images somehow? (ie: OCR for Receipts or Invoices)?

Use cases might be:

  1. Search by Job ID 1234 to locate the folder for that job
  2. Show me all the images for the Job ID 1234
  • Would you want to view these images in a card or deck view where you can get a preview of the image
  • Or, do you just want to look at the file names?
  1. When new images arrive, notify AppSheet User X so that they can review the Job Images and approve the job completion?
  2. When new images arrive, run them through the Doc AI service from Google AI to collect data from them automatically.

Iโ€™m totally guessing on the use cases, so please correct these.

Thanks,
Scott

Hi Scott.
My App shows all vehicles within our workshop, each vehicle has a unique ID via JobID which canโ€™t be duplicated. For each vehicle having a repair done there are a set of images take whilst in-progress.
So from your comments No 2 'Show me all the images for the JobID 1234.

Ideally for simplicity I would want a gallery, so it should be โ€˜Show me a gallery for images beginning with JobID 1234โ€™

All images are in one Google Drive folder called \ImagePath I changed how the upload structure works so all images are in one location and each set of images are altogether. So there is not a separate folder for each JobID:

3X_7_6_76b6950915ba915aa742ced36978acb33b80f29f.png

The images are uploaded once each day by an access database running in the workshop which is totally automated. The images are selected for upload via a csv file and that csv file should be the table for the gallery.
We donโ€™t need any fancy bells and whistle such as Doc AI or Google AI, we simply just need to show a basic gallery. So if I select the row which contains JobID 244786, then allow me to see a gallery where the filename (before the โ€œ-โ€ character) = JobID from the row.

Does this make any sense?
Cheers

Is this possible guys?

Hi @Dave_Willett,

Yes, this makes sense! I think putting all the files in one directory will help a lot to simplify how to navigate through the files. What I can suggest for now is to use an external process to put your file names and links to the files into a sheet. You could use Apps Script to do this and you could trigger the Apps Script to run at the end of your Access DB job that writes the files (or use an external tool as suggested by others above, or just modify your Access script?). One you have the file names in a sheet, you can use expressions and slices to get the views you want.

We do have an interesting feature on the roadmap that will make this easier in the future called the Google Drive Data Source. It will allow you to view a Drive folder as a Table, and each file as a row. Once that is available, you would not need an external process to get the view you want of the files in your app. Would you be interested in being an early adopter to try it out? If so, let me know.

All the best!
Scott

Oh man, this would be awesome! Could you sign me up for the beta test please?

Hi @Scott_Haaland, that would be such a fantastic addition to the suite of Appsheet, please sign me up.

Kind Regards

I almost overlook this important announcement! Better to push to new thread with other category rather than Quesion.

I wish to try this new feature as well

ID 473552

Im curious if this new feature enable us to access property of saved files, such as meta data, size, file ID (Google Drive assigned file ID), etcโ€ฆ Any quick list for the available property is there to share?

Thank you.

โ€โ™‚

Hi @tsuji_koichi - san,

Yes, it will be announced more broadly once we are a little closer to being able to get a preview release ready. We will not be able to start the beta/preview until after the New Year. Iโ€™ve marked you and all the others in this thread down as interested beta testers, so Iโ€™ll make sure to let you all know when we are ready to start.

Yes, you will have access to some of the metadata. We will start with the most commonly used fields like File Name and itโ€™s unique ID, File Size, DateTime Created and modified, etcโ€ฆ If you have any obscure properties that you think would be useful, it would be good to know about those, and what the use case would be for the additional properties. Each property would just be a column in your table.

Thanks for your interest!

It canโ€™t happen quick enoughโ€ฆ

Cheers

Hi @Scott_Haaland

Thank you for your response and explanation. I will looking foward to testing this new feature in due course.

I assume google Drive API could be involved somewhere in this new development. In that case, the properties which will be available under FILES object could be readable?

โ€œimageMediaMetadataโ€: {
โ€œwidthโ€: integer,
โ€œheightโ€: integer,
โ€œrotationโ€: integer,
โ€œlocationโ€: {
โ€œlatitudeโ€: double,
โ€œlongitudeโ€: double,
โ€œaltitudeโ€: double
},
โ€œtimeโ€: string,
โ€œcameraMakeโ€: string,
โ€œcameraModelโ€: string,
โ€œexposureTimeโ€: float,
โ€œapertureโ€: float,
โ€œflashUsedโ€: boolean,
โ€œfocalLengthโ€: float,
โ€œisoSpeedโ€: integer,
โ€œmeteringModeโ€: string,
โ€œsensorโ€: string,
โ€œexposureModeโ€: string,
โ€œcolorSpaceโ€: string,
โ€œwhiteBalanceโ€: string,
โ€œexposureBiasโ€: float,
โ€œmaxApertureValueโ€: float,
โ€œsubjectDistanceโ€: integer,
โ€œlensโ€: string
},
โ€œvideoMediaMetadataโ€: {
โ€œwidthโ€: integer,
โ€œheightโ€: integer,
โ€œdurationMillisโ€: long
},

We deal with bunch of โ€œimageโ€ files under Appsheet app. So access image file meta data is super useful. However, there is downsideโ€ฆ Currently, Appsheet is not retaining the meta data for image file once we upload image through the appsheet.
Along with this new feature development, I really wish your team to review the current limination to drop meta data of image file, expecially the datatime, lat/long and other valuable meta data could be retained when we up the image.

Arther is on a case to introduce new video upload feature as well. Once this happens, the access to videometadata could be also useful, as we can capture the duration and other meta data for the file and display withing appsheet app once uploaded.

This is a trick I m currently using to capture Google Drive file IDs, this process is painful, as I dont want to CODE as much as possible. Once we could capture Google Drive file ID automatically or rather simply than pushing own GAS, then it will be another game changer. We can create the download link for the file using IDs and can do more by manipulating the ids.

Thank you and really looking forward to having it.

Koichi

โ€“

Maybe it is a good idea to get the path & folder ID for the files as well.

@LeventK

Absolutely @tsuji_koichi san

Hi @tsuji_koichi - san,

Thanks for your valuable input. Iโ€™ll take this into considerationโ€ฆwe want the table to be as simple as possible, but if dealing with images primarily, I can see how all this meta data would be useful.

Thanks,
Scott

Well I have this working, but it is very slow. Google script:

*/
function listFilesInFolder(folderName) {
var sheet = SpreadsheetApp.getActiveSheet();
sheet.clear();
sheet.appendRow([โ€œNameโ€, โ€œDateโ€, โ€œSizeโ€, โ€œURLโ€, โ€œDownloadโ€, โ€œDescriptionโ€, โ€œImageโ€]);

//change the folder ID below to reflect your folderโ€™s ID (look in the URL when youโ€™re in your folder)
var folder = DriveApp.getFolderById(โ€œGoogle Folder ID Goes Hereโ€);
var contents = folder.getFiles();
var cnt = 0;
var file;
while (contents.hasNext()) {
var file = contents.next();
cnt++;
data = [
file.getName(),
file.getDateCreated(),
file.getSize(),
file.getUrl(),
โ€œhttps://docs.google.com/uc?export=download&confirm=no_antivirus&id=โ€ + file.getId(),
file.getDescription(),
โ€œ=image(โ€œhttps://docs.google.com/uc?export=download&id=โ€ + file.getId() +โ€")",
];
sheet.appendRow(data);
};
};

Which populates a column in your Google Sheet (used as the table)

3X_1_2_12db019b0b1b3a957747189fe3ff2b1c6360a0e5.png

But it is painfully slow on loading the image.

Well done, @Dave_Willett! Iโ€™ll bet that was fun to figure out!
Iโ€™ve built a few import tools like this and itโ€™s always a fun learning experience.

On your post, try enclosing the code with 3 tic-marks ``` on their own line, at the beginning and end.

That will separate the code with a scroll box like this:

*/
function listFilesInFolder(folderName) {
var sheet = SpreadsheetApp.getActiveSheet();
sheet.clear();
sheet.appendRow([โ€œNameโ€, โ€œDateโ€, โ€œSizeโ€, โ€œURLโ€, โ€œDownloadโ€, โ€œDescriptionโ€, โ€œImageโ€]);
...
...

And I canโ€™t tell for sure, but it looks like youโ€™re updating the sheet once for each row. That could be why itโ€™s so slow. Try building the array first, and then writing all the data to the sheet in one edit.

@Scott_Haaland
Iโ€™m highly interestedโ€ฆPlease opt me inโ€ฆ
ID: 245151

It has been really difficult to figure out, but I got there.
Populating the sheet is not too bad as it happens once every hour and there are (now) usually only 500 or so images. But the slow issue is that the column I take the image from is still a URL:

3X_e_1_e1a4173364fe311785c56e33b2ea83736f06e4dd.png

So Appsheet has to try to change that to an Image. Iโ€™m sure the guys at Appsheet will soon have an update so we can just load images direct from Google Drive.

Question: Can an array be built in Appsheet before the URL is converted to an image?

I was referring to building an array in your Apps Script code. Thereโ€™s no equivalent in AppSheet.

Instead of this:

-try storing a variable of the row instead of writing it directly to the sheet. Then push (append) the data of each new row to the variable to build up the full set of row before writing it to the sheet.

I wouldnโ€™t know where to start to be fair. Iโ€™m open to any assistance I can get with this.
Thank you for the advice though.

Kind Regards

Good Job @Dave_Willett! This is a great accomplishment you can be proud of.

It does seem that the issue is not in the loading of the sheet row by row (the Apps Script), but rather the App User looking at the images from the sheet (which should already be pre-constructed by the Apps Script on an hourly basis).

So, assuming you have a sheet with URLs pointing to images on Driveโ€ฆwe need to figure out why the app load of those images is slow.

According to this help article, the URL should have export=view instead of export=download, and the column should be of type โ€œImageโ€ in the table definition.

So your URL: https://docs.google.com/uc?export=download&confirm=no_antivirus&id=โ€ + file.getId() construction statement should change to:

https://drive.google.com/uc?export=view&id= + file.getID()

I suspect that the Download directive is what is taking the extra time, but Iโ€™m not sure. Could you try changing a few of the image URLs manually in your sheet and see if they load faster? If so, you can just change your Apps Script and re-run it. Please let us know what you find out.

R,
Scott

This is now getting exciting, let me see what I can do with your suggestion.

Much appreciated.

Ok so Iโ€™ve changed the URL in the script (Removed some characters from the folder ID for security):

If I have understood the recommendation, does this look correct to you guys?

The script returns this in the Google Sheet (Notice the column [Download] and [Image] both have data):

This is the structure of the table in Appsheet:

So if I look at the table data in Appsheet notice what happens in the [Image] column, it is empty but the [Download] column still has the URL. The column [Image] also returns nothing in a view:

Changing both URLโ€™s to:

Does allow the Gallery and Images to load slightly faster thoughโ€ฆ

Update
As my images are exported via a workshop TV (linked to a PC) into the local (C:\users\dave.willett\Google Drive\ImagePathโ€ฆ) via Microsoft Access, I am able to utilise the WIA - Windows Image Acquisition Library to reduce the imge size before it gets to the Google Drive and therefore a 737kb file is now approx 45kb and they load much quickerโ€ฆ

I did have a recent project where my clients save thousands of image and movie files directly down to the google drive (files not being saved from Appsheet App) initially, and then we display those images and movie file on the Appsheet App.

Firstly what I did was similar to yours, let the google app script to get the full path for those files, where the URL looks like https://drive.google.com/uc?xxxxx. I concat the strings of https along wiht the Google drive file ID to manually construct the public URL. Then I found URL itself is working fine, but it takes almost forever to render images on the Appsheet app. Just deadly slow.

My observasation was it is natually slow, as we query to Google s server to get the files. File IDs would give us a key to access to target file, but it was making sense it takes a bit of time to get the response back from Google server to render the target images, as query strings (google file ID) should search ALL the files stored witin the server. Meaning, query strings are checking ENTIRE google drive server to find a single file. It should be by design, so I wont complain.

Then I altered my approach a bit. I get the google apps script which will yeilds the FULL PATH for all those image and movie file. We need to run script manually to get the data donw to the spreadsheet, which is fine with me. No automation is needed on my use case.

After playing around a bit for strings, eventually I managed to get the strings which stands for the relative path + file name of each file as strings. This can be done within Appsheet Virtual column where we dynamically construct Appsheet Style of file path, meaning the folder name(s)/fine name.

Then set this vircual column as Image type.

Appsheet render images acceptably fast now. This could be because the relative strings give the shortest way and quickest access to the files stored my own google account drive, rather than checking any other folders owned by others. When we use the public url of https://drive.google.com/uc?xxxxx fileIDsโ€ฆ then naturally we are querying to other google drive account hold and owned by others.

Bit of tricks to manage this challenge.

Lesson learned from this use case.

Yes, public URL of https://drive.google.com/uc?xxxxx. fileID can be used within Appsheet app, but in case the amount of image files to deal with is small enough.
Once the image file are large or huge, then it takes almost forever to render image through this URL, then should swith to the โ€œrelative pathโ€ to construct the URL and render to Appsheet.

@Scott_Haaland

I would suggest as well as hope that we could access to each files RELATIVE PATH strings from new features should be useful based on my recent experience which mentioned here.

https://drive.google.com/uc?export=view&id= + file.getID()

would work as far as we are able to access to fileID value, but it becomes inactive as the number of the files glows to render on Appsheet.
Better and alternative approach is to get the files relative path, starting from root folder PLUS file name. Then we can construct relative path for each images as a string. This relative path (manually and virtually constructed) can be used as if the files are saved from Appsheet app.

Hope this makes sense with you as well.

Hi, thank you for that.
Having tested my results over the last few days and even compressing my images to approx 25kb ( with loss of quality ) the rendering is still very slow, so really I didnโ€™t necessarily have to compress them down too much.
I am at a loss with this and it is very important to fix this.

Any help I can receive to speed up the rendering would be greatly appreciated.

Regards

Sorry for confusion if I made, but using my tricks should be nothing to do with file size.
My suggestion is not to use

https://drive.google.com/uc?export=view&id= + file.getID()

type of public URL in case you have bunch of image files to deal with.

As it should render images within Appsheet deadly slow.

Do you have script which yield file names on the spreadsheet?
Are image files sitting in a single folder within your drive?

If your answers for both are positive, you are around the corner.

Let me assume you have spreadsheet with column with file name. [fileName]

Then if your image folder are sitting on the root position with folder name of โ€œImagesโ€, then you construct the virtcual column with expression something like

Concatenate("/images/",[fileName])

which returns the relative path (in Appsheet style) dynamically. Set this virtual column as image type.

Hope this is clear to you.

I just reviewed your code and looks your have file.getName() methods already as well as you have single folder to get those. So my suggestion naturally works.

Good luck.

Thanks @tsuji_koichi - san,

This is good information to know. In any case, with the Drive data source, it will only point to a single folder at a time, so the relative path should be a simple formula to add the known path (the folder that is currently configured) and the file.name(). Iโ€™ll raise this with engineering and suggest that we run some volume tests to try this out. Iโ€™m surprised that fileId is not an index in Drive that should be really fast compared to file names!! But based on your experience, it looks like there is some other design in place!

Weโ€™ll look into it and get to the bottom of this.

Thanks again, and Happy New Year!

Scott

Happy new year once again @Scott_Haaland

Yes, appsheetโ€™s โ€œrelative pathโ€ to the target file should be the most shortest way to reach out to files rather than Google drive file ID based URL. I m not checking from technical aspects, but it should make sense.

Google drive โ€œDriveAppโ€ Class is providing the methods to get the folders name where the target files are sitting in. So I believe your engineering team can relatively easily add this property to the new table feature.

I could not loacate your previous post, but I remember the new feature yet able to get the nested folders and tree structure to get the meta data. However, as you can imagine, the most of the practical use case of the drives are using tree structured folders system to store the required data. I hope the application to the tree folder system to get meta data even from the child folders is on the scope for the further development.

Thank you.

Hi Tsuji, happy new year and thank you for the work youโ€™ve done here.

Iโ€™m Mike, Iโ€™m new on the AppSheet team but have been at Google for a while now, and used to work on the Drive API (Google Developers Blog: Announcing Version 2 of the Google Drive SDK).

This is a really interesting thread. I understand weโ€™re using Drive as a datasource and experiencing slowness when rendering the images in this case. Also understood that you get a performance increase when using a different approach for querying the files (is this a different library altogether?)

Actually getting file metadata by fileId should be lightning fast. FileID is in fact a universal index and is, historically, the fastest way to retrieve file metadata. This of course assumes that querying by fileID is using the specific files.get() call in the underlying API, and not a generic files.list() command with โ€˜qโ€™ parameter query string and passing in file ID or something like that to the generic files.list() command. Weโ€™ll want to confirm that on our end.

Two things to try:

  1. You may be able to speed up the results of a get request, by explictly using the โ€œfieldsโ€ parameter to request only the fields needed when using the API:
    Return specific fields for a file  |  Google Drive API

  2. There is an alternate download mechanism for drive other than the /uc endpoint that, if youโ€™re just trying to get the bytes of the file, could be faster. They are referred to as โ€˜mediaโ€™ downloads and, in my experience, I would consider this the canonical way to retrieve binary file bytes. See here:

See how that does and let us know!

Hi Mike @Mike_Procopio

Thank you for your response and advice.
Most importantly, welcome to Appsheet community place,

Out of my own opinion, the issue, i.e. the difference in the speed to render image through the different URL is purely because of Appsheet image rendering system rather than Google Drive and file APIs.

The url is captured on the data source, and required URL should be in place.

I did a bit of investigation.

Lets say we have two different set ups.

Set up 1. URL to image file using google drive, which looks like something like https://drive.google.com/uc?export=view&id=xxxxxx

Set up 2. URL which Appsheet natively provide, which looks like something like /appsheet/data/myimage/dummy.jpg

Based on the setting, go to the app and get and copy the image URL by hovoring over the image and usual right click technique.

Naturally and interestingly, each set up returns the URL like this.

Set up 1 image full path url - https://www.appsheet.com/remote/https://drive.google.com/uc?export=view&id=

Set up 2 image full path url - https://www.appsheet.com/template/gettablefileurl?appName=

Both image url is starting with the same domain of www.appsheet.com, regardless of the difference of the set ups, but the rest of query strings are different. (naturally)

As a result, what is happning is set up 2 is rendering image far more faster than set up 1.

For now, to give the best tuned app set up to minimize the rendering speed on the app, my app stays with the set ups 2, as I recapped in my previous post, rather than using https://drive.google.com/uc?export=view&id=xxxxxx.

For me, it looks like the difference should be coming from Appsheet backend server logics and set up, in terms of how to reach to the target file , alghough it is entirely out of my guess work.

This is an awesome expedition tsuji-sanโ€ฆ
ใ‚ใ‚ŠใŒใจใ†ใ”ใ–ใ„ใพใ—ใŸใ€‚

Top Labels in this Space