Is AppSheet's modified rollout operation correct?

[takuya_miyai]

Hi AppSheet team and everyone.

I am curious about AppSheet's recent rollout of new features.
AppSheet initially said that they would provide the modified release to Free accounts, where it would be rolled out to Paid accounts after sufficient feedback was received.

However, the current release to paid accounts has started in no time, just as the release from Free accounts is starting after all.

As a result, frequent paid accounts are the first to notice and point out problems with releases that contain serious issues.
And the problem is actually occurring in the apps that are in production. (Horrible...😰)

The paying user reports the bug and the Free account user benefits from the bug fix.
This is not fair at all.

As other users have pointed out, I think AppSheet needs to take another look at its release procedures.

@zito 
@lizlynch 

@Koichi_Tsuji 

[zito · Accepted solution]

Okay, I understand now - appreciate the additional context.  I think that general feedback of a longer delay between paid and free is something we've heard previously, and in the example of some of the recent editor changes we've been making, we put in place a >1 month delay between free and paid, but given the limited rollout to free users, its possible that it may have seemed shorter if a free account was in the group that had the old experience.  In other words, we rolled out to a population of free users, waited ~6 weeks, then rolled out to the rest of the free users and then paid users.  So effectively we delayed the rollout to paid users quite a bit, but it may not have seemed that way - perhaps we should have gone: first group of free users->(six weeks)->remainder of free users->(2 weeks)->paid users.

Candidly, there's a tough balance to strike here - we've been chatting with a number of customers about their preferred rollout process/procedure, and there hasn't been a consensus around the optimal delay between free users and paid users.  Some customers want something closer to your Salesforce example, some want a 2 week delay, others want a configurable delay.  

Workspace offers the ability to choose between Rapid release and Scheduled release, which is a model that seems reasonable, but they also widely vary their rollout speeds, which makes the options less useful (in my opinion). 

For us, there's a balance, as having long periods of time where different users have different experiences complicates everything - documentation, testing, support, etc.  The more complicated the rollout process, the more chances for issues to arise.   

All this to say, this is very much on our mind, and I was just chatting with some folks on the team about this last week.  I hope that we can address this in a comprehensive way this year by offering a configurable option for rollout speed, but in the meantime, I'll work with the team here to try to put together a more formal set of guidelines for rollouts and share them here. 

And please keep the feedback coming - its not complaining.  I appreciate when folks take the time to clearly outline challenges they're facing and suggest improvements.  I wish we had the resources to fix everything, but we take all of the feedback into account as we go through our planning and development cycles.  Happy to discuss more here, of course. 

[zito]

Hey Takuya,

Thanks for bringing this to my/our attention.  Typically our process is to roll things out to free users first and then to paid users subsequently.  Occasionally that is not possible and it rolls out to everyone, but I can't think of a time when we have rolled out to paid users first and then to free.  That's not to say its impossible, and I've asked one of the folks on the team to investigate and will follow up here.

One thing that IS possible is that sometimes when we roll out features to free users, we roll it out to a portion of users as a test and keep it that way for a while. In that case, it's possible that a particular free user account might not be in the group that gets the new feature.   

Do you have examples of features that were rolled out to paid users before free users?  Then I can go back and look at the specific releases and their schedule. 

Thanks in advance. 

[takuya_miyai]

Thanks @zito 

My point here was not that the release will start before the Paid account over the Free account.
It is that the release to Paid accounts comes too soon after the release to Free accounts.

I feel that as soon as the release to the Free account is complete, they will begin releasing to the Paid account.

If a bug is included there, the result is that the Free account, which is only using the app for verification and trial use, will not notice it, and the problem will occur in the operational app that the Paid account is using.

As you probably know, Salesforce, for example, has three major releases per year, and the Sandbox environment is provided with future releases after one cool selection process.
They then get feedback from the Sandbox environment and fix it before the major release in 3-4 months.

https://admin.salesforce.com/blog/2021/salesforce-release-overview

Of course I don't think it's right for AppSheet to adopt a Salesforce-like release workflow. I think AppSheet has the other best releases.
But again, right now the release period from Free to Paid accounts is too short.

I also understand that there are so-called hotfixes that are not usually included in the releases.
However, if this is also a state of continuous hotfix releases, I believe that there is a problem with the process from development to release in the first place.

Now that the use of AppSheet is expanding dramatically, I would like App Creators who are not yet familiar with AppSheet to avoid unfamiliar Warning indications and bugs in the Automation process that make them uneasy.

Last but not least, despite all the complaining lately, I am getting more and more excited about AppSheet  development.
The number of users who are realizing how great AppSheet is has been growing rapidly in Japan recently.
I hope I can continue to give appsheet team feedback on how I can make the AppSheet platform even better.

Best regards.

[zito · Accepted solution]

Okay, I understand now - appreciate the additional context.  I think that general feedback of a longer delay between paid and free is something we've heard previously, and in the example of some of the recent editor changes we've been making, we put in place a >1 month delay between free and paid, but given the limited rollout to free users, its possible that it may have seemed shorter if a free account was in the group that had the old experience.  In other words, we rolled out to a population of free users, waited ~6 weeks, then rolled out to the rest of the free users and then paid users.  So effectively we delayed the rollout to paid users quite a bit, but it may not have seemed that way - perhaps we should have gone: first group of free users->(six weeks)->remainder of free users->(2 weeks)->paid users.

Candidly, there's a tough balance to strike here - we've been chatting with a number of customers about their preferred rollout process/procedure, and there hasn't been a consensus around the optimal delay between free users and paid users.  Some customers want something closer to your Salesforce example, some want a 2 week delay, others want a configurable delay.  

Workspace offers the ability to choose between Rapid release and Scheduled release, which is a model that seems reasonable, but they also widely vary their rollout speeds, which makes the options less useful (in my opinion). 

For us, there's a balance, as having long periods of time where different users have different experiences complicates everything - documentation, testing, support, etc.  The more complicated the rollout process, the more chances for issues to arise.   

All this to say, this is very much on our mind, and I was just chatting with some folks on the team about this last week.  I hope that we can address this in a comprehensive way this year by offering a configurable option for rollout speed, but in the meantime, I'll work with the team here to try to put together a more formal set of guidelines for rollouts and share them here. 

And please keep the feedback coming - its not complaining.  I appreciate when folks take the time to clearly outline challenges they're facing and suggest improvements.  I wish we had the resources to fix everything, but we take all of the feedback into account as we go through our planning and development cycles.  Happy to discuss more here, of course. 

[Steve]

Thank you for this post.

[takuya_miyai]

Thanks @zito 

Thank you for the detailed background explanation and the steps you are going to take.
I always appreciate your understanding of the feelings of both the platform developer and the App Creator and your accurate replies.🙏

As an App Creator, I will try to provide appropriate feedback as well.

[EliK]

Thank you @zito for taking the time to address these concerns. I too have felt that the number of feature releases in the past few weeks/ months have been dizzying. That's  a good thing that the Team is working to improve the platform in so many ways, but it has resulted in releases being rolled back after mass uproar from paid users. (such as the security filter violation warning). In that case, even if the filter violation warning had been rolled out to free accounts for a long time, the issues that users had were typically in advanced use cases which might not exist in free accounts. 

There needs to be a balance between giving the paid accounts the new features, but only after making sure they won't result in unintended behavior in a production app.

I like the idea of each app creator being able to opt in or opt out of each release, at least for several months after the initial release, in order to allow all the bugs to be fixed.

[zito · Accepted solution]

Hi folks - just to give a quick update on this thread.  I've reviewed a draft of a proposal around rollout scheduling, and we're iterating on the mechanics of how its going to work.  Someone from the team will start a new thread when we announce more formally, and we'll link back to it from this thread.  Thanks for the input and feedback, it's been helpful and informative.