I strongly believe A/B testing users without their knowledge and enthusiastic consent is unethical. If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all. Users don't want their shit changing all the time.
Hmmm, I am curious about which aspect of this you find unethical.
Is it unethical to do phased rollouts (where a small percentage get the new version) as a way to do safe deploys? If the issue is that two users making requests at the same time might see different things, then this would also be unethical? Yet, these sorts of phases rollouts is the best way to release something safely. When I worked at a large CDN with 50,000 servers around the world, we ALWAYS did phased releases, to make sure we didn't take down everything all at once, and to make sure we caught any performance regressions right away.
Is your issue that the user might be getting a version that won't stick around? That seems always the case, whether you do A/B or not. You might rollback if there is an issue, and you will certainly roll forward at some point, meaning users will get a new version at some point.
Would it be an issue if the A/B test was temporal? Like all users got one version today, and a different version tomorrow?
I guess I am just confused by this statement:
> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all.
This seems contrary to so many other complaints we see all the time, that companies push changes out without taking into account what users actually want. So, do we want companies that push out changes with no user feedback because they are confident that they know what users want, or do we want companies that get feedback from users on whether new changes are helping or hurting.
If I paid for a copy of Windows 95, I'm expecting Windows 95 in a box.
When I have two free hours to turn on the XBOX for the first time in half a year, I want it to turn on right away and play my game. I don't want the box I paid for and have been looking at to figure that it needs an OS update, and a game update, and that the game should now be slower and glitchier than it was the last time I played it.
When I play music on my phone in my car, I don't want to find out that the "Start Mix" button moved, or that showing the upcoming playlist now takes one more swipe, or that the UI won't load because YouTube Music doesn't cache it's UI anymore and when you have a cell network reporting 1 bar but it's actually zero bars, you get a spinner for ten minutes.
The six CD changer in my dashboard has worked exactly the same way since 2006, the discs play when the key turns on, and nothing moves. There's no engagement to be had other than "my music plays when I turn on the car in my driveway which also has spotty cell service". There isn't a KPI to be measured, a PM to be promoted, or anything. It's just a car radio.
Most consumer goods are solved problems. Nothing's changed since 2015. Even tech from 2015 is just a convergence of 2005 tech, like MP3 players, digital cameras, and Blackberries. People don't have radically different problems to solve in their day to day lives. People take pictures, share them, do email and group chat, voice calls, read the news, watch TV, pay for parking, do some banking. Watch a 90s TV show and all of those activites required different physical places and tactile goods. (Hell, that's why screens in Android are called *Activities*.).
When I pay for a product, I expect to be paying for a finished product, not some psychological experiment that's someone else's promo packet.
Most of the websites people are talking about here are free, they aren't things you paid for, so the argument that when you pay for a product you should expect something doesn't really apply.
OkayPhysicist · · focus · HN ↗
cortesoft · · focus · HN ↗
Is it unethical to do phased rollouts (where a small percentage get the new version) as a way to do safe deploys? If the issue is that two users making requests at the same time might see different things, then this would also be unethical? Yet, these sorts of phases rollouts is the best way to release something safely. When I worked at a large CDN with 50,000 servers around the world, we ALWAYS did phased releases, to make sure we didn't take down everything all at once, and to make sure we caught any performance regressions right away.
Is your issue that the user might be getting a version that won't stick around? That seems always the case, whether you do A/B or not. You might rollback if there is an issue, and you will certainly roll forward at some point, meaning users will get a new version at some point.
Would it be an issue if the A/B test was temporal? Like all users got one version today, and a different version tomorrow?
I guess I am just confused by this statement:
> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all.
This seems contrary to so many other complaints we see all the time, that companies push changes out without taking into account what users actually want. So, do we want companies that push out changes with no user feedback because they are confident that they know what users want, or do we want companies that get feedback from users on whether new changes are helping or hurting.
linster · · focus · HN ↗
When I have two free hours to turn on the XBOX for the first time in half a year, I want it to turn on right away and play my game. I don't want the box I paid for and have been looking at to figure that it needs an OS update, and a game update, and that the game should now be slower and glitchier than it was the last time I played it.
When I play music on my phone in my car, I don't want to find out that the "Start Mix" button moved, or that showing the upcoming playlist now takes one more swipe, or that the UI won't load because YouTube Music doesn't cache it's UI anymore and when you have a cell network reporting 1 bar but it's actually zero bars, you get a spinner for ten minutes.
The six CD changer in my dashboard has worked exactly the same way since 2006, the discs play when the key turns on, and nothing moves. There's no engagement to be had other than "my music plays when I turn on the car in my driveway which also has spotty cell service". There isn't a KPI to be measured, a PM to be promoted, or anything. It's just a car radio.
Most consumer goods are solved problems. Nothing's changed since 2015. Even tech from 2015 is just a convergence of 2005 tech, like MP3 players, digital cameras, and Blackberries. People don't have radically different problems to solve in their day to day lives. People take pictures, share them, do email and group chat, voice calls, read the news, watch TV, pay for parking, do some banking. Watch a 90s TV show and all of those activites required different physical places and tactile goods. (Hell, that's why screens in Android are called *Activities*.).
When I pay for a product, I expect to be paying for a finished product, not some psychological experiment that's someone else's promo packet.
cortesoft · · focus · HN ↗
spaqin · · focus · HN ↗