I strongly believe A/B testing users without their knowledge and enthusiastic consent is unethical. If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all. Users don't want their shit changing all the time.
Hmmm, I am curious about which aspect of this you find unethical.
Is it unethical to do phased rollouts (where a small percentage get the new version) as a way to do safe deploys? If the issue is that two users making requests at the same time might see different things, then this would also be unethical? Yet, these sorts of phases rollouts is the best way to release something safely. When I worked at a large CDN with 50,000 servers around the world, we ALWAYS did phased releases, to make sure we didn't take down everything all at once, and to make sure we caught any performance regressions right away.
Is your issue that the user might be getting a version that won't stick around? That seems always the case, whether you do A/B or not. You might rollback if there is an issue, and you will certainly roll forward at some point, meaning users will get a new version at some point.
Would it be an issue if the A/B test was temporal? Like all users got one version today, and a different version tomorrow?
I guess I am just confused by this statement:
> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all.
This seems contrary to so many other complaints we see all the time, that companies push changes out without taking into account what users actually want. So, do we want companies that push out changes with no user feedback because they are confident that they know what users want, or do we want companies that get feedback from users on whether new changes are helping or hurting.
How about actually asking the users instead of experimenting on them?
> or do we want companies that get feedback from users on whether new changes are helping or hurting.
You don't get that feedback. The feedback you get is whether some telemetry KPI goes up or down. That's not the same as actual utility for the user.
Asking users for feedback is notoriously bad at generating good feedback. Most people don't respond, and those that do ask for things they don't actually want, or are only wanted by very few people.
I've heard we have this amazing new technology now that can understand natural language and automatically convert it into structured data if we ask it to. Maybe we could use that to ask people at scale now.
> and those that do ask for things they don't actually want, or are only wanted by very few people.
But you can’t build a business on “wants” that people don’t act on.
If everybody says they want foo and don’t want bar, but when you make foo they don’t buy any but will buy a lot of bar, then are you “failing to provide what people want” if you just make bar?
OkayPhysicist · · focus · HN ↗
cortesoft · · focus · HN ↗
Is it unethical to do phased rollouts (where a small percentage get the new version) as a way to do safe deploys? If the issue is that two users making requests at the same time might see different things, then this would also be unethical? Yet, these sorts of phases rollouts is the best way to release something safely. When I worked at a large CDN with 50,000 servers around the world, we ALWAYS did phased releases, to make sure we didn't take down everything all at once, and to make sure we caught any performance regressions right away.
Is your issue that the user might be getting a version that won't stick around? That seems always the case, whether you do A/B or not. You might rollback if there is an issue, and you will certainly roll forward at some point, meaning users will get a new version at some point.
Would it be an issue if the A/B test was temporal? Like all users got one version today, and a different version tomorrow?
I guess I am just confused by this statement:
> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all.
This seems contrary to so many other complaints we see all the time, that companies push changes out without taking into account what users actually want. So, do we want companies that push out changes with no user feedback because they are confident that they know what users want, or do we want companies that get feedback from users on whether new changes are helping or hurting.
xg15 · · focus · HN ↗
> or do we want companies that get feedback from users on whether new changes are helping or hurting.
You don't get that feedback. The feedback you get is whether some telemetry KPI goes up or down. That's not the same as actual utility for the user.
cortesoft · · focus · HN ↗
xg15 · · focus · HN ↗
> and those that do ask for things they don't actually want, or are only wanted by very few people.
Then how do you know what people actually want?
Joker_vD · · focus · HN ↗
isabelc · · focus · HN ↗
cortesoft · · focus · HN ↗
If everybody says they want foo and don’t want bar, but when you make foo they don’t buy any but will buy a lot of bar, then are you “failing to provide what people want” if you just make bar?
fragmede · · focus · HN ↗
-(not) Henry Ford