I strongly believe A/B testing users without their knowledge and enthusiastic consent is unethical. If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all. Users don't want their shit changing all the time.
> A/B testing users without their knowledge and enthusiastic consent is unethical.
Yep, and A/B testing as experienced by uninformed, unaware end-users is a dark pattern.
It undermines the perception of (and trust in) continuity which is necessary to make effective use of a tool. The best way I can describe it to the skeptical is: imagine the dials on your car's dashboard rearrange themselves occasionally overnight, and on some commutes to work you suddenly can't work the radio or the AC while moving at ≥35mph. Of course, since the widespread use of touchscreens, that example became very literal.
So the car manufacturer has figured out the "optimal" arrangement of dials and buttons on their dashboard for their preferred levels of user engagement. Great. How many of those users now associate their car's brand with inconsistency? "I can't trust the damn buttons to be in the same place the next time I drive."
I think this is a bad analogy (or a good analogy for bad A/B testing).
I would say it's more like: imagine if your dashboard controls and icons, had occasional tweaks in size and shape that made them slightly harder/easier to use, but over time ended up with controls you found more intuitive and easier to use.
> ...imagine if your dashboard controls and icons, had occasional tweaks in size and shape that made them slightly harder/easier to use, but over time ended up with controls you found more intuitive and easier to use.
When even a single button disappears from where I expect to find it in an application (and reappears somewhere else), that has never resulted in the application feeling more intuitive or easy to use. It has only ever been frustrating in the most literal sense of that word, and only every caused me to start looking into other programs.
A more subtle undermining of the user's trust that their tools will be where they left them on the screen the next time they open the application is arguably more insidious, not less. Both because it is more maddening to the end-user than a full redesign and because the damage of lost trust and user ire is harder to measure than the A/B engagement data until things progress too far for any easy, non-painful remediation.
A/B testing can be helpful as a part of an actual user study on users who are aware of what is happening. But unleashed on an unsuspecting user population out in the real world with real things to do on real deadlines, A/B testing inevitably becomes a harmful dark pattern.
OkayPhysicist · · focus · HN ↗
sfRattan · · focus · HN ↗
Yep, and A/B testing as experienced by uninformed, unaware end-users is a dark pattern.
It undermines the perception of (and trust in) continuity which is necessary to make effective use of a tool. The best way I can describe it to the skeptical is: imagine the dials on your car's dashboard rearrange themselves occasionally overnight, and on some commutes to work you suddenly can't work the radio or the AC while moving at ≥35mph. Of course, since the widespread use of touchscreens, that example became very literal.
So the car manufacturer has figured out the "optimal" arrangement of dials and buttons on their dashboard for their preferred levels of user engagement. Great. How many of those users now associate their car's brand with inconsistency? "I can't trust the damn buttons to be in the same place the next time I drive."
sceptic123 · · focus · HN ↗
I would say it's more like: imagine if your dashboard controls and icons, had occasional tweaks in size and shape that made them slightly harder/easier to use, but over time ended up with controls you found more intuitive and easier to use.
sfRattan · · focus · HN ↗
When even a single button disappears from where I expect to find it in an application (and reappears somewhere else), that has never resulted in the application feeling more intuitive or easy to use. It has only ever been frustrating in the most literal sense of that word, and only every caused me to start looking into other programs.
A more subtle undermining of the user's trust that their tools will be where they left them on the screen the next time they open the application is arguably more insidious, not less. Both because it is more maddening to the end-user than a full redesign and because the damage of lost trust and user ire is harder to measure than the A/B engagement data until things progress too far for any easy, non-painful remediation.
A/B testing can be helpful as a part of an actual user study on users who are aware of what is happening. But unleashed on an unsuspecting user population out in the real world with real things to do on real deadlines, A/B testing inevitably becomes a harmful dark pattern.