The thing about the "AI can make mistakes, so double-check responses" thing is the essence of our future hellhole — deterministic software replaced with AI and legal disclaimers.
The reason the firms do not want to invest in making fact-checking a first-class feature is that the appearance of being right is what people want from AI.
> The reason the firms do not want to invest in making fact-checking a first-class feature is that the appearance of being right is what people want from AI.
No, actually being right is what people want from AI, "the appearance of being right" is all that AI companies can deliver. It comes with the benefit that many people will be fooled into thinking that AI is more capable/useful than it actually is. AI companies have to either convince others that their product is something that it isn't, or that at least it will one day be something much more than it is.
> No, actually being right is what people want from AI,
You are considerably more optimistic than I am about how people use things like this. IMO people are happy with these tools if they can be used to support their existing positions and biases.
Even vibe-coding is like this. OK it creates code that compiles, but its primary job is still to confirm a bias. I have yet to see people making significant novel discoveries about functionality this way.
It's certainly true that AI annoyingly sucking up to people and validating whatever insane rambling they feed it drives more engagement. When AI tells some guy who is 50 hours deep into AI psychosis that together they've discovered a new form of physics that will change the world he wants to believe it's all true.
I wonder how popular it'd be if someone started an AI company that was explicitly advertised as "blowing smoke up your ass as a service", and clearly said that their chatbot was written to always agree with anything you said and to tell you whatever you want to hear. I don't doubt there's at least some market for it, but I'm guessing not many people ask for that in their prompts with current AI offerings.
dofm · · focus · HN ↗
The reason the firms do not want to invest in making fact-checking a first-class feature is that the appearance of being right is what people want from AI.
autoexec · · focus · HN ↗
No, actually being right is what people want from AI, "the appearance of being right" is all that AI companies can deliver. It comes with the benefit that many people will be fooled into thinking that AI is more capable/useful than it actually is. AI companies have to either convince others that their product is something that it isn't, or that at least it will one day be something much more than it is.
dofm · · focus · HN ↗
You are considerably more optimistic than I am about how people use things like this. IMO people are happy with these tools if they can be used to support their existing positions and biases.
Even vibe-coding is like this. OK it creates code that compiles, but its primary job is still to confirm a bias. I have yet to see people making significant novel discoveries about functionality this way.
autoexec · · focus · HN ↗
I wonder how popular it'd be if someone started an AI company that was explicitly advertised as "blowing smoke up your ass as a service", and clearly said that their chatbot was written to always agree with anything you said and to tell you whatever you want to hear. I don't doubt there's at least some market for it, but I'm guessing not many people ask for that in their prompts with current AI offerings.