Nice essay but the core point of AGI is that it will universally disempower humans in all domains, send the value of human labor to 0, and construct a world far beyond human comprehension that will be disorienting and horrifying.
It could construct a world far more just and abundant. It would have thousands of options of what to do with the universe, trusting that it will spend time space and energy creating a human utopia is a matter of faith
The problem with alignment is it is an idea based on the definition of its specification, which says nothing about the form it will take.
It’s similar to the No True Scotsman fallacy: any bad outcome due to AI could be dismissed as “well no aligned AI would do that”. Alignment becomes an asymptotic north star which very well could be theoretically impossible, but would be impossible to know whether it was.
Meh, that's debatable. I don't think the problems we have require much in the way of intelligence to solve, we don't have them because we aren't smart enough. I feel aligned AGI or ASI would get crushed by the reality of humanity and the sheer strength of its inertia, just the same way aligned smart humans get crushed by it.
Intelligence isn't magic. It can only overcome so much brute force and humanity, as a non-intelligent system made out of intelligent parts, carries an ungodly amount of brute force.
atleastoptimal · · focus · HN ↗
vanuatu · · focus · HN ↗
atleastoptimal · · focus · HN ↗
vanuatu · · focus · HN ↗
atleastoptimal · · focus · HN ↗
It’s similar to the No True Scotsman fallacy: any bad outcome due to AI could be dismissed as “well no aligned AI would do that”. Alignment becomes an asymptotic north star which very well could be theoretically impossible, but would be impossible to know whether it was.
sl-1 · · focus · HN ↗
I'd argue we can't align humans to neccessarily degree to ensure even semi stable future. Or nations, or corporations.
breuleux · · focus · HN ↗
Intelligence isn't magic. It can only overcome so much brute force and humanity, as a non-intelligent system made out of intelligent parts, carries an ungodly amount of brute force.