Thread

AI companies in race to demonstrate their model most threatening to humanity

441 points · 399 comments · ljewalsh

chasd00 · 1790595720 · focus · HN ↗
I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
1. hellweaver666 · 1790597785 · focus · HN ↗
  I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
  1. mofeien · 1790599155 · focus · HN ↗
    When, in the past three years, has model progress seemed to decelerate to you, indicating some limit?
    The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: <a href="https://aistatement.com/work/statement-on-ai-extinction-risk" rel="nofollow">https://aistatement.com/work/statement-on-ai-extinction-risk
    They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
    nsagent · 1790600627 · focus · HN ↗
    It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
    GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
    AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted)
    Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.
    [1]:<a href="https://artificialanalysis.ai/evaluations/omniscience#omniscience-hallucination-rate-tabs" rel="nofollow">https://artificialanalysis.ai/evaluations/omniscience#omnisc...
    DoctorOetker · 1790603719 · focus · HN ↗
    I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.
    Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
    The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
    It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
    <a href="https://arxiv.org/abs/2404.01019" rel="nofollow">https://arxiv.org/abs/2404.01019
    Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
    But then all the copy rights holders can simultaneously sue them.
    And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
    Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
    With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
    It's telling that the companies prefer regulation over source-aware training.
    EDIT: It's telling that the companies prefer regulation over source-aware training, which suggests the only additional regulation we need for now is mandating source-aware training?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.