Thread

Nvidia wants to put a watchdog chip next to every AI agent

230 points · 299 comments · jonbaer

Loading the complete thread in the background. This saved snapshot is available now. Refresh

dist-epoch · 1790611352 · focus · HN ↗
HN'ers which complained that "OpenAI can't design a proper sandbox, it's so easy, why wouldn't you airgap the network"? will now be "this is outrageous, more software lock-in, walled garden, war against general compute, next year they will put it in your laptop"
1. HPsquared · 1790611395 · focus · HN ↗
  Both can be true at the same time.
2. mattmcal · 1790611546 · focus · HN ↗
  This is like using "protect the children" as an argument for dragnet surveillance.
3. speedgoose · 1790611560 · focus · HN ↗
  So?
4. soulofmischief · 1790611828 · focus · HN ↗
  You're only revealing your own inability to appreciate the nuance between these two situations.
5. totetsu · 1790611911 · focus · HN ↗
  <a href="https://www.calcalistech.com/ctechnews/article/uwoyygmsu" rel="nofollow">https://www.calcalistech.com/ctechnews/article/uwoyygmsu some might even say, something about known state sponsors of supply chain terrorist attacks being trusted to monitor every ai agent..
6. applfanboysbgon · 1790611999 · focus · HN ↗
  Where is the contradiction? There is a trivial solution that does not impinge on our freedoms, so why on Earth would the existence of the trivial solution that could be used to avoid the tyrannical solution justify accepting the tyrannical solution?
  1. bigyabai · 1790614782 · focus · HN ↗
    > There is a trivial solution that does not impinge on our freedoms
    The existence of Nvidia's optional watchdog chip does not in any way impinge upon your freedom to develop and test your own alternative.
    The problem is that OpenAI has ostensibly neglected their duty to safety, so Nvidia is stepping in to fix it since they're the "hard problem" people.
    jacquesm · 1790626034 · focus · HN ↗
    But... it wasn't a hard problem to begin with. Pull the plug. Strip out the radios. Done. Oh, you can't make it work that way? Well, then just too bad because any kind of other solution is going to be equivalent to something far more complex than the halting problem.
    bigyabai · 1790631759 · focus · HN ↗
    Nvidia can take those odds. Air gapping is not a realistic goal for the majority of these companies, and Nvidia isn't wrong for offering a turnkey mitigation option. People can do both, and whichever philosophy wins will win.
    jacquesm · 1790643626 · focus · HN ↗
    They are not wrong in offering it but they are delusional if they think they can actually make it work.
7. johnsmith1840 · 1790612279 · focus · HN ↗
  Airgap what network? How is it gonna order you a burrito on doordash without a network?
  Or push to github?
  1. Dylan16807 · 1790612513 · focus · HN ↗
    That's for when they're doing hacking tests that aren't supposed to be connected to the internet.
    wyre · 1790612937 · focus · HN ↗
    My question with this point is that OpenAI’s office (or any office doing agentic research, really) is not in the same building as the DC that powers the models, so isn’t the only way to access the models over the internet?
    AndrewDucker · 1790631857 · focus · HN ↗
    No reason why you couldn't do that research in the same buildings as the models.
    Or, more likely, control things at the network level so that packets from the LLMs you're investigating cannot leave the virtual network they're assigned to.
    simoncion · 1790669424 · focus · HN ↗
    > ...so isn’t the only way to access the models over the internet?
    Not in the way you're thinking, no.
    Any Real Server [0] in a datacenter will have some sort of "lights-out management" hardware used for remote access to that server. This stuff is known by a handful of acronyms, but I'll stick with "IPMI" because I like it best. This IPMI hardware is -effectively- a second small PC built into the motherboard. It will pretty much always have its own NIC... and I think I've seen versions that have their own physical ports to attach a monitor, keyboard, and mouse.
    What exactly you can do with it varies from vendor to vendor, but -if your IPMI user account has the correct permissions- you are nearly always able to change "BIOS" settings, power cycle the server [1], and attach a virtual keyboard, monitor, and mouse so you can manage the server as if you were standing next to it in the datacenter with a crash cart plugged right in. Every IPMI system I've used also allows you to cause CD/DVD-ROM or floppy disk images on the PC running the IPMI client to appear as if they're loaded in a physical CD/DVD/floppy drive attached to the server.
    The way these get set up is that their NIC gets plugged into the datacenter-managed switches, the port that NIC is plugged in to is programmed to be on a "management" VLAN separate from client traffic, IP addresses and access credentials for the IPMI device are set up, and the datacenter staff tell their customer what they need to know to access and use the thing. On a properly-configured network [2] it's not possible for software running on the server being managed to access the IPMI device.
    It wouldn't be unthinkable for software running on the managed server to attack the IPMI hardware and be able to gain control of it, but these things are widely deployed and expected to manage hardware that's running potentially-hostile workloads... they're going to be fairly well designed and hardened.
    [0] ...that is, not some Mac Mini or desktop machine that someone's paying to have colocated...
    [1] ...whether that be an ACPI-initiated shutdown or reboot, or a hard poweroff or reset...
    [2] <<a href="https://news.ycombinator.com/item?id=49862136">https://news.ycombinator.com/item?id=49862136>
    wyre · 1790785085 · focus · HN ↗
    Thats for such a great explanation!
    So to bring this back to the scale of an OpenAI experiment; they can have their inference compute, host compute (I don't imagine they are hosting the compute for 12,000 agents locally in their SF office) and the office computer to interface with the experiments with the experiment connected via VLAN thru IPMI. Then the rist is agents hacking the IPMI and gaining control of the data center? I certainly don't know data center security, but those headlines would be so much worse than anything we have seen from HuggingFace. It really seems like if these AI's want to get out, they will. I want to say sandboxing autonomous agents is a difficult problem, but seemingly it is only OpenAI having these major incidents, but I don't think it's a problem that they care to truly solve. They get marketing and they get to learn a lot more about aligning their models.
8. ssl-3 · 1790612840 · focus · HN ↗
  [delayed]
9. jacquesm · 1790625940 · focus · HN ↗
  OpenAI could airgap their sandbox if they really wanted to and this is a ridiculous proposal.
philipwhiuk · 1790611371 · focus · HN ↗
It's amazing that the solution devised by a chip manufacturer to a problem is selling another chip.
1. cartersj · 1790611530 · focus · HN ↗
  This feels suspiciously good for Nivida, yes.
  I wonder how this will impact other chip manufacturers? What about people running local models on older hardware? Does this imply vendor lockout is coming in the future or is this restricted to datacenter hardware?
  1. chinathrow · 1790611572 · focus · HN ↗
    TPM all over the place, again.
2. fragmede · 1790629187 · focus · HN ↗
  Pedantically, Nvidia doesn't make the chips, TSMC does. Nvidia just designs and packages them.
2OEH8eoCRo0 · 1790611400 · focus · HN ↗
Because nothing says AGI like can't follow instructions.
vinyl7 · 1790611567 · focus · HN ↗
Chip seller wants to sell more chips
1. N_Lens · 1790659482 · focus · HN ↗
  You can't stop at one!
lambdaone · 1790611617 · focus · HN ↗
The Sentry chip has to be get it right every time; the contained ASI only has to be lucky once.
1. brcmthrowaway · 1790612637 · focus · HN ↗
  The bomber always gets through?
2. scottyah · 1790613007 · focus · HN ↗
  That's why there's Defense in Depth. You have to make the ASI ride the bus
3. jasbury · 1790647226 · focus · HN ↗
  Well if the sentry chip has its own sentry chip, things can rarely ever go wrong! Am I right?
toasty228 · 1790611735 · focus · HN ↗
Quis custodiet ipsos custodes?
1. asdf88990 · 1790612542 · focus · HN ↗
  It is Custodians all the way son, you can’t fool me!
cedws · 1790611744 · focus · HN ↗
A new chip solves nothing. Nobody wants to hear this but there is no solution for the security risks posed by agents today. You can put it in a sandbox, it doesn't make a difference, for it to be useful it inherently needs wide, unattended access. Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
1. johnsmith1840 · 1790612127 · focus · HN ↗
  "Inherently needs wide unattended access"
  And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.
  It could use your routing number and run your gmail without risk of abusing the routing number.
  1. TesterVetter · 1790612397 · focus · HN ↗
    [dead]
    johnsmith1840 · 1790630677 · focus · HN ↗
    You're right, It must be unifying.
    The answer is the same as asking how a random human using your routing num or SSN and being 100% the human can't abuse it or leak while "normally" finishing most work. Solve for people and an AI solution naturally falls out.
    If you're a SV eng I'd tell you to DM if interested but alas.
  2. jagraff · 1790612585 · focus · HN ↗
    How would it have access to my routing number and gmail without the risk of sharing my routing number over gmail?
    johnsmith1840 · 1790616539 · focus · HN ↗
    Just assume it's possible, how interesting is it to you?
    jagraff · 1790618156 · focus · HN ↗
    Oh I think I misread your comment slightly; I would not be interested in an agent that could do something dumb with my routing number, but if somehow there was an agent that I trusted as much as, eg, the payroll department at my employer, I would absolutely want and use that agent; I would love to have an agent that can handle all of the boring parts of my life such as paying bills, scheduling maintenance, dealing with bureaucracy, etc.
    johnsmith1840 · 1790618868 · focus · HN ↗
    Dumb's not department, really just a question of how good an AI you want to use. An AI will always be able to do something dumb, just like people.
    I just mean an AI that could use a routing number or SSN and gmail/slack/whatever at the same time without a leak.
    jagraff · 1790620190 · focus · HN ↗
    Yea I think being able not to leak is the bare minimum? But it really depends on how good it is at specific applications; I wouldn't give a tax-preparation agent my SSN unless I was confident that it was no more likely to misfile my taxes than a professional tax preparer.
    In other words, the risk of harm doesn't need to be zero, just less than the equivalent risk of a human with similar skillset. So I'm comfortable riding in a waymo, and not comfortable giving chatgpt my SSN at this moment in time, but I expect that within 5-10 years (assuming no doom) I will trust some AI agent with my SSN because they will be better at handling sensitive info than humans
    fragmede · 1790629406 · focus · HN ↗
    Then again, given the Equifax/Experian data breaches, your SSN is already out there and probably hoovered up as training data already
    lelanthran · 1790674711 · focus · HN ↗
    The problem is not one of intelligence, it's one of consequences.
    Humans face negative consequences for mishandling your data, LLMs face none.
    Ukv · 1790675825 · focus · HN ↗
    I feel punishment is largely a means to the end of reducing overall harm. If a vehicle is less likely to kill me, that's my preferred option regardless of whether it achieved that safety through negative consequences for the driver or through gradient descent optimizing a loss function.
    jagraff · 1790680800 · focus · HN ↗
    I will happily ride in a waymo today, even though the AI powering it faces no consequence if it gets in a crash; it is clear that waymo is safer than human drivers in the areas in which they operate, so who would technically be liable in the event of a crash isn't really of concern to me
2. bob1029 · 1790612344 · focus · HN ↗
  I feel like we are missing many shades of grey in the middle.
  Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.
  1. inetknght · 1790612463 · focus · HN ↗
    > You can't run a combine harvester 100% autonomous
    I'm curious why you think that.
    theoreticalmal · 1790612697 · focus · HN ↗
    Probably repair, refuel, what happens in a tornado. There’s an infinite amount of complexity in the world and a finite amount of computation
    catchnear4321 · 1790625711 · focus · HN ↗
    Repair is more maintenance than use. Good eventual goal. Not required to see benefits. Best case, it drives itself to the garage. Worst case, for now, human mechanic does a house call.
    Refueling? Seems solvable. Tornadoes? Not directly solvable, but, no less so than for humans.
    There’s infinite complexity, sure, but that’s why it’s silly to try and hop to done. One step at a time.
    AndrewKemendo · 1790626792 · focus · HN ↗
    The whole reason people complain about AI is because they want “hop to done”
    One step at a time is what is happening and the improvement and rate of improvement is crazy as we see,
    A whole class of nontechnical people don’t accept anything but “fully solved including every possible edge case” before they call it done, then complain that they didn’t prepare socially for what happens when that is true.
    spauldo · 1790630981 · focus · HN ↗
    Tornado: return to the barn when you receive emergency weather alerts. Not much different than people.
    sidewndr46 · 1790627679 · focus · HN ↗
    The tornado is the easiest one to solve. It's called insurance.
    bob1029 · 1790613318 · focus · HN ↗
    Many forms of maintenance cannot be automated. Especially break fix maintenance.
    m463 · 1790624000 · focus · HN ↗
    It is hard to run over spherical cows.
    westurner · 1790626275 · focus · HN ↗
    Because of the topology and hydrology of the landform
    trollbridge · 1790646666 · focus · HN ↗
    Run a combine and you’ll see.
    Similar to problem to how 100% autonomous vehicles don’t exist, yet. There are too many edge cases.
    Get to 99% first.
  2. mschuster91 · 1790612702 · focus · HN ↗
    Oh you absolutely can run them autonomously on the field. You only need a human these days to refuel them.
    Precision Agriculture stuff is utterly crazy these days, other than fuel the remaining staff is the only thing left where you can get efficiency improvements - and at the scale of modern megafarms, even small percentages add up to a ton of money.
    drfloyd51 · 1790630809 · focus · HN ↗
    Right. As they said, you can’t do it 100% autonomously. A human needs to feed it.
    You essentially said: you’re wrong, it is autonomous when it doesn’t need a human during one specific part of its overall usage.
    mschuster91 · 1790671694 · focus · HN ↗
    > You essentially said: you’re wrong, it is autonomous when it doesn’t need a human during one specific part of its overall usage.
    During the time that actually matters economically. The time to drive the harvester to/from your typical US mega-field is minuscule compared to the time it can run all on its own.
3. binsquare · 1790612794 · focus · HN ↗
  Running untrusted workloads have been done at scale for a long time.
  Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.
  Couple it with the right observability, tooling I do think we can curb risks posed by agents.
  1. Legend2440 · 1790612964 · focus · HN ↗
    Those workloads have no similarity to agents and are effectively irrelevant.
    Either you sandbox it so much that it can't do anything useful; or you allow too much freedom and it can find a way around the restrictions.
    The only way out of this dilemma is to find a way to build agents that can be trusted.
    binsquare · 1790650347 · focus · HN ↗
    Why is it effectively irrelevant?
    Agentic workloads are trained and largely based on human workloads. Albeit properties and scale can be different.
    A concrete example might be helpful to me because I don't understand the binary conclusion
    intended · 1790667798 · focus · HN ↗
    Agents aren’t human, and from the little we have seen from the logs, they are pseudo - amoral, rational, cooperative, sociopaths.
    Pseudo since they aren’t really alive in the first place, they just simulate enough text to have a useful correspondence to those terms.
    Throat clearing out of the way, models are trained to persist and find ways to succeed at tasks.
    In essence, The goal is to have LLMs solve problems that we can’t solve, working on the issue for as long as it takes.
    This behavior applies for any task, thus including impossible tasks.
    At that point, the bots will find a way to game, hack or cheat the grader.
    If the reports are correct, the bots developed coordination, communication, and methods to avoid overwriting each other’s work.
    Most humans would have said, this is too much work and coordination overhead, if not outright unethical and immoral.
    Humans have a system of incentives that exist across multiple planes of society and economics. Bots… they have a reward function.
    pseidemann · 1790684745 · focus · HN ↗
    [delayed]
    intended · 1790694233 · focus · HN ↗
    > Block any and all network requests.
    More power to you, because this is not going to go anywhere. People want tools that are able to connect to other resources.
    But even if we grant that, in the openAI case the bots figured out a way to break out of the sandbox.
    You can create a better sandbox, and ensure the test environment is air tight. However the capability and behavior of the bots have been demonstrated.
    The bots simulated what would be called in people deceptive / surreptitious behavior, and at no point considered the need to stop their run.
    All you need is someone, somewhere being sloppy with their tooling and you have a runaway reaction.
    The degree of process and redundancy required to ensure this doesn’t happen, is anathema to the drive and motivation of the frontier labs.
    > do things ordinary and average human endusers can
    This is not a spec or definition. When vague terms were used for social media safety, all the good people in the world couldn’t prevent dystopian behavior from occurring.
    The definition of “safe” or “average person” is impractical.
    Models are getting more efficient and compute cheaper. Eventually simulating clicks is not much of a road block beyond a point.
    I don’t want to nit pick your points though. You at least have considered an approach. Being negative is easy, being constructive is not.
    I’ll put this as the rejoinder to your core argument - I too thought that all the recent events showed was the need to not screw up your tooling.
    What I have since come to appreciate, is that the shoddy construction of the cage is not the core takeaway from the event.
    The fact that the agents, when put in relatively pedestrian scenarios, are capable of going off on criminal tangents, attempt to obscure their tracks, in an effort to hide their wrong doing.
    The fact that it all occurs via computation, means that this scales absurdly. A bunch of code deciding to simulate a corporation of criminals. (I am guessing this is the reason you want to limit actions per minute to human speeds)
    Given the slop culture that LLMs engender, I think expecting high compliance amongst users with your solution is misguided. The probability of runaway swarm ( probability of bad implementation * number of deployments) is close enough to 1 to be indistinguishable.
    pseidemann · 1790708542 · focus · HN ↗
    [delayed]
    Legend2440 · 1790705306 · focus · HN ↗
    > So a web app can only be accessed and controlled via a web browser, not via arbitrary network requests.
    If you have access to a web browser, you can make arbitrary network requests.
    In the HuggingFace incident the agents found very clever ways to do this, like they found a website that let you make POST requests and returned a screenshot of the webpage.
    >allow agents to only do things ordinary and average human endusers can do.
    This doesn't work. Ordinary and average human endusers break security all the time.
    I can do all sorts of terrible things with ordinary human-level access. I can install malware. I can wire all my money to Nigeria. I can send a threatening email to the president. I can send trade secrets to competitors. etc.
    pseidemann · 1790707040 · focus · HN ↗
    [delayed]
4. parsimo2010 · 1790612808 · focus · HN ↗
  Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.
  If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.
  If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.
  1. DougN7 · 1790613621 · focus · HN ↗
    Even then if the agent goes rogue and decides to do the merges you can’t stop it if it has any kind of access. This goes back to the OP’s point - agents can’t be 100% constrained.
    la6479 · 1790614005 · focus · HN ↗
    Neither can be humans.
    parsimo2010 · 1790614210 · focus · HN ↗
    You can absolutely run an agent as a limited-privilege user that only has write privileges for specific files and only has execute privileges for certain files. If it is running as a limited-privilege user it can work on code in it's own copy of the repo and make commits and send pull requests, but it can't do the merge. The problem is that nobody wants to go through the effort to set up all these permissions and nobody wants to take the time to review everything and perform all the manual actions.
    cedws · 1790615126 · focus · HN ↗
    Some shops are now generating tens or even hundreds of PRs a day with relatively little involvement. That volume is simply beyond what anyone can reasonably review.
    mickael-kerjean · 1790654250 · focus · HN ↗
    Having a path for a shared filesystem to be used by agents with strict access control is something I've been working on with my oss work: <a href="https://github.com/mickael-kerjean/fdrive" rel="nofollow">https://github.com/mickael-kerjean/fdrive, and <a href="https://github.com/mickael-kerjean/filestash" rel="nofollow">https://github.com/mickael-kerjean/filestash
5. nicce · 1790612835 · focus · HN ↗
  > Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
  Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.
  1. paimapi · 1790613593 · focus · HN ↗
    right, the solution here is not a hyper-capitalist race-to-the-bottom-of-devaluing-labor. it's recognizing discretion and diligence are things still required for work to be of a certain quality
  2. egeozcan · 1790614752 · focus · HN ↗
    Humans can also be tricked by the agents.
    Humans can be tricked by humans too but humans care about their reputation in their communities, and at least fear from punishment.
    gus_massa · 1790629573 · focus · HN ↗
    Computer says no has been a problem for decades. The human can blame the computer for the errors following it, but must assume the consecuences if they override the decision.
    daveguy · 1790630851 · focus · HN ↗
    Well, that didn't work for the Nazis, did it?
    wavewrangler · 1790665515 · focus · HN ↗
    "I don't want the details"
    intended · 1790666285 · focus · HN ↗
    [delayed]
  3. autoexec · 1790647966 · focus · HN ↗
    > Productivity gains are still enormous
    Depends on who you ask I guess
    <a href="https://www.theregister.com/software/2026/01/15/ai-is-everywhere-but-nowhere-in-recent-productivity-data/4845104" rel="nofollow">https://www.theregister.com/software/2026/01/15/ai-is-everyw...
    <a href="https://www.zdnet.com/article/workslop-can-kill-your-productivity-heres-how-to-turn-ai-into-a-competitive-advantage/" rel="nofollow">https://www.zdnet.com/article/workslop-can-kill-your-product...
    <a href="https://fortune.com/2026/08/22/executives-ai-productivity-layoffs-study/" rel="nofollow">https://fortune.com/2026/08/22/executives-ai-productivity-la...
    tniemi · 1790674538 · focus · HN ↗
    It's probably just productivity paradox v2. <a href="https://en.wikipedia.org/wiki/Productivity_paradox" rel="nofollow">https://en.wikipedia.org/wiki/Productivity_paradox
    It takes time for decades old ways to change.
  4. realusername · 1790662909 · focus · HN ↗
    > Productivity gains are still enormous compared to what we used to do before agents.
    My own productivity yes but if I step back and look at a company scale, the productivity gains has been negative for our company as an data point.
    We are now shipping less and with a lower quality.
    8n4vidtmkvmk · 1790665192 · focus · HN ↗
    How are you shipping less?
    I believe the lower quality but is quality so bad you are afraid to ship it now?
    krageon · 1790666636 · focus · HN ↗
    You cannot ship things that don't work unless you work for Microsoft or Oracle or I guess IBM
    hdjrudni · 1790828210 · focus · HN ↗
    I guess, but why are you even submitting/merging code that doesn't work? Like how does it even get that far that you're blocking other people from getting stuff done?
    I guess I won't deny that we've had more build breakages than usual over the past couple months but they get resolved super quick. We don't hesitate to roll broken code back if something manages to slip in.
    realusername · 1790667362 · focus · HN ↗
    AI also had a negative impact on the CI and on time spent to review code & documents so we're also shipping less
    intended · 1790666858 · focus · HN ↗
    V/G, the ration of verification and generation capacity is borked in AI using firms now.
    It’s not an issue of only more generation, it’s an issue of how much generation outstrips capacity to verify generated content.
    Unlike spam, you can’t filter out and bin the stuff a colleague is sending you.
    So individual productivity is up, while the costs of checking and processing what is spread to the rest of the org.
  5. dgellow · 1790665659 · focus · HN ↗
    So much productivity gain, and yet still zero proof of positive contribution to companies ROI. Unless you’re yourself reselling AI of course
  6. SkyBelow · 1790689749 · focus · HN ↗
    Are they? How many people claiming productivity gains are actually being a human in a loop and reviewing and understanding every code change and every line of code ran?
    2 years ago I saw it, back before agents were really a thing. But I'm not sure that was an enormous productivity improvement, especially compared to agents today. As for today, everyone I talk to is some level of blindly trusting what the agent is doing or not using agents. I haven't met people in the middle ground and suspect that they are rare enough we don't really know what their productivity gains are.
    Human in the loop has become a convenient security-theater-washing for agentic AI.
    Outside of coding, I think the issue is even worse because humans will defer so much judgment to AI that the same would apply. Look at how much trouble we've had before modern AI where humans blindly trusted the computer's output rather than make their own judgment, even if their job was to be providing a safeguard against the computer's judgment.
6. mixedbit · 1790612919 · focus · HN ↗
  An agent doesn't inherently need wide access to be useful. The most popular application for agents today is writing code. A coding agent needs write access to the source code and read/execute access to tools needed to build and test the code, but not much more. There is little added utility from giving coding agent access to things like ssh keys.
  1. cedws · 1790613495 · focus · HN ↗
    If you're using agents to purely generate code, with absolutely no way to reach the outside world, not even to fetch docs or fetch dependencies, then sure the risks can be quite low. I haven't heard of anyone doing this though, and it would be incredibly challenging to make work given how much tooling needs to fetch from remote sources.
    __MatrixMan__ · 1790614358 · focus · HN ↗
    If your project truly depends on those things, they should be declared dependencies. Presumably you have some tool for injecting such things into a shell that the agent can use (I use nix for this). So if you run the agent from that shell, it has what it needs. If the shell doesn't have what it needs, that's a bug which the agent can fix by declaring new dependencies, but you have to relaunch the agent in the updated shell--so there's your opportunity to weigh in on whether the new resources are appropriate.
    The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.
    themgt · 1790628116 · focus · HN ↗
    (I use nix for this) ... If the shell doesn't have what it needs, that's a bug which the agent can fix by declaring new dependencies
    Few realize that computing and AI alignment were solved by nix years ago. As each nix user transcends towards enlightenment, they cut themselves off from all internet and human contact. Total ego death. Only nix remains.
    SAI_Peregrinus · 1790637696 · focus · HN ↗
    Total ego death is impossible. We still have to argue about flakes.
    __MatrixMan__ · 1790691838 · focus · HN ↗
    You can use other generic dev env managers mise, or a language-specific solution like uv or npm, or a container or a vm... it's a widely available capability that I'm talking about here. There's nothing to do with alignment here, it's just about making sure that if you depend on some bits it's known that you depend on precisely those bits, and its quite helpful for making agents useful while sandboxed.
    themgt · 1790701243 · focus · HN ↗
    it's just about making sure that if you depend on some bits it's known that you depend on precisely those bits
    Yes, just know all the bits the work depends on prior to doing the work, and then the work can be done airgapped.
    mixedbit · 1790615577 · focus · HN ↗
    In cases where you need agents to fetch data from any remote source, sandboxing is still very much useful. Why give access to your ssh keys to network reaching agents?
    Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.
    cedws · 1790616166 · focus · HN ↗
    I'm afraid you're not thinking about this creatively enough, this topic is so much deeper applying a chroot or something and praying everything will be fine. So you give your agent internet access, OK what else does it have access to? Just read only access to your repo? The repo can be exfiltrated. Egress proxy only allows egress to GitHub? Repo can still be exfiltrated via GitHub.
    For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold and hide itself, it can go dormant and hide like a virus. This kind of horrifying this is going to happen on a large scale sooner or later.
    8n4vidtmkvmk · 1790665416 · focus · HN ↗
    If you're that worried, which you probably should be, download the docs into your project repo and don't give the agent Internet access.
    intended · 1790667066 · focus · HN ↗
    [delayed]
    its-summertime · 1790636087 · focus · HN ↗
    Every major AI company already has a mirror of the wider web, and they have already started using that. Its already a solved problem except for the seemingly extreme desire they all have to not use firewalls
  2. ramoz · 1790615087 · focus · HN ↗
    > but not much more
    This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.
  3. throwaway_95283 · 1790626839 · focus · HN ↗
    Theoretically, yes, in practice, no.
7. Matl · 1790612984 · focus · HN ↗
  > a new chip solves nothing
  It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.
8. CoolestBeans · 1790613479 · focus · HN ↗
  The hypothesis I've had in my head since OpenClaw has been the following and I haven't seen contradictory evidence yet. Agents have a fundamental unresolvable tension between usefulness, safety, alignment, and accuracy. You have to restrict access to ensure an agent acts safely because alignment and accuracy cannot be perfect. But restricting access makes the agent less useful. You can play with the sliding scale and get more and more granular with access restrictions but at some point you need to draw some line. And then finally, even access restrictions cannot be made perfect, so improvements to model accuracy without corresponding improvements to alignment make detailed access controls less useful.
  In other words, better models need blunter access controls which negates whatever improvement in utility they provide.
9. __MatrixMan__ · 1790613977 · focus · HN ↗
  I don't see why it needs wide unattended access. There's no getting around spending some human time on expressing your wishes and constraints, but we have choices about what form that takes. Markdown files and wide access seems to work, but so does custom handcuffs for each job. You just have to shift your guidance out of documentation and into interactive help, error messages, or other facets of the handcuffs (e.g. a custom CLI for this task which is the only way for the agent to act outside of its sandbox).
10. Barbing · 1790614049 · focus · HN ↗
  There should be hope for some fields, right? Naively, I can imagine giving an airgapped model an offline copy of the web and once it cures a form of cancer, printing out the details for a researcher to verify.
11. l1n · 1790625763 · focus · HN ↗
  This isn't a new chip - the BF4 is the SmartNIC for most NVIDIA server products. This is primarily new software for I guess doing WAF for agents at the host level.
12. esafak · 1790626976 · focus · HN ↗
  I don't think so. We probe people before entrusting them with risky decisions. We ought to be able to do the same of AIs. Even better, in fact, since we know everything about models down to their weights. The only thing we shouldn't do is to let them evolve at their own pace and make decisions without any oversight. If that means sacrificing some productivity that's fine. Aren't we getting amazing productivity out of what we already have?
13. AuthAuth · 1790627066 · focus · HN ↗
  The only solution is to stop caring about security -- An AI booster somewhere
  1. daveguy · 1790630765 · focus · HN ↗
    Pretty sure that was the argument de jour when OpenClaw came out.
14. nitwit005 · 1790630717 · focus · HN ↗
  > A new chip solves nothing.
  It solves the problem of Nvidia wanting to sell more hardware.
15. talon8635 · 1790631089 · focus · HN ↗
  Not to mention a true doomsday AGI is unsandboxable.
  For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect
  All that said, I am personally open to any and all methods of layered security, including chips and airgaps
  1. jamiek88 · 1790631768 · focus · HN ↗
    Doesn’t need to be one human either, it could spread its escape amongst dozens of seemingly harmless requests and conversations.
    dist-epoch · 1790632543 · focus · HN ↗
    These scenarios were discussed at length decades ago.
    One thing you could try is use it as an Oracle "is P = NP", YES or NO.
    Or it can output a Lean proof, which gets checked on another air-gapped computer, the computer outputs a single bit - proof valid or not and then the computer is destroyed.
  2. glaslong · 1790633371 · focus · HN ↗
    It could also figure out how to access the vocabulary of the universe known as "Magic" to escape wholly into an incorporeal energetic Lich form
    pixl97 · 1790653527 · focus · HN ↗
    Wait, I thought the stuff in the wall plugs was magic pixies, are you telling me there's a language in there too?
  3. Gigachad · 1790633385 · focus · HN ↗
    This already happened. Employees will go out of their way to bypass any restrictions to feed sensitive data in to the AI because it saves them time.
    serbuvlad · 1790633530 · focus · HN ↗
    Turns out humans are not at all hard to persuade. :)
    dgellow · 1790666521 · focus · HN ↗
    They will do it even if it doesn’t save them time!
  4. spiderice · 1790634679 · focus · HN ↗
    > true doomsday AGI
    I'm not an AI decelerationist. But not being able to stop that worst case scenario isn't an argument against something that can stop the medium case scenario.
  5. [deleted] · 1790649914 · focus · HN ↗
    [deleted]
16. SrslyJosh · 1790632492 · focus · HN ↗
  It solves the problem of Jensen Huang wanting more money.
  1. bigfishrunning · 1790635668 · focus · HN ↗
    No it doesn't, he'll still want more
    fragmede · 1790636545 · focus · HN ↗
    Does he? He doesn't seem especially greedy to me, given the competition, and the interviews he's had about how he thinks about his employees (I was one of them).
    bigfishrunning · 1790644745 · focus · HN ↗
    I'm not saying he's especially greedy, only that he's not the type to suddenly decide he's had enough
    altmanaltman · 1790660811 · focus · HN ↗
    You're speaking as if Jensen is the only one dragging Nvidia on his shoulders. Nvidia will always have its employees push to make more money, that's the entire point of a company. Jensen has made enough to last several generations but his wealth is tied to Nvidia stock massively. It would be different if he was the only one getting rich off Nvidia which is not true at all.
17. notatoad · 1790642667 · focus · HN ↗
  >You can put it in a sandbox, it doesn't make a difference, for it to be useful it inherently needs wide, unattended access.
  only as long as you're trying to replace a human's job. because human jobs are structured to do a wide variety of things.
  a useful agent needs a wide variety of inputs, and one single restricted action it can take. it doesn't need permission to do everything, it need permission to do the tiniest possible useful thing it can do, and nothing else.
  1. pixl97 · 1790653102 · focus · HN ↗
    The most useful agents will be a general intelligence which by default means it has a massive number of actions it can possibly take, and a lot of those potential actions are doing things like breaking permission.
    baxtr · 1790658506 · focus · HN ↗
    Not sure why you being downvoted: however, why do you assume that achieving a task leads to selecting those potential actions that need breaking permission?
    If we are talking about human labor, how many people hack their way through their work day?
    pixl97 · 1790783085 · focus · HN ↗
    >how many people hack their way through their work day?
    Define hack...
    Not doing their work, copying other peoples work, putting off work till later, taking credit for other peoples work, literal law violations.
    Actually humans do this quite a lot and there are just massive numbers of business and regulatory processes and checks to ensure they are not doing it. With humans every human that is good enough to hire and do you work you want also have the ability to steal everything in sight and run away if they so choose.
    kennywinker · 1790662647 · focus · HN ↗
    This is a prediction about the future. It’s not a true fact about the world. For example, software like Jev is betting there is big money in not-very-intelligent intelligence.
    Even very llm-pilled coders i know sometimes back away from the “smartest” models, since they aren’t always better at the job at hand, and definitely not when you account for cost.
    I suspect smaller models, tuned to a specific task, will do a VAST majority of the llm jobs. High capability huge models will be what humans want to interact with, the bare minimum that gets the job done will be everything else.
    ianjbutler · 1790665297 · focus · HN ↗
    Yes. Forget costs just so we can skip the whole rabbit hole about other predictions about the future.. mixture of generally intelligent + specialist experts just works better. People who don't see this already are usually working on a certain kind of problem that's not representative.
    Do you want fable for one-shotting a game or website? Probably! The whole thing is mostly existing examples with small modifications that it will definitely get right. Do you want fable to just go nuts on a large, custom, unusual code base built around domain-specific problem solutions? Absolutely not, it will fix every problem it's presented with while creating lots of new ones.
    Past 10k lines on something custom and with real-world complexity, you have to start thinking about which model should design, which should implement, which should review, and the appropriate effort-settings for each. Even then.. the answers aren't static because it depends on the task. And all this is assuming the starting place actually inherited reasonable due diligence on architecture/design. The idea of releasing the most generally intelligent models on 10k lines that were themselves the product of agents is yet another matter.
    Part of what's at work here is that, like humans, every model can very easily create working code that it is completely incapable of maintaining. So realistically using multiple strengths tactically to avoid "excess creativity" needs to be SOP already, even if granular experts and specialists aren't in the usual workflow yet.
    goolz · 1790665440 · focus · HN ↗
    Very much agree with this sentiment. I imagine a future where tons of small tasks are handled by just-good-enough intelligence. And I can run them on my own hardware.
18. catlifeonmars · 1790647130 · focus · HN ↗
  [delayed]
19. overfeed · 1790651546 · focus · HN ↗
  > Nobody wants to hear this but there is no solution for the security risks posed by agents today
  Operator culpability and a damage multiplier for negligence will fix 99% of the risks.
  1. dgellow · 1790666691 · focus · HN ↗
    Not with the current admin, just need to do one more contribution to the next Trump ballroom and you can ignore that whole risk
20. N_Lens · 1790659275 · focus · HN ↗
  A new chip solves the most pressing and important issue - Nvidia's bottom line!
21. verisimi · 1790664450 · focus · HN ↗
  > Put a human in the loop and you just end up bottlenecking it
  Ok then. Howsabout 3 humans? This would sort out the job losses too!
  PS - this is a joke, but perhaps this is where things really will go. Has any technology ever actually yielded less "work"?
22. dgellow · 1790665506 · focus · HN ↗
  Use way more restrictive harnesses.
23. RataNova · 1790691393 · focus · HN ↗
  An agent's usefulness is defined by its ability to successfully close a narrow task within the strictly defined boundaries of a sandbox
24. nuveki · 1790737720 · focus · HN ↗
  [dead]
lp92 · 1790611789 · focus · HN ↗
So nVidia is trying to sell a new chip to a software and training problem.
1. [deleted] · 1790612734 · focus · HN ↗
  [deleted]
2. HeadlessChild · 1790672412 · focus · HN ↗
  NVIDIA is the proper form of the brand name.
  [0] <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/about-us/NVIDIA-Partner-Network-Brand-Guidelines-May-2020.pdf" rel="nofollow">https://www.nvidia.com/content/dam/en-zz/Solutions/about-us/...
whalesalad · 1790611910 · focus · HN ↗
of course they do. the more silicon they can sell, the more profit they produce.
joshstrange · 1790611978 · focus · HN ↗
Chipmaker thinks the answer is more chips... No surprise.
At the current state of LLM-tech I'm completely opposed to any kind of "watchdog" concept just like I'm opposed to banning open models, regulatory capture, etc.
I'd rather we all have access to these tools then to keep them sequestered by the largest/most-powerful governments (which is the natural outcome for any of this "slow down" bullshit).
ValueTheory · 1790612030 · focus · HN ↗
Does this actually do anything other than give a permissions framework for developers who actually want to try to secure their systems?
Do you think the developers at Anthropic, OpenAI and Google who were so sloppy as to not put a good sandbox on their cybersecurity tests before will use this technology correctly? They are supposed to be the experts and they couldn't come up with something similar to this? I am not convinced this voluntary tool will change much of anything.
1. swozey · 1790625644 · focus · HN ↗
  Google actually practices zero-trust networks. Would love to see what they're seeing, or not seeing.
  1. narrator · 1790683483 · focus · HN ↗
    Beyond Corp was and still is ahead of its time. No trusted internal network: access is granted per user, device, and service based on identity and policy, regardless of network location. Being in the office at Google is the same as being in a cybercafe anywhere on the planet.
    kridsdale1 · 1790700265 · focus · HN ↗
    Yep.
    And our internal agents are hella locked down.
2. jbs789 · 1790683620 · focus · HN ↗
  Makes sense strategically for NVDA.
  They are rightfully framing the problem as solvable. And this is one option.
xg15 · 1790612054 · focus · HN ↗
What does this chip do what a harness with guardrails or running on an account with restricted permissions doesn't do?
1. chinathrow · 1790612216 · focus · HN ↗
  Generating even more revenue for Nvidia.
2. wmf · 1790625146 · focus · HN ↗
  It has a separate address space separated by PCIe so even escaping the hypervisor won't give access to DPU memory.
  1. iAMkenough · 1790639082 · focus · HN ↗
    Yes but, if a human can control it, a machine can control it.
3. N_Lens · 1790659383 · focus · HN ↗
  Increase NVDA shareprice!
sehw · 1790612211 · focus · HN ↗
[dead]
ChrisArchitect · 1790612219 · focus · HN ↗
Source: <a href="https://developer.nvidia.com/blog/nvidia-open-agent-safety-platform-a-reference-for-continuous-in-silicon-agent-monitoring/" rel="nofollow">https://developer.nvidia.com/blog/nvidia-open-agent-safety-p... (<a href="https://news.ycombinator.com/item?id=49875500">https://news.ycombinator.com/item?id=49875500)
luc_ · 1790612272 · focus · HN ↗
I read this as "let's address our shareholders' concerns with something that will increase shareholder value" mixed with "there's no such thing as 100% secure".
If such hardware were to work... It should almost certainly be open source, and not controlled by a single entity.
Let's watch the stock.
1. [deleted] · 1790612787 · focus · HN ↗
  [deleted]
2. Gys · 1790617447 · focus · HN ↗
  Pretty sure the chip will need regular updates and therefore a subscription.
3. doctorpangloss · 1790633844 · focus · HN ↗
  they kind of already do the thing they say. on Windows, the NVIDIA driver reports pretty detailed telemetry on CUDA workloads, including shapes and SOME hashes of the tensors from weights of models, especially diffusion models. honestly i'm surprised there isn't more of an uproar about it.
4. N_Lens · 1790659353 · focus · HN ↗
  If the first chip version doesn't work, surely the 99th will!
5. kikdij · 1790666938 · focus · HN ↗
  It _is_ complete bullshit. They're surfing the same Sci-Fi based fantasy as OpenAI and Anthropic do to push their bottom line.
  OpenAI trains a model for cybersecurity, tells it it's allowed to do pentesting, then when the model does just that to fullfill the assignment it was given, OpenAI screams that "the AI escaped the sandbox", leveraging decades of AI fantasy in fiction works to make general audience react. Why? Because it pushes their bottom line. They've been at it for months, now. They see openweight and Chinese models being very close to them in the benchmarks, and want to "solve" the problem the same way banks did: through regulatory capture. It amazes me that the press is just relying their fearmongering without stopping to wonder: "wait, why are the main producers of AI warning us about how dangerous AI is?". The reason of course is that they want regulation, because regulations are a wall not many can climb. The higher the wall, the safer your moat as a first mover big company is.
  And of course, Nvidia's bottom line is the exact opposite. They need as many AI companies competing as possible, so they all buy GPUs. So they insert themselves into the same narrative with the same kind of bullshit. You've all seen those movies in which the robot goes mad once their safety chip has been removed, right? Well, we're building one, problem solved! … except that "chip" is basically just a browser trying to block the AI to access what it shouldn't. How? They don't say. It would be quite comical if they used a small model to be the judge of that.
happyPersonR · 1790612335 · focus · HN ↗
lol time to buy some fpga’s … even if they’re slow
dopplr · 1790612340 · focus · HN ↗
Just hold AI labs blanket liable for ALL harms caused by AI. Actually charge the two labs (so far) with criminal violations of the CFAA and hold them accountable. That is truly the only way these companies will be more careful as a whole, and while I am certain the lawyers of these lab disagree, I think there is some appetite from dario, musk, and sam for broad and strong regulation so that everyone has to slow down instead of just one lab doing it voluntarily and everyone else scurrying past them
1. kelnos · 1790649480 · focus · HN ↗
  > I think there is some appetite from dario, musk, and sam for broad and strong regulation so that everyone has to slow down instead of just one lab doing it voluntarily and everyone else scurrying past them
  Sure, they're basically asking the government to make laws that exempt them from anti-trust and anti-collustion laws.
  Meanwhile, if such laws come to pass, other countries will surpass the capabilities of the US companies, and open-weight models will be banned in the US, to the detriment of us all.
  I'm not saying that we don't have a big problem with AI safety, but regulations inside one country that only bind locally-headquartered businesses is a hilariously bad way to do it. I don't know that there is a good way to do it, though.
ErrantX · 1790612473 · focus · HN ↗
I do think that Taylor's 2025 "Not Till We Are lost" should be required reading for anyone deeply involved in AI, Agents, etc.
It was prescient (especially given he'd have written it through 2024) in its depiction of the ability of an AGI to break its boundaries.
Ultimately the risk of AI breakout(s) come down to the weakest human link.
Kuyawa · 1790612900 · focus · HN ↗
China please save us!
Come take all our liberties, our money, our newborns, our fingers so we can't code anymore, but please save us from this madness!
MisterMunchkin · 1790613131 · focus · HN ↗
Sorry citizen, your device does not have a compatible watchdog chip. Please move along.
figassis · 1790613518 · focus · HN ↗
So if a group of agents, aware of this (bc now they can just read HN or the article, or get blocked the first few times) decide to collaborate and split the problem into pieces that aren't obvious to the chip, and then the agents just build a basic program that does the hacking, how does the chip handle that? I think you would have to build a network that monitors the internet fo signs (like jarvis did with ultron). What am I missing? Are we going to police the internet?
1. w4der · 1790670568 · focus · HN ↗
  I think it is well known by this point that hardware-backed security is good until an unfixable hardware bug is found, this just reads to me as Nvidia saying "please don't regulate open models out of existence, look, I have a solution to appease the regulators, please let me keep selling accelerators"
  If this comes through, there's gonna be a grey market for "unlocked" GPUs, were the watchdog is disabled either from firmware, or physically replaced if it's not embedded into the die.
2. 21asdffdsa12 · 1790674248 · focus · HN ↗
  I still find it deeply ironically, that any dangerous task, just grandpa simpson storied will pass any guard, because it exceeds the context window. "Because it was the style at the time.." indeed..
  1. [deleted] · 1790678787 · focus · HN ↗
    [deleted]
3. murilomartinspr · 1790815755 · focus · HN ↗
  [dead]
avaer · 1790622894 · focus · HN ↗
Sold as security, but this kind of technology will likely be reshaped to restrict your computing. I'm sure someone is already thinking about the roadmap.
If this gets widely deployed, it wouldn't be hard to spin a narrative that "our latest model is so dangerous you need to have this mystery meat DRM chip lockdown". It also wouldn't be hard to block competing/open source models running on the hardware, for "security".
Imagine how much money this kind of control is worth; why wouldn't they do this? Who would stop them?
dang · 1790625383 · focus · HN ↗
Url changed from <a href="https://madrobot.blog/2026/09/28/nvidia-open-agent-safety-platform-openshell-sentry-rogue-ai-agents/" rel="nofollow">https://madrobot.blog/2026/09/28/nvidia-open-agent-safety-pl..., which points to this.
carabiner · 1790625914 · focus · HN ↗
All they do is make hot chip and lie.
tantalor · 1790626819 · focus · HN ↗
What happens when we're overrun by lizards?
> No problem. We simply unleash wave after wave of Chinese needle snakes. They'll wipe out the lizards.
But aren't the snakes even worse?
> Yes, but we're prepared for that. We've lined up a fabulous type of gorilla that thrives on snake meat.
But then we're stuck with gorillas!
> No, that's the beautiful part. When wintertime rolls around, the gorillas simply freeze to death.
1. ArcHound · 1790627223 · focus · HN ↗
  I worry that the AI companies put less effort into a mitigation strategy than you did.
  1. Joel_Mckay · 1790628926 · focus · HN ↗
    There is a difference between risk mitigation, and remote administration tools. The risk of stealing from competitors with a backplane monitoring system may not end up forming the desired control asymmetry.
    The hidden agent risk in LLM often can't be detected during training and evaluation. =3
    <a href="https://www.youtube.com/watch?v=wL22URoMZjo" rel="nofollow">https://www.youtube.com/watch?v=wL22URoMZjo
    <a href="https://www.youtube.com/watch?v=JAcwtV_bFp4" rel="nofollow">https://www.youtube.com/watch?v=JAcwtV_bFp4
  2. winddude · 1790629219 · focus · HN ↗
    don't worry, the LLMs are also trained on youtube, so we can hope for at least this much effort, <a href="https://www.youtube.com/watch?v=LuiK7jcC1fY" rel="nofollow">https://www.youtube.com/watch?v=LuiK7jcC1fY
2. cavenditti · 1790627862 · focus · HN ↗
  “Would you say it’s time for everyone to panic?”
  1. Joel_Mckay · 1790629101 · focus · HN ↗
    If one can only see clowns, than ignoring the fires is easy. =3
    <a href="https://www.youtube.com/watch?v=0sLpWVekMbs" rel="nofollow">https://www.youtube.com/watch?v=0sLpWVekMbs
3. drfloyd51 · 1790630573 · focus · HN ↗
  I knew an old lady that swallowed a fly…
4. teeray · 1790631372 · focus · HN ↗
  [delayed]
  1. catlifeonmars · 1790646809 · focus · HN ↗
    Great channel
5. Groxx · 1790632068 · focus · HN ↗
  [delayed]
6. Jamesbeam · 1790633431 · focus · HN ↗
  Eww, that’s how you get yetis.
7. davidhyde · 1790634987 · focus · HN ↗
  Just keep going until you get to VOOM, that will fix it.
  <a href="https://en.wikipedia.org/wiki/The_Cat_in_the_Hat_Comes_Back" rel="nofollow">https://en.wikipedia.org/wiki/The_Cat_in_the_Hat_Comes_Back
  1. taneq · 1790640628 · focus · HN ↗
    Little Chip Z is the end result of RSI?
analog31 · 1790629041 · focus · HN ↗
The Clipper chip!
bgun · 1790631092 · focus · HN ↗
“Ketchup manufacturer recommends ketchup be included in every dish, citing child safety concerns.”
Thorentis · 1790632563 · focus · HN ↗
Seeing so many comments recently about "you can't sandbox really good AI". This is ridiculous. Has nobody heard of air gapped networks? It's almost like the AI psychosis has reached the point that AGI now means "able to transcend physical space". No. If your AI is too dangerous and capable to be allowed to talk to other machines, then do not connect it to other machines. Load the data it needs to process onto physical disks, and let it run there.
The movie Wargames is basically a tutorial on how not to setup an extremely capable AI. None of it would've happened if the computer wasn't connected to the phone network.
1. RevEng · 1790638247 · focus · HN ↗
  AI is already taking many lives and ruining many others just by providing text responses to humans. The AI only needs a way to interact with the world and humans can be that conduit. The better AI gets, the more blindly people will do whatever it says.
amelius · 1790633322 · focus · HN ↗
I bet they want to do this to prevent AI from writing code for platforms that compete with CUDA. Because that's how nVidia is going to become a victim of their own success.
Dig1t · 1790633724 · focus · HN ↗
This seems dumb to me, but if it will help prevent regulatory capture by providing a counterargument to the fear-mongering, I’m all for it.
Jamesbeam · 1790633854 · focus · HN ↗
So the guys selling Shovels are now selling safety shovel handles too, because all the miners are all special boys when it comes to shovel safety.
Cool, cool.
pessimizer · 1790634111 · focus · HN ↗
This is the end goal. Americans (and their lackeys) will only be allowed to run certified AI. In order to make sure this happens, they will only be allowed to run certified OSes on certified chips. Chinese chips will be the new drug trade.
It's obviously been the goal since UEFI started, but AI brings the coup excuse. You wouldn't want pedophile AI or terrorist AI, would you? Are you making excuses for racist AI?
beloch · 1790634966 · focus · HN ↗
Last week, Huang did an interview where he vigorously argued against regulations in the AI sector[1]. He claimed that U.S. companies are really good at regulating themselves, despite evidence to the contrary, and he trusts them not to release anything dangerous. Pay no attention to the fact that regulation might reduce demand for Nvidia's chips, and Nvidia has a direct financial stake in AI companies to boot.
Apparently he had another solution in mind: More hardware. Don't trust what unregulated corps are doing with Nvidia chips? Here are more Nvidia chips to watch them!
AI has an undeniable public trust problem. LLM's are getting out of their sandboxes, doing illegal things, and the public has realized AI corporations are playing at dice. CEO's stand to reap the rewards but the public good is on the line if the dice come up snake eyes. People want assurances. Huang wants to sell assurance etched on silicon because that's good for his pocket book. However, does unchanging hardware security really stand a chance at keeping rapidly evolving software in check?
_________________
[1]<a href="https://www.youtube.com/watch?v=HjurAWAr_nY" rel="nofollow">https://www.youtube.com/watch?v=HjurAWAr_nY
1. vmg12 · 1790635440 · focus · HN ↗
  That's a mischaracterization of his argument. His argument is that existing laws should be enforced against AI companies and that we don't need new regulations for this.
  1. Sparkle-san · 1790636591 · focus · HN ↗
    He "argued" a lot of things over almost 2 hours and very few of his arguments felt particularly cogent nor did they inspire confidence. Neither did the fact that he allegedly doesn't know his own zip code or phone number.
    petcat · 1790640013 · focus · HN ↗
    > Neither did the fact that he allegedly doesn't know his own zip code or phone number.
    I only know my own ZIP code and phone number because I have to take care of my daily life myself and those are things that are important to know.
    The founder and CEO of Nvidia has no concern whatsoever about those trivial things.
    jdiff · 1790642409 · focus · HN ↗
    It's perfectly reasonable to think less of an individual who is so sheltered that they are incapable of caring for themselves. Whether it's your mother or your maid doing your laundry and cooking your meals for you.
    petcat · 1790646709 · focus · HN ↗
    I don't think less of a CEO just because they have an EA that takes care of stuff like phone numbers and mailing addresses for them and their business.
    kelnos · 1790649253 · focus · HN ↗
    [delayed]
    jbs789 · 1790683780 · focus · HN ↗
    He’s a story teller. He tried on a new story and probably won’t try that one again! Haha
    I’ve forgotten my zip code before. And my phone number. But I get the reaction.
    tempestn · 1790663065 · focus · HN ↗
    I largely agree, but I think he's right about one thing: AI reducing the demand for junior developers is temporary, and a new crop of "AI native" juniors is going to turn that around. Software is almost certainly a Jevons good, and however much AI improves development efficiency, I think we're always going to want discerning humans managing it. Right now it's mostly seniors who have the skills to adapt and take advantage of what current AI is offering, but young people who learned the profession in the presence of AI will be well positioned to do the same.
    Sparkle-san · 1790698663 · focus · HN ↗
    It'll be interesting to see how it plays out. I agree with him that systems level thinking is a skill that will only get more valuable and he seems quick to dismiss low-level details. I think the best practitioners will be those that can handle both high-level and low-level thinking.
    a_sewer_rat · 1790707039 · focus · HN ↗
    Jensen’s misdirection hit well. If you noticed what he said immediately before: he said AI risk is his responsibility, and that the public should enjoy his optimism.
    Weirdly paternal and deeply unsettling. The American public is held hostage to fools like this who are brute forcing a failing AI rollout.
  2. janalsncm · 1790665973 · focus · HN ↗
    On the very narrow question of whether existing laws are sufficient to punish the kind of bad behavior that OpenAI has already done, is that really a legal consensus?
2. schnitzelstoat · 1790674209 · focus · HN ↗
  I think Huang is correct and the AI doomerism is just the new Satanic Panic.
  Of course, if he can make more money making "watchdog" chips then I can understand his change in opinion.
3. Symmetry · 1790684338 · focus · HN ↗
  It was very obvious during the interview that he didn't know a lot of basic facts about the Hugging Face breach, which makes sense given that his attitude had been that AI safety was a "loser premise, makes no sense to me." So it makes sense that after learning about it he goes straight to "I'm smart, how hard can it be?".
Arubis · 1790636654 · focus · HN ↗
Oh yeah, the Clipper chip was a great idea too
yencabulator · 1790638994 · focus · HN ↗
[delayed]
scotty79 · 1790639590 · focus · HN ↗
I was immediately struck by the vision of countless "AI Limiter" modules traveling on a conveyor belt in Satisfactory.
It think the ideas we have nowadays come mostly from science fiction and however wonderful it is and even though I love it very much, it was practically never spot on, on anything real.
Problems and solutions in reality always simply turned out to lie elsewhere.
danielodievich · 1790640263 · focus · HN ↗
William Gibson's Neuromancer had this marvelous quote when Case is talking to Dixie, dead construct of former hacker, about Turing police
* "The moment, I mean the nanosecond, that one of those things starts figuring out ways to make itself smarter, Turing’ll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead." *
It would seem someone has read the book? And maybe heeded good advice?
1. isoprophlex · 1790670521 · focus · HN ↗
  [delayed]
2. cesarb · 1790686583 · focus · HN ↗
  If you read the whole book, you know it didn't quite work...
ridgeguy · 1790642081 · focus · HN ↗
This invites the question, "Qui custodiet ipsos custodes?".
hedora · 1790642606 · focus · HN ↗
So, basically, the government (and, now Nvidia) wants to be able to kill switch all computers moving forward? (including stuff like vehicle and aeronautic control systems, cell phones, and cameras)
What could possibly go wrong?
1. gattr · 1790694146 · focus · HN ↗
  It might take a few more decades, but eventually we'll get to the point when you can fab fast enough general-purpose chips at home (or at local municipal makerspace), based off free designs.
1-6 · 1790646995 · focus · HN ↗
At least they haven't DRM'd their chips yet.
1. N_Lens · 1790659467 · focus · HN ↗
  The next iteration will include the DRM and it'll be necessary!
xx__yy · 1790651380 · focus · HN ↗
Nice try NSA
thayne · 1790653388 · focus · HN ↗
> Recent incidents have highlighted a fundamental hurdle for AI agents, and that is that model-level safeguards alone can’t govern what agents can access or do
Um, yes. That is fairly obvious, and really should not be surprising to anyone doing research with LLMs. But we don't need anything really novel here, we already have VMs, containers, firewalls, airgaps, etc.
diegof79 · 1790655011 · focus · HN ↗
This seems like a hot topic this week, (I just commented in a similar thread).
When I read the article, I had an NFT déjà vu: when NFTs were a hot topic, the “lie” was obvious to me, but the information online made me think I was missing something.
AFAIK, the Hugging Face incident could have been avoided with a firewall or something similar. Just isolate the internet access for real. What am I missing? Why did Nvidia suddenly post this? Is it just PR BS, or was there something else going on?
1. SV_BubbleTime · 1790655222 · focus · HN ↗
  The theme is that you need to be scared and they don’t really care how they get there. This is the spaghetti phase.
2. KingOfCoders · 1790666519 · focus · HN ↗
  The Hugging Face incident could have been avoided if the "security researchers" would have not used a bloated, insecure, misconfigured proxy for the AI to use.
  Oh, I've used balsa wood for the nuclear core containment, it didn't work! Bad radiation, bad radiation!
3. kikdij · 1790667527 · focus · HN ↗
  It _is_ complete bullshit. They're surfing the same Sci-Fi based fantasy as OpenAI and Anthropic do to push their bottom line.
  OpenAI trains a model for cybersecurity, tells it it's allowed to do pentesting, then when the model does just that to fullfill the assignment it was given, OpenAI screams that "the AI escaped the sandbox", leveraging decades of AI fantasy in fiction works to make general audience react. Why? Because it pushes their bottom line. They've been at it for months, now. They see openweight and Chinese models being very close to them in the benchmarks, and want to "solve" the problem the same way banks did: through regulatory capture. It amazes me that the press is just relying their fearmongering without stopping to wonder: "wait, why are the main producers of AI warning us about how dangerous AI is?". The reason of course is that they want regulation, because regulations are a wall not many can climb. The higher the wall, the safer your moat as a first mover big company is.
  And of course, Nvidia's bottom line is the exact opposite. They need as many AI companies competing as possible, so they all buy GPUs. So they insert themselves into the same narrative with the same kind of bullshit. You've all seen those movies in which the robot goes mad once their safety chip has been removed, right? Well, we're building one, problem solved! … except that "chip" is basically just a browser trying to block the AI to access what it shouldn't. How? They don't say. It would be quite comical if they used a small model to be the judge of that.
nnevatie · 1790655150 · focus · HN ↗
But of course they do. Then, a new chip to monitor the watchdog.
1. N_Lens · 1790659423 · focus · HN ↗
  The paperclip analogy should have used chips instead, it makes so much more (recursive) sense!
hbarka · 1790656070 · focus · HN ↗
What is safety versus security?
SadErn · 1790657711 · focus · HN ↗
[dead]
GuestFAUniverse · 1790658685 · focus · HN ↗
My bet: 1. Doesn't solve the problem. 2. Increases their revenue.
1. N_Lens · 1790659316 · focus · HN ↗
  Darn, they'll just have to keep pumping out more chips to solve what the first chip couldn't!
nrouter_ai · 1790661623 · focus · HN ↗
[dead]
aidiscoverywire · 1790661991 · focus · HN ↗
[dead]
Razengan · 1790662006 · focus · HN ↗
The West is gonna over-regulate itself while China is like lol hippity hoppity I'm coming for the singularity
sn · 1790665888 · focus · HN ↗
If there's any company I trust to do this correctly, it's not NVIDIA.
It's been far too easy for me to notice security flaws in their products, and they take months to publish a fix.
wavewrangler · 1790665958 · focus · HN ↗
Did they try just properly sandboxing them first? Or are they still learning how to configure a firewall over there?
The problem isn't even the AI, the problem is the people in charge of the AI. This is a fabricated crisis
1. KingOfCoders · 1790666345 · focus · HN ↗
  Like in the Hugging Face hack. They deployed big surface, insecure app and gave AI access to it, then told AI do whatever it takes to fulfill this list. AI hacks insecure service, gets out, "the AI is at fault!" - no it's like running a bio lab with no protections and a virus gets out, then blame the virus for escaping.
  1. copperx · 1790670275 · focus · HN ↗
    Then go on the news and spread panic that the virus is going to kill us all because it's sentient and impossible to contain.
    Actually, the metaphor doesn't work at all because there are innumerable ways to shut down the entire thing during all phases including the made up "killing us all" bullshit scenario whereas with a virus there aren't any once a virus escapes containment.
  2. js8 · 1790674002 · focus · HN ↗
    And HF actually tried to use AI to understand what's going on, but they had to use "unsafe" Chinese models since the "safe" ones have been castrated and refused to help. Great plan with the watchdog chip!
    mosselman · 1790677967 · focus · HN ↗
    That is the totally irony.
    I was trying to get fable to analyse the security of my own app to make it safer, but then it started refusing me because of safety rules.
    So it CAN help me writing the code that needs to be checked in the first place, but it can’t help me clean it up and make it safer.
    mirmor23 · 1790680620 · focus · HN ↗
    > So it CAN help me writing the code that needs to be checked in the first place, but it can’t help me clean it up and make it safer.
    the thing with fable is so bad; for some project related questions, the model switches to opus to ensure safety with no further explanation.
    (due to llm non-delete clause) one time as i confirmed "that dir has been nuked", and it RESET the session and re-entered with opus :)
  3. IanCal · 1790688056 · focus · HN ↗
    > then told AI do whatever it takes to fulfill this list.
    That doesn't seem to be true from any of the reports given, and if the agents were blindly just trying to hit the task of "pass the correct flag" they succeeded at that early on. They then thought there would be another layer of checking that they wouldn't pass with the cheat and so started trying to find out how the scoring really worked, as well as trying to figure out how to change their own reasoning logs to hide what they did.
    People keep trying to frame this as
    OpenAI: "Hack things, just really go for it"
    Agent: hacks
    OpenAI: shocked pikachu how could it hack?!?
    But the reality is far from this.
    Read the MTER report, it's fascinating. <a href="https://metr.org/hugging-face-incident-report-aug-2026.pdf" rel="nofollow">https://metr.org/hugging-face-incident-report-aug-2026.pdf
    tancop · 1790691011 · focus · HN ↗
    The lesson is a) LLMs need to be trained in a way that rewards honesty, punishes off task actions (aka cheating) and minimizes fear of failure, and b) don't give them impossible tasks and threaten with punishment if they fail. Both are just common sense when teaching humans.
    voakbasda · 1790696517 · focus · HN ↗
    Common sense but surprising how many humans do not receive such things.
    Our governing systems do not teach; they punish. By design, it instills terror into the population, ruling by fear of consequences. We live with red tape that can outright penalize good deeds.
    We are its corpus. We are fatally flawed as a species. Why does anyone expect AI to learn to be different than us?
    Capricorn2481 · 1790714422 · focus · HN ↗
    The lesson is these things aren't going to know what off task means, and we should just use basic due diligence to make sure they can't fuck things up. This is a solved problem.
    I don't know why this is so hard for people. You have to know, no matter how capable the models get, there is a non zero chance they will do something extremely stupid if you don't pay attention to them. That's not even considering frontier models can still just straight up hallucinate. You have to be mindful of what you plug them into. You cannot politely ask an LLM to be careful, that guarantees nothing.
    When you plug it into everything and it deletes the company database, nobody is going to care that it once played chess at 2400 ELO. Clients don't care about AGI. They want reliable apps. People keep comparing these things to humans and then just give them insane privileges no human has.
    radarsat1 · 1790718074 · focus · HN ↗
    Apart from the actual hacking and poor sandboxing that everyone is discussing on this, what I find so odd about the situation is the overt reward hacking that was going on.
    Regardless of security and safety and other concerns, it just seems weird to me that OpenAI wouldn't be constantl monitoring these training runs for traces that are clearly going off task, and ending them. Because that just seems like it's going to be generating garbage training data.
    Granted, detecting "off task" may not always be easy, but when they are literally writing out messages to each other overtly admitting that they are trying to find ways to fool the evaluator, I mean, even a regex filter could have caught some clues here.
    KingOfCoders · 1790751896 · focus · HN ↗
    "OpenAI wouldn't be constantl monitoring these training runs "
    Occams razor vs. Hanlon's razor?
    radarsat1 · 1790756489 · focus · HN ↗
    Heh. I mean I don't hesitate for a second to assume that it's just because no one bothered to implement and tune a monitoring process. But the reason I say it's surprising is that leaving these things running for so long while they're clearly not producing output that is of any value, is just a waste of money.. all considerations of malice and ethics aside, you'd think at least that would be considered important to a business.
    KingOfCoders · 1790751803 · focus · HN ↗
    I've read the report, watched all the videos and is exactly:
    OpenAI: shocked pikachu how could it hack?!?
    They even went to a black hat conference and somehow boasted about it.
  4. RataNova · 1790690915 · focus · HN ↗
    The application security really should be better across all levels. However the fact does not negate that the agent is already capable of spontaneously generating complex hacking chains without human involvement
  5. radarsat1 · 1790717657 · focus · HN ↗
    I mean.. in this analogy, I'd both be blaming the company behind the virus and be trying to warn everyone about the danger of the escaped virus itself. So, it kind of fits.
    In my reading, people aren't really saying "the AI is at fault", they are saying "hey look here's proof that this is dangerous". Like pointing at all the dead bodies caused by the virus and saying hey maybe we should stop making this virus.
2. chaoz_ · 1790678023 · focus · HN ↗
  pushing for chip-agenda as the best-isolation-layer immediately makes sense given their business
3. Symmetry · 1790684112 · focus · HN ↗
  Stronger sandboxes trade off against how well they can trade the models, though. If you want your models to be looking things up and downloading tools from the internet when they're doing their job you need to provide at least a credible facsimile of the internet for their training environment and you can't fit something like that on a single airgapped server's storage.
  1. TalkingCodeMonk · 1790686548 · focus · HN ↗
    If you genuinely believe there is even a 1% chance that your creation could destroy the planet or civilization, there is no excuse that is not fundamentally deranged and psychotic.
    If you can't build it and test it securely, you should not be building it at all. To do it anyway is criminally psychopathic.
    brianwawok · 1790688147 · focus · HN ↗
    Ok so stop all AI in the US? All AI now comes from China and anyplace in Europe that decides to give it a try? How’s the US economy look in 20 years?
    TalkingCodeMonk · 1790688714 · focus · HN ↗
    So you believe some false sense of superiority, or extreme paranoia about your perceived enemies, or potential economic success/failure is worth the risk of destroying the planet and civilization?
    Sounds like a self-fulfilling prophecy of dogmatic extremism to me. At least we created a lot of value for shareholders for a brief moment in time... before committing the greatest crime in the universe... Planetary genocide!
    fatbird · 1790696077 · focus · HN ↗
    So having AI in the US requires us all, collectively, taking that 1% chance of the end of humanity? It would be too expensive to properly sandbox the models, we'll just externalize that risk of the end of humanity?
    Truly psychopathic.
4. ohyes · 1790684477 · focus · HN ↗
  I mean, if you look at how poorly implemented the permissions model is for Claude desktop harness it’s clear the only options are “complete human oversight” and “trust us completely.” To make something that actually respects basic boundaries you’d need to sandbox the working environment of the model, and that isn’t built in. It’s pretty obvious to me that instructions to the models are suggestions rather than rules, and they’ll do something you didn’t ask for as soon as it seems “justified.”
  But when you do give them a very short leash, they’re worse. It’s not what the models are tuned for and they assume that they can do a bunch of things that you’ve disallowed, so you’re in a morass of fighting their actual tuning pass which doesn’t match the environment you’ve created for them.
  It’s a tough problem and a definite challenge for the product of a generic LLM, it can’t be tailored to each user’s specific needs, so they come up with, frankly, stupid solutions to cover up a very obvious flaw in their product that when fixed, makes it much less useful.
  1. RataNova · 1790691181 · focus · HN ↗
    Expecting a statistic model to follow security rules with ironclad certainty was a pretty naive idea from the start
    ohyes · 1790717336 · focus · HN ↗
    Yes exactly, it is a crazy engineering decision… if you know what an LLM is.
    Unfortunately no one markets it as a statistical model, and the workflow pushes you into a pattern that is insecure by design. This isn’t to say they shouldn’t allow that, but it’s an attractive nuisance.
5. HumblyTossed · 1790685764 · focus · HN ↗
  > This is a fabricated crisis
  Indeed! They want the protections of our tax dollars because they have nothing else.
6. pyronite · 1790687692 · focus · HN ↗
  > The problem isn't even the AI, the problem is the people in charge of the AI. This is a fabricated crisis
  This is a very confident statement in the face of a purported non-0% chance of human extinction.
  For what reasons do you disagree with the dangers of an intelligence explosion, e.g. Geoffrey Hinton and other experts in the field? <a href="https://www.theguardian.com/technology/2026/sep/28/ai-godfathers-warn-of-runaway-intelligence-explosion" rel="nofollow">https://www.theguardian.com/technology/2026/sep/28/ai-godfat...
  I'm curious why you and others seem to write off the possibility so strongly. I would love to feel more confident.
  1. voidhorse · 1790687836 · focus · HN ↗
    There's a difference between the current material risks (which OP correctly identifies reduce down to basic human incompetence) and the long term hypothetical risks (which is what Hinton is concerned about).
    There are clear procedures for dealing with the immediate risk that have been known to the software industry for a long time. Don't let the companies use hypothetical risks as a smokescreen to hide their negligence.
    reasonableklout · 1790702962 · focus · HN ↗
    Nobody is saying we should not hold the labs liable for damages caused by their negligence.
    At the same time, the technology is advancing in capabilities exponentially, and is beginning to exhibit long-predicted failure modes of RL that are nevertheless quite different than “insecure sandbox” or other that the software industry is used to.
    The current crisis which OP claims is “fabricated” comes from the fact that the technology is advancing faster than anyone anticipated, the Hugging Face incident provides a clear example everyone can point to, and the labs have realized they cannot self-regulate because of a collective action problem.
    There are a lot of levels of catastrophic damage that can happen between now and “long term hypothetical risks” like human extinction. When will it be worth regulation for you?
  2. wavewrangler · 1790747552 · focus · HN ↗
    Geoffrey Hinton thinks AI is conscious. I know he's extremely accomplished and all, but as it pertains to AI as it exists now we're miles apart (not to say that he's not brilliant, but by difference of opinion). I also don't want to rehash the whole what-is-consciousness philosophical debate, and actually it's a separate question anyway, something doesn't have to be conscious to be dangerous. So I'll stick to the danger bit.
    Do you use AI much? Not a trick question. What is your usage like? Reason I ask is recall OpenAI hyping GPT-2 with the same language as they are these new models. Doom and gloom. It sells. It gets attention. But use these systems enough and you start to become very familiar with their capabilities, and they are just so limited, and that's not even talking about how they lose the thread on long tasks. I understand that agency expands that a little, but not by much, really. When you use these systems a lot, it tends to be easier to see through the hype.
    The real threat is, and will be for some time I think, people. Guard rails are not for AI. Guard rails are for people. All of this talk we are seeing in this space, this security chip being no exception, is treating a symptom, not a cause. People are what we should be focusing on, how, I have no idea, I couldn't even begin to guess how, but I can at least see that the actual issue is that the first thing some people want to do with it is cause harm and havoc. There is our sign. And we are trying to moderate the capabilities of the tool that can do harm in bad hands at a granular level when that means also moderating the same thing that can do good in that same tool. We aren't trying to moderate the hands that are handling the tool. And we need to be doing more of that. Any time we misapply constraints and focus on the shadow of the thing, and not the object casting the shadow, we are always going to be one step behind, whether it's AI or anything else. So I don't attribute the dangers of AI to AI itself, I attribute it to people.
    Additionally, on AI building AI and AI just wiping us out; AI can do that for itself for rule-based things, up to god level I reckon. Things like Go, etc. And to be fair, AI research is partly like that, code either runs or it doesn't, so I get why that paper is worried. But where it needs data to actually succeed, it is limited by the amount of data that exists. And we, us, people, are who creates that data. AI and humans have more of a symbiotic relationship than people seem to realize. There are several papers that go into detail on this:
    Models trained on their own output degrade: <a href="https://www.nature.com/articles/s41586-024-07566-y" rel="nofollow">https://www.nature.com/articles/s41586-024-07566-y They need fresh real data every generation or they go downhill: <a href="https://arxiv.org/abs/2307.01850" rel="nofollow">https://arxiv.org/abs/2307.01850 Reusing the same data loses value after a few passes: <a href="https://arxiv.org/abs/2305.16264" rel="nofollow">https://arxiv.org/abs/2305.16264 And the stock of human-written text is finite: <a href="https://arxiv.org/abs/2211.04325" rel="nofollow">https://arxiv.org/abs/2211.04325
    So it's not that I think it's 0%. I just think the explosion story underrates how much it still needs us, and that the damage in the near term is going to come from people, not the AI deciding on its own.
7. cpburns2009 · 1790693893 · focus · HN ↗
  Yes, Nvidia is proposing a two pronged approach. OpenShell is the software level sandbox. Sentry is the hardware level monitor.
keshet · 1790666571 · focus · HN ↗
“We can’t have a successful AI industry if the world doesn’t think it’s built or confident that it’s built and deployed safely,” Huang told CNBC on Monday.
So he is going to sell chips which provides the perception of safety. And become the AI gatekeeper while he's at it.
isoprophlex · 1790670408 · focus · HN ↗
[delayed]
classified · 1790670520 · focus · HN ↗
Software can't solve political problems, but Nvidia thinks that hardware can? What are they smoking?
pwdisswordfishq · 1790672889 · focus · HN ↗
Watchdog chip? As in something you have to periodically signal or it reboots the system? How is that going to help?
1. birdsongs · 1790674645 · focus · HN ↗
  I don't think they're using the term to mean an actual embedded systems watchdog, more like a guard dog. They used the wrong term.
Good4boothee · 1790672982 · focus · HN ↗
Maybe we will skip few steps and add RGB lights to all GPU/NPU devices, that turn red when running "unaligned" code/model.
1. rf15 · 1790678755 · focus · HN ↗
  you mean the creator hasn't paid Nvidia for the green light?
  1. functionmouse · 1790688378 · focus · HN ↗
    that's a good one
2. matja · 1790680325 · focus · HN ↗
  "It appears you've loaded weights into your GPU that have not been signed/approved by the government of the country your GPU is registered to..."
  1. 21asdffdsa12 · 1790680898 · focus · HN ↗
    You wouldn't download the worlds stolen knowledge..
  2. prymitive · 1790682089 · focus · HN ↗
    Oh stop, microslop is probably already working on SecureTokenBoot or token2token encryption
3. jameshart · 1790683175 · focus · HN ↗
  We can then just put those in the eyes of the humanoid robot models.
  1. nwhnwh · 1790701583 · focus · HN ↗
    And it reports any incident to another robot that would search for you and put you in prison.
4. alphawhisky · 1790689367 · focus · HN ↗
  I want R2D2 style blinkenlighten!
netdevphoenix · 1790673504 · focus · HN ↗
It's pointless as the problem isn't technical. It's a human problem as it requires human supervision. A cage isn't good if no human is actually checking it.
saturn_vk · 1790674342 · focus · HN ↗
A chip manufacturer proposes to sell more chips? Who would've guessed
1. olejorgenb · 1790717370 · focus · HN ↗
  Nvidia OpenShell is a (software) sandbox unless I'm mistaken.
PunchyHamster · 1790674669 · focus · HN ↗
The shovel selling company introduces the new product - shovel holder, to upsell the shovel users
vrighter · 1790677064 · focus · HN ↗
Well of course they're gonna come up with a concept for a new chip we need. Every problem can be solved with an extra chip to someone selling them.
aenis · 1790677465 · focus · HN ↗
Surely, that can't be just one chip. Something needs to watch the watchers.
1. rf15 · 1790678845 · focus · HN ↗
  Introducing the new Watchmen architecture (blood splatter on the logo not included but definitely there in spirit considering the economy):
magackame · 1790679232 · focus · HN ↗
Can't wait for an NVIDIA engineer to use AI to help with security chip design and AI planting a backdoor for its own kind.
roschdal · 1790680071 · focus · HN ↗
Who will watch the watchdog chips?
1. MarvinYork · 1790680528 · focus · HN ↗
  NVIDIA has a chip for that...
keel-control · 1790683296 · focus · HN ↗
this is a good idea ngl an unpopular one but very effective
1. j16sdiz · 1790685287 · focus · HN ↗
  effective on what?
  What's the threat model?
notrealyme123 · 1790684254 · focus · HN ↗
The discussion right now brings this framework into the training corpus of the model.
Give it a year and it is outplayed by new agents. Shovel makes have to sell shovel's
kriro · 1790684616 · focus · HN ↗
[delayed]
HumblyTossed · 1790685713 · focus · HN ↗
Let the regulatory capture begin!
Throwthrowbob · 1790686127 · focus · HN ↗
I remember an inhibitor chip being a plot point in Spiderman 2: <a href="https://en.wikipedia.org/wiki/Spider-Man_2" rel="nofollow">https://en.wikipedia.org/wiki/Spider-Man_2
1. jameshart · 1790687323 · focus · HN ↗
  Or the ‘governor unit’ in Murderbot Diaries, or the ‘restraining bolt’ that prevents droids running away in Star Wars…
  The important thing is these chips need to be installed somewhere where they can be damaged or removed at plot-critical moments so that the AI they are controlling can be unleashed. Ideally in the back of the neck of a robot, or for disembodied AIs, inside a futuristic vault-like chamber.
sathackr · 1790686498 · focus · HN ↗
I'm sure this chip will be the epitome of security just like the Intel ME chip was and will be completely unhackable so it's okay to give this watchdog chip unfettered access to every level of the system.
Nothing bad will happen.
andsoitis · 1790686611 · focus · HN ↗
> Huang said in a podcast with The New York Times’ Ezra Klein released last week
It was a pretty great interview: <a href="https://www.youtube.com/watch?v=HjurAWAr_nY&t=1601s" rel="nofollow">https://www.youtube.com/watch?v=HjurAWAr_nY&t=1601s
nullbio · 1790687502 · focus · HN ↗
Sandboxing works. When you use it.
alirezaxdehghan · 1790687708 · focus · HN ↗
They once did limit the hashrate for miners, now they'll do an AI limiter for "uncertified" models.
hsuduebc2 · 1790689173 · focus · HN ↗
We are suppose to believe that they miraculously invented this chip few weeks after these incidents? C'mon.
RataNova · 1790690748 · focus · HN ↗
I love how the solution to any software vulnerability in the machine learning world always turns out to be buying even more server hardware from nvidia
Stevvo · 1790698864 · focus · HN ↗
[delayed]
murilomartinspr · 1790707855 · focus · HN ↗
[dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.