Thread

1. edot · 1791067681 · focus · HN ↗

   Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.

   There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.

   1. Barrin92 · 1791070504 · focus · HN ↗

      >Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.

      I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.

      I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent

      1. jswelker · 1791071216 · focus · HN ↗

         Wasn't Bill Gates mostly just saying terrorists could use AI to build a bioweapon? That sounds legit to me.

         1. zozbot234 · 1791072179 · focus · HN ↗

            It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.

            1. jswelker · 1791072417 · focus · HN ↗

               So a terrorist can't use a local llm to help grow anthrax? That is not farfetched at all.

               AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.

               1. zozbot234 · 1791073165 · focus · HN ↗

                  It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.

                  1. simianwords · 1791095166 · focus · HN ↗

                     This take belongs in 2024. It has been falsified multiple times but it never seems to go away.

                     1. zozbot234 · 1791096134 · focus · HN ↗

                        Are you thinking about model capabilities in coding and math starting late 2025 or so? Those were intentionally boosted via automated RLVR, and there's nothing even loosely comparable to that in applied biology work, let alone in the speculative "helping a bad actor do something crazy" domain that the AI safety folks are worried about. You can't extrapolate from one to the other.

                        The implied concerns from sensible safety advocates are also about someone jailbreaking the latest proprietary AI frontier model for something like this (which is why their current guardrails are so extreme), not about toy local models.

                        1. simianwords · 1791097171 · focus · HN ↗

                           I actually agree with you. Verifiable domains will have better performance.

                           But there’s nothing specially bad about LLMs that don’t allow it to work outside of its training set. It’s just that biology has to verify itself in physical realm and it’s a bit slower.

                           So yeah, I also don’t think some bad actor will find the secret to manufacturing a bio weapon using LLMs. But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.

                           1. zozbot234 · 1791099972 · focus · HN ↗

                              > But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.

                              The problem is that in order for the scaremongering to make any kind of sense and for "stop frontier AI immediately" to be the right response (which is what the "AI safety" folks seem to be pushing for), you don't just need this to be possible in the abstract at some undetermined point in the future. You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance using their trusted access to unreleased super-smart AI, and thereby devise appropriate defenses with that same AI's help. That, if anything, is the most implausible part about this entire scenario.

                              1. pixl97 · 1791133852 · focus · HN ↗

                                 >You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance

                                 Sigh.

                                 Attackers only need to win once. Defense needs to work every time.

                                 A single wide scale attack affecting around 100k people or more will have your neighbors stomping on your face telling you to shut up, and to lock this shit down.

                                 It's insane how you can watch a technology get better and better and better and come up idea that everything will remain the same. We are currently in the middle of development of the most powerful weapons on earth and you don't want to think about it because it's uncomfortable.

Unofficial Hacker News client; not affiliated with Y Combinator.