Thread

1. edot · 1791067681 · focus · HN ↗

   Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.

   There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.

   1. Barrin92 · 1791070504 · focus · HN ↗

      >Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.

      I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.

      I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent

      1. jswelker · 1791071216 · focus · HN ↗

         Wasn't Bill Gates mostly just saying terrorists could use AI to build a bioweapon? That sounds legit to me.

         1. zozbot234 · 1791072179 · focus · HN ↗

            It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.

            1. jswelker · 1791072417 · focus · HN ↗

               So a terrorist can't use a local llm to help grow anthrax? That is not farfetched at all.

               AI democratized the knowledge needed to build bioweapons. Like how with a 3d printer anyone can build a gun with no expertise.

               1. zozbot234 · 1791073165 · focus · HN ↗

                  It turns out that LLMs, especially local LLMs, tend to hallucinate a lot when thinking about anything that's overtly fiddly or technical. This is even more the case when they're in a domain that isn't a natural part of their training data. If you have to "jailbreak" the model to get it to talk, you're so wildly out of the expected distribution that you'd be crazy to trust anything it says. It's basically making up stuff as it goes along. These are foundational issues with how the models are created, not something that a bad actor can just hack around.

                  1. simianwords · 1791095166 · focus · HN ↗

                     This take belongs in 2024. It has been falsified multiple times but it never seems to go away.

                     1. zozbot234 · 1791096134 · focus · HN ↗

                        Are you thinking about model capabilities in coding and math starting late 2025 or so? Those were intentionally boosted via automated RLVR, and there's nothing even loosely comparable to that in applied biology work, let alone in the speculative "helping a bad actor do something crazy" domain that the AI safety folks are worried about. You can't extrapolate from one to the other.

                        The implied concerns from sensible safety advocates are also about someone jailbreaking the latest proprietary AI frontier model for something like this (which is why their current guardrails are so extreme), not about toy local models.

                        1. simianwords · 1791097171 · focus · HN ↗

                           I actually agree with you. Verifiable domains will have better performance.

                           But there’s nothing specially bad about LLMs that don’t allow it to work outside of its training set. It’s just that biology has to verify itself in physical realm and it’s a bit slower.

                           So yeah, I also don’t think some bad actor will find the secret to manufacturing a bio weapon using LLMs. But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.

                           1. zozbot234 · 1791099972 · focus · HN ↗

                              > But maybe these people think it’s possible. I’m skeptical but I’m going to also listen to the people who know it best.

                              The problem is that in order for the scaremongering to make any kind of sense and for "stop frontier AI immediately" to be the right response (which is what the "AI safety" folks seem to be pushing for), you don't just need this to be possible in the abstract at some undetermined point in the future. You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance using their trusted access to unreleased super-smart AI, and thereby devise appropriate defenses with that same AI's help. That, if anything, is the most implausible part about this entire scenario.

                              1. pixl97 · 1791133852 · focus · HN ↗

                                 >You also need to argue that it will not be helpful for white-hat biosafety researchers (there will hopefully be several orders of magnitude more white-hat biosafety folks than attackers, with orders of magnitude more resources available) to red-team that exact scenario several months or even years in advance

                                 Sigh.

                                 Attackers only need to win once. Defense needs to work every time.

                                 A single wide scale attack affecting around 100k people or more will have your neighbors stomping on your face telling you to shut up, and to lock this shit down.

                                 It's insane how you can watch a technology get better and better and better and come up idea that everything will remain the same. We are currently in the middle of development of the most powerful weapons on earth and you don't want to think about it because it's uncomfortable.

               2. com2kid · 1791086352 · focus · HN ↗

                  [delayed]

                  1. altmanaltman · 1791090789 · focus · HN ↗

                     Also it kind of infantalizes terrorists as if not having an LLM access or 3d printer is what stopping their attacks. Like okay I ask LLM to help me create a dirty nuclear bomb but the moment i start taking steps to do that, law enforcement will already be tracking me.

                     Most people saying LLMs can make terrorism easy have never given doing terroism a serious thought imo.

                     1. DonHopkins · 1791104805 · focus · HN ↗

                        The infantile terrorist in the White House already has missiles, and he has used them to murder 120 Iranian schoolchildren. What AI offers Trump is a convenient patsy. Automating the targeting doesn't automate away responsibility, and Trump doesn't get to pretend it does.

                        Wearing AI like a glove can improve your grip and keep your fingerprints off the gun, but it doesn't make the glove the murderer.

                     2. hackinthebochs · 1791108770 · focus · HN ↗

                        The raw materials to make bioweapons are easily purchased on line with no pre-emptive tracing. It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.

                        1. simoncion · 1791115169 · focus · HN ↗

                           > It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.

                           Mmm. Especially in this arena, LLM assistance is like The Anarchist's Cookbook. A quarter of the time following the instructions will seriously injure you... and if you know enough to identify which instructions are the hazardous ones, you know enough to not need the assistance.

                           1. hackinthebochs · 1791121777 · focus · HN ↗

                              That may be true (and I hope it is). But it's not a limitation that will remain indefinitely.

                              1. simoncion · 1791124119 · focus · HN ↗

                                 > But it's not a limitation that will remain indefinitely.

                                 The Sun is going to fail in many hundreds of millions or a few billion years. This will either turn the surface of the earth into slag, freeze it, or both. Either way, all life on the planet is doomed. This fact is not a reason to fail to switch from hydrocarbon-burning electricity generators to photovoltaic, fission, wind, hydroelectric, and geothermal electricity generators. Extinction events that will happen in the extremely distant future shouldn't prevent us from doing the things that are smart to do in the medium- and long-term.

                                 But, -to bring things to the present day- companies that are solidly on track to hit their promised growth targets don't come out and publicly say "We're working on WMDs. [0] We cannot safely work on these WMDs. We refuse to stop working on these WMDs. However, if you lawmakers make special laws and regulations just for us and include us in the process, we'll be quite happy to submit the stop work order to our employees!". That's a statement you only make if there's no way in hell you're going to keep your promises and you're willing to risk jail time and annihilation of your companies for a shot at being able to con Congress into giving you an ironclad excuse to fail to keep your promises.

                                 Given enough time and focused effort, we will end up with widely-available automated librarians that are very good. We're not there yet, and -based on current events- are absolutely not going to get there in the near future.

                                 [0] Anything with a 10% chance of destroying all humanity is a WMD.

                                 1. aesthesia · 1791140703 · focus · HN ↗

                                    > The Sun is going to fail in somewhere between many hundreds of millions and a few billion years.

                                    Cool, what does a process that's about 9 orders of magnitude slower than AI development have to tell us about AI risks?

                        2. altmanaltman · 1791116904 · focus · HN ↗

                           Sure, that's why we keep seeing so many bioweapon attacks in recent times, because AI + Amazon can end the world.

                           1. hackinthebochs · 1791121740 · focus · HN ↗

                              It's strange how quick people are willing to dismiss concerns with horrible reasoning as long as it suits their biases. Perhaps a little intellectual honesty is called for considering the stakes? There are many plausible reasons why we haven't seen AI fuel bio-terror attacks yet. None of which implies it's unlikely or implausible in principle.

                              1. tripledry · 1791123963 · focus · HN ↗

                                 But hasn't bio attacks already been possible before AI, what has changed? Maybe the barrier to entry lowered, but one could argue the availability of info has not been the bottleneck for as long as the internet has existed.

                                 The threat is ofc real, but AI won't magically "do the thing" still, it's not code that's the bottleneck AFAIK? Correct me where I'm wrong.

                                 1. hackinthebochs · 1791130084 · focus · HN ↗

                                    Synthesizing complex information from a range of technical sources enough to build bioweapons is cognitively impenetrable for the vast majority of people. The ability of AI to synthesize this information into a step-by-step recipe to follow is a step change in accessibility.

                                    There's an opportunity cost to terrorism just like with any time sink. The effort to build up knowledge enough to produce some weaponized pathogen will be compared to just doing traditional terrorism. For some low capability terror cell, its easy to see how the cost/benefit analysis has been in favor of traditional terrorism up to now. The kinds of terror acts that take years of sustained effort to execute are rare. But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.

                                    1. tripledry · 1791186212 · focus · HN ↗

                                       > Synthesizing complex information from a range of technical sources enough to build bioweapons is cognitively impenetrable for the vast majority of people. The ability of AI to synthesize this information into a step-by-step recipe to follow is a step change in accessibility.

                                       I agree with this part.

                                       > But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.

                                       But I argue that the barrier to entry is not about information, it's about needing to build, execute and maintain a lab and production facility? It's not the theoretical part that's the bottleneck.

                                       Or maybe I'm overestimating the difficulty of implementing such a thing, this is not in my area of expertise.

                                       1. hackinthebochs · 1791189146 · focus · HN ↗

                                          >Or maybe I'm overestimating the difficulty of implementing such a thing, this is not in my area of expertise.

                                          Not mine either. But there is a vocal contingent of experts in biotech that have been sounding the alarm for the last couple of decades at the potential for rogue groups to develop weaponized pathogens for terrorism. And this was years before LLMs. Apparently constructing a sufficiently capable lab isn't really a bottleneck due to how freely available equipment is by mail order and how unsophisticated the processes are in practice.

Unofficial Hacker News client; not affiliated with Y Combinator.