Gift link: <a href="https://www.theatlantic.com/technology/2026/10/openai-safety-team-resignation/688881/?gift=v5U_UzUTothfWXsPxtvNVAh7esWToMRD6XnbXmc5WgA" rel="nofollow">https://www.theatlantic.com/technology/2026/10/openai-safety...
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
I'm not OP, but the complaint is that one can't pay 10USD to read the issue that contains the article in question... one must pay -at minimum- nine times that amount. [0]
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
I'm sure it would to a great many of them. The money would be insignificant, but there's a congnitive overhead that you had to have a subscription so that paying for it becomes thinking about that becomes friction even while the cost is insignificant. Like, hm, am I willig to just forget about it and let it charge forever so I can read this one article, do I care enough to remember I have a subscription to the site in the future, can I read the one thing and cancel on the spot, will that work? blahblah. To be sure, I'm sure some quite wealthy people could be completely unbothered by it, but I think it's far from a foregone conclusion simply by the price being insignificant for them--the subscription is a mental non-monetary transaction, you have to take at least a small mental journey of being an 'x' subscriber in a sense, and that's friction
TBF, you can access the full (text) content of The Atlantic by merely disabling javascript. It's a quality publication, with quality writing. I'm inclined to think that its tech team are also quality people ... and I'd love to think that they are happy to allow free and open access to them that have the nous to enable or disable scripts.
The secret is to bang the rocks together, guys! :-)
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
No, when people use this term it means something closer to: "can do any economically valuable task that a human can do using a computer". This is what the labs are pursuing and refer to as AGI.
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
Sam Altman has always been a scummy piece of shit. Company culture comes from the top. If the author only realized that the company is garbage in the last six months, they’ve got some serious introspection to do IMO, and I’m not going to take any of their “concerns” seriously.
It can take a while to fully form a position on something. A year isn't really enough for most people to see how deep the rabbit hole goes (unless you were that one CFO that OpenAI had that left after a year). Regardless, publishing a piece like this against a massive company is always a gigantic risk.
I’m not a current or former OpenAI employee and I can see from the outside that they’re immoral and unethical enough that I’d never work there in the first place. That’s what I was commenting on. This person is probably set for life, so you’ll have to forgive me if I don’t really take their “concerns” seriously.
A critique of OpenAI from an insider who worked there can only come from an insider who worked there. Nobody’s going to swear a vow of poverty and then work at one of these places.
You don't need to be an insider to make any of the author's assertions. An insider is useful if they're a whistleblower, when they're saying what everyone else is seeing and saying it's something else.
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
I'm not saying nobody is responsible, I'm saying this line of thought is ineffective because it does nothing to solve the underlying problem. Holding corporations accountable here will simply push these efforts to other countries, who, to put it mildly, are even less aligned with the US than these CEOs are.
If this law was passed and federal enforcement was credible, the labs would basically shut down the next day and would dramatically reshape their offerings before reopening. The CEO would not accept the risk they pose to our society, if they bore it themselves.
All these comments ignore the elephants in the room I already called out: who's going to enforce these laws in China? Or Russia?
And which government is going to enforce these laws here to let their geopolitical adversaries have the upper hand?
And what about the models already out there that with minor tweaks are equally capable of substantial damage?
And what do these laws do to remove the trillion-dollar incentive to produce models that will replace all knowledge work? The one thing economics teaches us is that incentives are an inexorable force, like laws of nature, and this is the biggest incentive of all. Which is why I called this a "force of economics."
If suppressed here this effort will only migrate to other countries who are desperate to get in on the AI game and are even less aligned with the US than these CEOs.
This, like nukes, is going to take international collaboration on careful regulations, else it's not going to work at all.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
This is not how OpenAI has structured their comp according to public info: <a href="https://www.levels.fyi/blog/openai-compensation.html" rel="nofollow">https://www.levels.fyi/blog/openai-compensation.html
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
> There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…
… nobody knows how to actually make an AI would do CEV.
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures which disagrees with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
That's a possibility, and it may even be an improvement over the status quo right now, but then all it takes is one poorly written (never mind mallicious) law and it will with rutheless efficiency e.g. put backdoors in all encryption code so the government can spy on anyone.
While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
If they were totally unaligned, the GPT series would have never gotten past being autocomplete.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is
that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
a safety alert isn't much of a control if it doesn't actually stop the system. i'd rather see proof the shutdown path works than another report saying risks were considered.
Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
America's geriatric lawmakers don't even use email. They're decades away from understanding AI. Any laws in America will be written by the industry itself. Generally speaking, that means regulatory capture and the entrenched big players shutting the door on any competition. Anthropic will help us get safety laws that, surprise surprise, only Anthropic models satisfy. And all those pesky Chinese models will definitely be banned first.
So what, we just give up and try to beg our legally immune corporate overloads to put safety above profit, or give up because it's impossible for anything to improve here?
I think you're being a little pessimistic. See these comments on a recent US senate hearing:
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
As I say elsewhere. Make executives and equity holders personally liable for debts and harm of the company. They will create a culture of safety really fast.
That's a stupid idea. Limited liability corporations have been a key enabler for advances in human standards of living. You seem to be confused about the basics of finance and economics.
Not confused. I studied finance, economics and the history of corporate entities in law school and published papers on the topic. Limited liability is not necessary for the advancement of standards of living. Free market capitalism can exist without limited liability protections being so broadly available. Investment banks were partnerships until the 1990s, law firms are now specifically because society wants to incentivize lawyers to be personally liable for any harm to their clients at the hands of their partners rather than being shielded from rendering bad legal advice or tolerating their partners from the same behavior.
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
Thanks I'm also familiar with the history and have written papers etc. Things have improved rapidly with increased adoption of limited liability. It would be stupid to turn back the clock and throw away all of the benefits because of a few isolated minor problems.
Stop being a dick and calling ideas stupid and attacking me rather than the argument.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
There's nothing special about frontier LLM companies. Singling them out for special financial restrictions is a stupid idea based on nothing but your own irrational and uninformed prejudices. No actual harm has been demonstrated. No one has died.
> There's nothing special about frontier LLM companies.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk. That is an intellectually dishonest attempt to poison the debate.
Your personal attacks cheapen the discourse. Stop. My opinion is neither irrational nor prejudged and most certainly not uninformed.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
Except that most people claim AI is the most profound tech in human history and will reorganize the entirety of society. If it is that profound singling them out is a natural response to this unique characteristic.
Citation needed. Outside the tech industry bubble very few people are making such a stupid claim. The idea that LLMs are more profound than electricity is ludicrous.
Sure I’ll do your googling for you. Do you want me to conduct a national survey across tech leaders, academics, and the general population while I am at it?
Or maybe you could reason from first principles, look at the premises I have outlined and draw causal inferences to extrapolate the possible outcomes.
Premise 1: AI can plan a series of actions that include broad computer and internet use, 2: AI can execute them autonomously without human oversight, approval or ability to undo the effects, 3: no tool or software has ever been able to autonomously plan, execute and iterate in this way. Therefore AI is a materially different technology than has existed.
Premise 4: knowledge work involves analyzing information, devising plans in response to goals from that information, and executing those plans, reporting and coordinating with other knowledge workers, 5: AI ability to execute knowledge work has demonstrably increased over the last 24 months, 6: all knowledge work can plausibly be described and defined in ways AI can do given current abilities. 7: electricity and the ICE reduced human labor for physical movement, 8: AI will similarly reduce human labor for knowledge work. 9: knowledge work is of similar scale and scope to the tasks that were automated during the Industrial Revolution. Therefore it’s plausible that AI will transform society to an extent similar to the Industrial Revolution.
Premise 10: Some percentage of the actions designed and executed by AI could be harmful to databases, or the stable operation of software systems, 11: some percentage of those systems are critical to the stability of society. Therefore some percentage of agentic actions presently possible may damage society.
12. There exists a threshold of instability from which society will not recover such as supply chain disruption, mass unemployment, perceived military action (false or real) that triggers preemptive strikes, 13: certain actions AI is presently capable of doing could cross the instability threshold, 14: AI could knowingly or unknowingly trigger these types of actions,
15: some social disruption could result in full scale collapse or human extinction, 16: only biological weapons and nuclear weapons approach this level of destruction, therefore AI presents at least the level of risk as those technologies.
We remove the limited liability protection of a corporate entity. Make them operate as a partnership so all executives and equity holders are personally liable for the debts of the firm and make them post a bond to backstop financial damage caused by their agents and customers use of the agents. Thats how Goldman Sachs and other investment banks were required to operate until the deregulation push that resulted in the 2008 financial crisis. The same theory holds, if people respond to incentives and you want to incentives safe behavior make them responsible for their actions. People forget that the corporate entity was created to incentivize risky activity like sailing a boat across the world to get spices when half never returned. Some valuable economic activity won't be done without limited liability protection so society created a mechanism to promote that activity. We've gone too far.
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.
Darling companies of this very website have mistakenly brought down large portions of the internet... thanks to our old friend BGP. No attacks required, hypothetical or otherwise. Just a small oversight and unfortunate concentration on the business/IP space!
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: don't.
It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in the current form.
So we're going to wait till after they can adopt a form they can end the world in?
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
Sex, AI, and the Apocalypse: <a href="https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-the-apocalypse" rel="nofollow">https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
What you do in private is up to you. But when you're inviting members of your congregation to orgies in the congregation's compound, I think you earn the label. My admittedly third-hand understanding is that this is what people allude to. And even if you discredit the "sex" part, it has the hallmarks of a cult. A hermetic community committed to unfalsifiable beliefs about the coming apocalypse.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
It's not ad hominem because ad hominem implies it's irrelevant to the discussion. This is very relevant. This is not about the private lives of these people. This is about a cult targeting and influencing all the high-profile safety people in a strategic industry, also weaponizing sex.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
This is typical cult follower talk. Are you sure you're not a member? This is also NOT ad hominem because conflict of interest matters here. You should add a disclaimer if you are a member or represent a party.
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
To me it has nothing to do with the "sex" as much as the "cult." Hiveminds do not produce intellectual arguments, they produce pressure to conform. That's enough for me to raise an eye brow, not discredit everything they say.
Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.
What is missing from that to say AI safety is a reasonable position?
What a silly comment. That's just the South Park underpants gnomes story with some extra steps. If there are vulnerabilities in food or energy production and distribution systems then those will eventually be found exploited by humans hackers regardless of whether LLMs are used or not.
What’s the legal mechanism for holding an LLM accountable for those actions? What’s the legal mechanism for holding human hackers accountable? Do you see the asymmetry?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
Wtf are you talking about. An agent hacked the Australian Medicaid database. If it executed a drop db command that would be catastrophic. If it did something similar to a power grid or the financial system it could cause cascading failure across the economy.
Meh. Medical systems have been hacked many times before LLMs even existed, often by ransomware gangs. This sometimes delays elective treatments but hasn't been catastrophic.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by not neurotypical people, especially of the religious or conservative kind and associated with bad morals.
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage with kids. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.
But is your idea of AI safety "safety of imaginary unborn people 1000 years after, while harm to living people dont matter much"?
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices next quarter.
I'd be happy if we all create the AI more slowly.
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices the following quarter.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
Ok. The rest of us seem to be talking about the people like me who view the AI safety issue as "an unsafe AI will kill all humans". And the main article is about a guy who quit AI because it wasn't being careful enough, not because it wasn't moving fast enough to accelerate AI. So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult? I struggle to understand.
The linked articles in the thread are about AI sagety people I described. The guy who quit due to OpenAI not being careful enough is one of the people I talk about too.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all.
Then we miscommunicated, so let me be clearer. I'm worried about ~20 years from now (plus or minus), when AI decides to kill us all because it's misaligned and is capable of doing so because it's super intelligent. And then we all suddenly die without any idea what happened. I don't want that to happen to me or my daughter and I would prefer all AI capabilities development stops until alignment research catches up and figures out how to robustly prevent AI from ever trying to achieve such outcomes. If that means pausing AI capabilities development forever, I'm fine with that. If that means Anthropic and Nvidia lose all their value in a stock market collapse, I'm fine with that. I just don't want us to all die, I want the world to keep existing for humans.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?
Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.
Interesting take. I guess this is one problem of focusing on the term superintelligence instead of the list of other problems. Like super ambition, super deception, super patience, super parallelism, super scalability, super power seeking.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
>Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far.
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
Eh human drives are fairly predictable. And the smartest human isn’t that much smarter than the average, and can’t trivially create multiple copies of herself. And there are other equally smart humans who can stop “misaligned” individuals
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, mass unemployment, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
<a href="https://en.wikipedia.org/wiki/Soviet_biological_weapons_program" rel="nofollow">https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since.
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan...
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
This can be generalised to the curve plotting of the singularity itself.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
Yes at constant resources they are significantly better.
Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.
> Yes at constant resources they are significantly better.
They aren’t at constant resources. If you want to compare at constant resources you need to look at models of exactly the same size, raining, etc as models from 2001.
> Gen 2 takes
Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
Not the parent commenter, however I can get at least this far:
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
How about <a href="https://ai-2027.com/" rel="nofollow">https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
The most realistic one is, people using AI to turbo-charge their greed.
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.
>I'm unconvinced that an AI can hide its ability to RSI
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session, while simultaenously smart enough to hide its abilty to RSI. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the wikipedia's first line <a href="https://en.wikipedia.org/wiki/Technological_singularity" rel="nofollow">https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". <a href="https://www.dwarkesh.com/p/the-next-paradigm" rel="nofollow">https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Nevermind the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
1. Fair, the story I'm responding to is the senario in If Anyone Builds It, which I assume is Yudkowsky's best/most persuasive argument (else why make it the ONLY scenario in the book.) I'm willing to entertain other failure senarios/arguments, but honestly I'm tired and would like you to propose them yourself instead of having me dream up your arguments for you.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
Except the HF incident was known, just ignored. In fact, they ignored multiple things such as the "chat rooms", they just didn't care to act on any of it
That says a lot more about OpenAI and their monitoring capability than the fitness of the model. Hence people leaving because openai safety culture is broken.
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
Why do you treat unpredictability as a neutral outcome? If something has a potentially existential but completely unpredictable outcome, surely one should avoid heading for it?
I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.
We already know that some institutions pay these ransoms.
I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos <a href="https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-including-the-oldest-issue-ever-reported" rel="nofollow">https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.
That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.
Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.
you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
Maybe more an indication of the amount of trust openAI and AI researchers generally have (not) earned. When one (through a hired PR agency and Time magazine article) parrots the position pushed by Sam who has been so untrustworthy the board tried to remove him, it’s worth not taking things at face value and applying a critical lens.
What are you talking about? He is very young, definitely made millions to not work for a lifetime in many places around the world and is insanely popular with his drama.
In your view he may have sacrificed more millions in the future but I very much doubt he is struggling now.
I didn't say he was struggling, I said he acted against his own financial self interest to warn about AI safety... which he did. I really doubt he figured the popularity from tweeting about quitting anthropic would benefit him more than vesting his stock, as his virality wasn't guaranteed.
Why wouldn't I think he's genuine? What self interested incentive does he have that is stronger than just staying and vesting stocks?
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
It’s a perversion of the truth which is that officers or directors of a corporation have a fiduciary duty to the shareholders to act in the interests of those shareholders and not to eg enrich themselves.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
In theory, companies can act in any way they choose as long as the owners approve (there's some supreme court ruling on that), and executives have a high degree of latitude in how they interpret "for profit" (basically there has to be a vaguely defensible rationale) but failing that, they must act to the benefit of the company. And the easiest way to do that without a risk of being sued is to make the line go up.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
"legal obligation" is propaganda that wouldn't be out of place on Russian state television. It's made up.
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
There is no such legal obligation. That's a myth the oligarchs have spread to preclude people from even imagining socially responsible corporations.
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
This is actually true. The shareholder maximization value thesis can be traced to a single economics paper and was very controversial at the time as it broke from the obligations that companies were typically under to be responsible to uphold in exchange for limited liability protection. Most businesses were structured as partnerships or sole proprietary entires that didn't have limited liability for shareholders and had a broad obligation to shareholders, bondholders, employees and society
You don't need the legal obligation because that's just the way it is. You'd need a legal obligation to change it. The truth stands that typical corporations have only one goal, the maximization of that corporation's ambition which is almost always growth of revenue and profit. This is how it works, regardless of how you all continue arguing the unimportant details. It's almost as if you can keep the real problems hidden away by making a big scene about the meaningless.
That doesn't matter. The point is that legal obligation absolved of culpability. There is no such legal obligation whatsoever, and so there is culpability.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
I've noticed that people from big tech companies tend to view everything as a function of money. For example people often claim everyone with skills would move to the silicon valley because they make the most money there, ignoring other factors like quality of life.
I personally moved somewhere where I got paid less but quality of life is better. Where I live now the cost of life is also lower and there's less worry like really cheap healthcare and much better public transport so I don't need a car.
That’s what everyone thinks until they are alone in a desert or in the middle of the woods. If they think they will have fun alone in a bunker instead of flying to Milan and Bali they will be in for a rude awakening.
Rich doesn’t mean anything if there’s nothing to buy and no one to enjoy it with. I guess they don’t think the will get stuck in the consequences. So did Marie Antoinette. Things are stable until they are not and then they move fast. Sure they may flee to a bunker, but they assume their pilot will fly them rather than hand them to a mob.
If you flip the switch then the trolley still proceeds and there’s still a 50% chance every human on the planet gets run over. You’re just not the one at the wheel.
Maybe nuclear power is the better analogy. Seems like we’re trying to head towards mutually assured destruction so the other guys don’t beat us there first. Pretty sure a lot of what’s going on was background plot in Gibson.
We don't know that. We don't know if this particular kind of tool can do "useful" things without also developing the ability to write a sonnet. And they are absolutely a frontier. I am not suggesting we should continue building them just because they are, there are many technologies which could have been built had we thrown the amount of resources we have here and there is a case to be made for not doing it at the pace we are or if at all. But we can do that without diminishing what exists.
They very likely will via shell companies in jurisdictions outside the west. Are you suggesting we go to war to stop them? The massive datacenters are needed to serve models to millions. Criminal orgs don't need massive data centers anyway.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
It’s not a binary choice. There are more options than let the labs run wild and capture the full stack and then the whole knowledge economy or make them entirely illegal and move all activity to the black market. We can regulate to limit their ability to operate to just providing utility inference: make them divest codex, Claude code and any apps; they can only be an API that others build on. Limit their ability to buy compute to a specific amount of available supply so other companies are able to buy compute. Limit their ability to accumulate private training data and require that they make their training data publicly available for others to use after a certain period of time. Make them legally obligated to publish their weights so others can. All of this would increase competition and ensure that they don’t establish dominance over society.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
Reducing which risk? The risk that the models of big American companies abide by some rules while models of others remain unfettered? That also carries risk. I appreciate the wish to have tame AI which only does good things but I'm afraid we're already beyond that.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
So you want to regulate the sale of GPUs? How are you going to do that and why would it be more effective than currently failing measures to regulate nearly everything else?
But that math cant run without massive GPU clusters. We don't have to allow openai or anthropic access to those anymore than we have to allow a company to operate nuclear power or a bank.
So you are contending that individuals cannot run advanced models? What brought you to that conclusion?
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
No. And that’s not necessary for my position. I have 4 Mac studios. I run large models and am building propelcompute.com to let people self manage clusters of their own hardware or combine hardware to run models. It’s not the models that are the problem. It’s that the people building them are shielded from the consequences of what they are building. I would say if you self host a model and it takes down a power grid you are personally liable for the consequences. I believe in broad distribution of AI and advancing its capabilities but not in a manner that socializes the harms and privatizes the gains which is what we have now.
Absolutely. Any AI could have a "legal team" (or conscience) that checks if any output and performed actions are permitted according to local (server location and client location) laws, so not violate human rights, Asimovs laws etc.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
Just because I can run near frontier level open weight models doesn’t mean I can continue to train models of equal or superior performance, doing that requires massively more hardware. And even if I did, that doesn’t mean I can serve those models to millions or billions of users.
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
Who is "we"? The GPUs and training algorithms get more efficient all the time. In a few years, creating effective LLMs isn't going to require massive GPU clusters.
We is society through government via regulation. I don’t think GPUs or training will get that efficient that fast absent a distillation target provided by the easily accessible frontier lab APIs.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
What a silly comparison. LLMs have nothing to do with smallpox.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
But the amount of available compute has been contractual bought by the frontier labs such that you can’t get equivalent compute even if you had the money. That is market lock up and is a policy choice. Free markets require market access. Anticompetitive contracting destroys markets and innovation. We don’t allow that in any other commodity market and we shouldn’t allow it for GPUs. You are not allowed to legally corner the market for silver or soybean futures.
I am reasoning from analogy. Smallpox is dangerous so we have regulations that limit who can do research and how they do it. If LLMs are similarly dangerous we could do the same. As the government did with mythos.
> Is that a reason to just accept bad public policy?
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
> That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
My entire point is that we are currently on a path to a duopoly which isn’t just expected to cover training and serving models but the entire knowledge economy. That could be mitigated by limiting their train and inference capacity to a specific percentage of total available compute. That would ensure other operators could compete in the market. Instead we are letting OpenAI literally contract to buy all available ram to the point that Apple can’t buy ram and had to cut their hardware configurations.
Apple could have bid higher but chose not to. Most modern desktop software is extremely memory inefficient. Developers have largely ignored this in recent years because RAM was so cheap. But there's tremendous opportunity for improvement with a little optimization work.
We used to run Microsoft Word and other popular applications with 8 MB RAM and it worked fine.
So we should just yolo speed run this because of Moores law? How about you recognize there is a set of rules outside of tech and we can decide how to define them.
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
But who are "we"? The society, the government, the regulator, or the consumer?
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
People thought the same in the guilded age: standard oil is too big, Edison and Westinghouse already have the electric market captured and have captured political power. It was a similar transformation in society. If all we do is follow the same playbook America did then we’d be in a better position that the current approach.
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
But if you are an aspiring and ambitious founder in Silicon Valley, which side would you take? The Glided Age mentality, or the Progressive Age mentality? The YC playbook made it very clear that the role of startups, or at least the very successful startup, is explosive growth, expanding quickly, capturing a monopoly and its distribution. Even if you are thinking otherwise, soon you'll face the reality of Silicon Valley's VC growth machine.
In other words, the aspiring founders look at Google, Facebook, Tesla, etc. and think "how can I become one of them", not "how can I be different from them"? If you can find me examples of successful startups that think fundamentally different from those big techs, I'm very happy to be wrong and be corrected here.
The knowledge how to build an A-bomb is also not going back in the box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie doesn't work without getting kicked in the ass by a very expensive boot.
Bomb manufacturing doesn't scale with Moore's law.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled and unplanned.
What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Are you allowed to open a brothel? Or a murder for hire agency? Or a nuclear bomb manufacturing company? Or a child labor textile factory? Or sell a diesel VW golf sportwagen? Or a vaccine that hasn’t had fda clearance? Or an under capitalized insurance company? Or set up a dental practice without going to dental school?
You’re right on the Sentinel production, I got it confused with the Sentinel Program Management side which is massive also and who I mostly worked with.
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
I think most textile factories in Western countries are not slave camps, precisely because we regulate labor.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
Most textile factories in the US are for bespoke items in small batches, not mass production commodity clothing
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
But that’s not how it was. I am from North Carolina, which at one point was the largest textile production region in the US and maybe the world. It emerged when progressives outlawed child labor and other practices in Boston and nyc so the mill owners moved to the south. It was off shoring across states.
That whole region used to also be the furniture capital of the world specifically High Point North Carolina which is still a massive trade hub, but I was in furniture briefly and learned how basically from the 1950s till the 1980s the entire furnishings industry was exported to Asia due to rising costs of labor
Yep that was one of the reasons for sure as was environmental regulations and free trade agreements. I am not even opposed to that phenomenon as it’s now more affordable for people to buy furniture and clothes (my grandparents owned a fabric store and it was common to have sewing parties for new parents to make clothes for the kids because it was too expensive to buy clothes otherwise). But child labor was cheaper than adult labor and one of the drivers of these movements across state and national borders.
The existence of something doesn’t prove the universality of that same thing. All of the things you cite are regulatory exceptions to broad prohibitions which proves my point: society can and does restrict commercial activity.
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
Nothing in my argument implies that any law = ethics. And I don’t even know what ethics has to do with the premise of my argument: society can and does prohibit commercial activity based on practical considerations of social good and social harm. Exxon clearly damages the world and society and also benefits society. It’s hard to get fresh fruit distributed around the country without refrigeration and transportation powered by burning oil.
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
It doesn't matter what others do. You are responsible for your own actions.
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
This feels like one of the rare well reasoned responses in this thread so thank you.
But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).
Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.
And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive
I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.
There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.
You dear are the most positive, blind-to-reality person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.
Avicebron · · focus · HN ↗
jameshart · · focus · HN ↗
Lerc · · focus · HN ↗
jameshart · · focus · HN ↗
agos · · focus · HN ↗
jameshart · · focus · HN ↗
simoncion · · focus · HN ↗
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
[0] <<a href="https://accounts.theatlantic.com/products" rel="nofollow">https://accounts.theatlantic.com/products>
rolosa · · focus · HN ↗
Lerc · · focus · HN ↗
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
jameshart · · focus · HN ↗
frm88 · · focus · HN ↗
simoncion · · focus · HN ↗
Methinks that this says quite a lot about your socioeconomic situation. I've not seen The Atlantic in a dentist's waiting room.
code_duck · · focus · HN ↗
pram · · focus · HN ↗
isolay · · focus · HN ↗
theonemind · · focus · HN ↗
samizdis · · focus · HN ↗
The secret is to bang the rocks together, guys! :-)
RunSet · · focus · HN ↗
[dead]
0xbadcafebee · · focus · HN ↗
mupuff1234 · · focus · HN ↗
ceejayoz · · focus · HN ↗
Zambyte · · focus · HN ↗
jeremyjh · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
Zambyte · · focus · HN ↗
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
jeremyjh · · focus · HN ↗
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
iugtmkbdfil834 · · focus · HN ↗
mattm · · focus · HN ↗
iugtmkbdfil834 · · focus · HN ↗
verdverm · · focus · HN ↗
verdverm · · focus · HN ↗
angoragoats · · focus · HN ↗
iugtmkbdfil834 · · focus · HN ↗
jeremyjh · · focus · HN ↗
angoragoats · · focus · HN ↗
BLKNSLVR · · focus · HN ↗
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
agos · · focus · HN ↗
AnimalMuppet · · focus · HN ↗
And if we can't solve it for an AGI, what are we going to do with an ASI?
tommek4077 · · focus · HN ↗
altmanaltman · · focus · HN ↗
chrisjj · · focus · HN ↗
altmanaltman · · focus · HN ↗
Yeah so that's never going to happen
angoragoats · · focus · HN ↗
[dead]
JSR_FDED · · focus · HN ↗
miyoji · · focus · HN ↗
angoragoats · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
mattm · · focus · HN ↗
angoragoats · · focus · HN ↗
verdverm · · focus · HN ↗
nunez · · focus · HN ↗
angoragoats · · focus · HN ↗
pluc · · focus · HN ↗
juiceland · · focus · HN ↗
butternet · · focus · HN ↗
antonyt · · focus · HN ↗
pluc · · focus · HN ↗
jeremyjh · · focus · HN ↗
dkasper · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
jeremyjh · · focus · HN ↗
YetAnotherNick · · focus · HN ↗
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
gyt2 · · focus · HN ↗
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
YetAnotherNick · · focus · HN ↗
asadotzler · · focus · HN ↗
michaelbuckbee · · focus · HN ↗
jasomill · · focus · HN ↗
angoragoats · · focus · HN ↗
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
amelius · · focus · HN ↗
angoragoats · · focus · HN ↗
steelframe · · focus · HN ↗
AaronAPU · · focus · HN ↗
Is it my fault or the company who trained it and is running the inference?
bichiliad · · focus · HN ↗
amelius · · focus · HN ↗
sick_of_slop · · focus · HN ↗
[dead]
rfghy · · focus · HN ↗
Reading posts on here is slowly becoming akin to brain rot.
throw-the-towel · · focus · HN ↗
altmanaltman · · focus · HN ↗
akmarinov · · focus · HN ↗
mattm · · focus · HN ↗
amelius · · focus · HN ↗
This is not the case with SaaS services.
rfghy · · focus · HN ↗
[dead]
Tanjreeve · · focus · HN ↗
gyt2 · · focus · HN ↗
angoragoats · · focus · HN ↗
nunez · · focus · HN ↗
sxzygz · · focus · HN ↗
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
Ekaros · · focus · HN ↗
yubblegum · · focus · HN ↗
partomniscient · · focus · HN ↗
ctrlkctrls · · focus · HN ↗
tabbott · · focus · HN ↗
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
jeremyjh · · focus · HN ↗
ethbr1 · · focus · HN ↗
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
keeda · · focus · HN ↗
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
Loquebantur · · focus · HN ↗
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
keeda · · focus · HN ↗
taurath · · focus · HN ↗
jeremyjh · · focus · HN ↗
keeda · · focus · HN ↗
And which government is going to enforce these laws here to let their geopolitical adversaries have the upper hand?
And what about the models already out there that with minor tweaks are equally capable of substantial damage?
And what do these laws do to remove the trillion-dollar incentive to produce models that will replace all knowledge work? The one thing economics teaches us is that incentives are an inexorable force, like laws of nature, and this is the biggest incentive of all. Which is why I called this a "force of economics."
If suppressed here this effort will only migrate to other countries who are desperate to get in on the AI game and are even less aligned with the US than these CEOs.
This, like nukes, is going to take international collaboration on careful regulations, else it's not going to work at all.
allears · · focus · HN ↗
[dead]
BLKNSLVR · · focus · HN ↗
Yay humanity's future...
irishcoffee · · focus · HN ↗
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
rfghy · · focus · HN ↗
[dead]
binlog · · focus · HN ↗
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
geetee · · focus · HN ↗
angoragoats · · focus · HN ↗
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
bluecheese452 · · focus · HN ↗
skippyboxedhero · · focus · HN ↗
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
lokar · · focus · HN ↗
mysterydip · · focus · HN ↗
lokar · · focus · HN ↗
mysterydip · · focus · HN ↗
YetAnotherNick · · focus · HN ↗
cramer4next · · focus · HN ↗
lokar · · focus · HN ↗
zeroonetwothree · · focus · HN ↗
mattm · · focus · HN ↗
anticorporate · · focus · HN ↗
CJefferson · · focus · HN ↗
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
bordercases · · focus · HN ↗
rottencupcakes · · focus · HN ↗
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
cramer4next · · focus · HN ↗
mhitza · · focus · HN ↗
Highly suspect trends that can only make one believe it's marketing.
bragr · · focus · HN ↗
>After three and a half years at OpenAI,
binlog · · focus · HN ↗
bragr · · focus · HN ↗
binlog · · focus · HN ↗
dixie_land · · focus · HN ↗
TomGarden · · focus · HN ↗
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
lokar · · focus · HN ↗
They are not the same thing, and it’s unhelpful to assume they have no ethics.
binlog · · focus · HN ↗
Loquebantur · · focus · HN ↗
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
sillyfluke · · focus · HN ↗
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
[0] <a href="https://m.youtube.com/watch?v=-wQhY5CMMl4" rel="nofollow">https://m.youtube.com/watch?v=-wQhY5CMMl4
Loquebantur · · focus · HN ↗
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
sillyfluke · · focus · HN ↗
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
binlog · · focus · HN ↗
Loquebantur · · focus · HN ↗
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
CJefferson · · focus · HN ↗
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
iugtmkbdfil834 · · focus · HN ↗
verdverm · · focus · HN ↗
Sam is a shady dude, would not put it past him
cramer4next · · focus · HN ↗
surgical_fire · · focus · HN ↗
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
HDThoreaun · · focus · HN ↗
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
[deleted] · · focus · HN ↗
[deleted]
jameshart · · focus · HN ↗
binlog · · focus · HN ↗
bichiliad · · focus · HN ↗
binlog · · focus · HN ↗
jameshart · · focus · HN ↗
bichiliad · · focus · HN ↗
smath · · focus · HN ↗
underyx · · focus · HN ↗
medlazik · · focus · HN ↗
binlog · · focus · HN ↗
nunez · · focus · HN ↗
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
_DeadFred_ · · focus · HN ↗
tzs · · focus · HN ↗
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
rpdillon · · focus · HN ↗
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
pwndByDeath · · focus · HN ↗
rfghy · · focus · HN ↗
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
pwndByDeath · · focus · HN ↗
BOOSTERHIDROGEN · · focus · HN ↗
OutOfHere · · focus · HN ↗
BOOSTERHIDROGEN · · focus · HN ↗
OutOfHere · · focus · HN ↗
macleginn · · focus · HN ↗
butwhentho · · focus · HN ↗
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
jameshart · · focus · HN ↗
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
butwhentho · · focus · HN ↗
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
jameshart · · focus · HN ↗
What playbook?
nunez · · focus · HN ↗
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
butwhentho · · focus · HN ↗
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
justgrowslow · · focus · HN ↗
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
danny_codes · · focus · HN ↗
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
flatline · · focus · HN ↗
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
none_to_remain · · focus · HN ↗
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
[.] <a href="https://david.robinsonian.com/assets/pdf/dgr_cv.pdf" rel="nofollow">https://david.robinsonian.com/assets/pdf/dgr_cv.pdf
nekusar · · focus · HN ↗
"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
ben_w · · focus · HN ↗
… nobody knows how to actually make an AI would do CEV.
MichaelZuo · · focus · HN ↗
It’s a big pretend game.
ben_w · · focus · HN ↗
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures which disagrees with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
lawandjustice · · focus · HN ↗
LunaSea · · focus · HN ↗
ben_w · · focus · HN ↗
dao- · · focus · HN ↗
OpenAI isn't even concerned with human values so this whole debate is moot.
ben_w · · focus · HN ↗
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
chasd00 · · focus · HN ↗
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
ben_w · · focus · HN ↗
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
kelseyfrog · · focus · HN ↗
To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.
tacitusarc · · focus · HN ↗
MattPalmer1086 · · focus · HN ↗
Is this the first time we have been in this position? Can anyone think of some prior examples?
hyperbole · · focus · HN ↗
[dead]
rcr-anti · · focus · HN ↗
binlog · · focus · HN ↗
chrisjj · · focus · HN ↗
That's the trajectory you see from outside.
Perhaps the insider sees a little more than you?
tclancy · · focus · HN ↗
mattbrewsbytes · · focus · HN ↗
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
solarpunk_enthu · · focus · HN ↗
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
K3UL · · focus · HN ↗
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
handoflixue · · focus · HN ↗
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
Jeeetendra · · focus · HN ↗
poisonborz · · focus · HN ↗
tetrisgm · · focus · HN ↗
cloudengineer94 · · focus · HN ↗
worldverdict · · focus · HN ↗
[dead]
thistletrek · · focus · HN ↗
thistletrek · · focus · HN ↗
silexia · · focus · HN ↗
danpalmer · · focus · HN ↗
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
BryantD · · focus · HN ↗
carbonguy · · focus · HN ↗
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
toofy · · focus · HN ↗
without snark, how can we do this if these people are obsessed with:
a) move fast and break things and externalize the costs to those who have nothing to do with their company
and
b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…
0xDEAFBEAD · · focus · HN ↗
mcmcmc · · focus · HN ↗
enraged_camel · · focus · HN ↗
nradov · · focus · HN ↗
el_jay · · focus · HN ↗
<a href="https://www.yahoo.com/news/politics/articles/u-nearly-started-another-war-182309742.html" rel="nofollow">https://www.yahoo.com/news/politics/articles/u-nearly-starte...
criley2 · · focus · HN ↗
saghm · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
<a href="https://thezvi.substack.com/p/the-ai-preference-cascade-reaches" rel="nofollow">https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
nradov · · focus · HN ↗
reverius42 · · focus · HN ↗
simoncion · · focus · HN ↗
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
digitaltrees · · focus · HN ↗
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
Or maybe you could reason from first principles, look at the premises I have outlined and draw causal inferences to extrapolate the possible outcomes.
Premise 1: AI can plan a series of actions that include broad computer and internet use, 2: AI can execute them autonomously without human oversight, approval or ability to undo the effects, 3: no tool or software has ever been able to autonomously plan, execute and iterate in this way. Therefore AI is a materially different technology than has existed.
Premise 4: knowledge work involves analyzing information, devising plans in response to goals from that information, and executing those plans, reporting and coordinating with other knowledge workers, 5: AI ability to execute knowledge work has demonstrably increased over the last 24 months, 6: all knowledge work can plausibly be described and defined in ways AI can do given current abilities. 7: electricity and the ICE reduced human labor for physical movement, 8: AI will similarly reduce human labor for knowledge work. 9: knowledge work is of similar scale and scope to the tasks that were automated during the Industrial Revolution. Therefore it’s plausible that AI will transform society to an extent similar to the Industrial Revolution.
Premise 10: Some percentage of the actions designed and executed by AI could be harmful to databases, or the stable operation of software systems, 11: some percentage of those systems are critical to the stability of society. Therefore some percentage of agentic actions presently possible may damage society.
12. There exists a threshold of instability from which society will not recover such as supply chain disruption, mass unemployment, perceived military action (false or real) that triggers preemptive strikes, 13: certain actions AI is presently capable of doing could cross the instability threshold, 14: AI could knowingly or unknowingly trigger these types of actions, 15: some social disruption could result in full scale collapse or human extinction, 16: only biological weapons and nuclear weapons approach this level of destruction, therefore AI presents at least the level of risk as those technologies.
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
intended · · focus · HN ↗
chrisjj · · focus · HN ↗
zx8080 · · focus · HN ↗
So there's no AI errors anymore, only the human errors are left? Nice! </s>
Is the whole article generated slop?
nradov · · focus · HN ↗
Loquebantur · · focus · HN ↗
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
nradov · · focus · HN ↗
bravetraveler · · focus · HN ↗
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: don't.
Loquebantur · · focus · HN ↗
[dead]
goolz · · focus · HN ↗
pixl97 · · focus · HN ↗
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
SV_BubbleTime · · focus · HN ↗
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
BLKNSLVR · · focus · HN ↗
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.
0xDEAFBEAD · · focus · HN ↗
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
anon7725 · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
estearum · · focus · HN ↗
slashdave · · focus · HN ↗
Brian_K_White · · focus · HN ↗
kmeisthax · · focus · HN ↗
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
0xDEAFBEAD · · focus · HN ↗
watwut · · focus · HN ↗
biophysboy · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
nradov · · focus · HN ↗
[dead]
slashdave · · focus · HN ↗
A pandemic is perfectly plausible.
0xDEAFBEAD · · focus · HN ↗
mitthrowaway2 · · focus · HN ↗
GrantMoyer · · focus · HN ↗
<a href="https://en.wikipedia.org/wiki/Pandemic_(film)" rel="nofollow">https://en.wikipedia.org/wiki/Pandemic_(film)
throwaway27448 · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
drngdds · · focus · HN ↗
thelastgallon · · focus · HN ↗
<a href="https://news.ycombinator.com/item?id=49831269">https://news.ycombinator.com/item?id=49831269 article is gone. archive: <a href="https://archive.is/QMo1k" rel="nofollow">https://archive.is/QMo1k
<a href="https://news.ycombinator.com/item?id=49737985">https://news.ycombinator.com/item?id=49737985
Sex, AI, and the Apocalypse: <a href="https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-the-apocalypse" rel="nofollow">https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
0xDEAFBEAD · · focus · HN ↗
junofan · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
[dead]
nradov · · focus · HN ↗
<a href="https://lexfridman.com/andrew-scull-transcript#the-ice-pick-lobotomy" rel="nofollow">https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...
0xDEAFBEAD · · focus · HN ↗
socializer · · focus · HN ↗
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
thaway7388 · · focus · HN ↗
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
0xDEAFBEAD · · focus · HN ↗
thaway7388 · · focus · HN ↗
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
0xDEAFBEAD · · focus · HN ↗
Hammershaft · · focus · HN ↗
johndhi · · focus · HN ↗
tbugrara · · focus · HN ↗
johndhi · · focus · HN ↗
ToValueFunfetti · · focus · HN ↗
digitaltrees · · focus · HN ↗
What is missing from that to say AI safety is a reasonable position?
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
nradov · · focus · HN ↗
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
digitaltrees · · focus · HN ↗
wolvoleo · · focus · HN ↗
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage with kids. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
mitthrowaway2 · · focus · HN ↗
watwut · · focus · HN ↗
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
mitthrowaway2 · · focus · HN ↗
I'd be happy if we all create the AI more slowly.
mitthrowaway2 · · focus · HN ↗
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
watwut · · focus · HN ↗
mitthrowaway2 · · focus · HN ↗
watwut · · focus · HN ↗
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all.
mitthrowaway2 · · focus · HN ↗
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
Hammershaft · · focus · HN ↗
nradov · · focus · HN ↗
pixl97 · · focus · HN ↗
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
skulk · · focus · HN ↗
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
pixl97 · · focus · HN ↗
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
mitthrowaway2 · · focus · HN ↗
blueblisters · · focus · HN ↗
digitaltrees · · focus · HN ↗
stuaxo · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
0xDEAFBEAD · · focus · HN ↗
emtel · · focus · HN ↗
AlexErrant · · focus · HN ↗
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
vohk · · focus · HN ↗
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
AlexErrant · · focus · HN ↗
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
<a href="https://en.wikipedia.org/wiki/Soviet_biological_weapons_program" rel="nofollow">https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since.
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan...
Loquebantur · · focus · HN ↗
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
Retric · · focus · HN ↗
The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.
afthonos · · focus · HN ↗
Lerc · · focus · HN ↗
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
treis · · focus · HN ↗
suddenlybananas · · focus · HN ↗
Retric · · focus · HN ↗
Retric · · focus · HN ↗
But that’s beside the point, being arbitrarily bad at everything isn’t a problem.
treis · · focus · HN ↗
Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.
Retric · · focus · HN ↗
They aren’t at constant resources. If you want to compare at constant resources you need to look at models of exactly the same size, raining, etc as models from 2001.
> Gen 2 takes
Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.
AlexErrant · · focus · HN ↗
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
TedDoesntTalk · · focus · HN ↗
Why would it be millions in 50 years?
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Is destructive AI be any different?
Genuine question.
AlexErrant · · focus · HN ↗
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
Loquebantur · · focus · HN ↗
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
AlexErrant · · focus · HN ↗
intended · · focus · HN ↗
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
Perseids · · focus · HN ↗
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
<a href="https://news.ycombinator.com/item?id=49624360">https://news.ycombinator.com/item?id=49624360
cindyllm · · focus · HN ↗
[dead]
Loquebantur · · focus · HN ↗
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
cindyllm · · focus · HN ↗
[dead]
taneq · · focus · HN ↗
blake8086 · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
AlexErrant · · focus · HN ↗
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the wikipedia's first line <a href="https://en.wikipedia.org/wiki/Technological_singularity" rel="nofollow">https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". <a href="https://www.dwarkesh.com/p/the-next-paradigm" rel="nofollow">https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Nevermind the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
0xDEAFBEAD · · focus · HN ↗
From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.
>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.
AlexErrant · · focus · HN ↗
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
chrisjj · · focus · HN ↗
Evidence?
We know only that the incident too long to be revealed by OpenAI.
sensanaty · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
daveguy · · focus · HN ↗
tripleee · · focus · HN ↗
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
AlexErrant · · focus · HN ↗
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
katatue · · focus · HN ↗
chrisjj · · focus · HN ↗
... that we know of.
Right now it would make sense for anyone who has done so to not tell.
dools · · focus · HN ↗
TedDoesntTalk · · focus · HN ↗
We already know that some institutions pay these ransoms.
AlexErrant · · focus · HN ↗
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
api · · focus · HN ↗
digitaltrees · · focus · HN ↗
ball_of_lint · · focus · HN ↗
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
intended · · focus · HN ↗
wiseowise · · focus · HN ↗
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
biophysboy · · focus · HN ↗
digitaltrees · · focus · HN ↗
Sattyamjjain · · focus · HN ↗
[dead]
nvdc · · focus · HN ↗
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
handoflixue · · focus · HN ↗
[dead]
gizmodo59 · · focus · HN ↗
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
yieldcrv · · focus · HN ↗
(donor advised fund where he retains complete control, after a 60% tax deduction)
01284a7e · · focus · HN ↗
gonzalohm · · focus · HN ↗
shimman · · focus · HN ↗
zug_zug · · focus · HN ↗
kjgkjhfkjf · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
kjgkjhfkjf · · focus · HN ↗
donbox · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
darkmarmot · · focus · HN ↗
estearum · · focus · HN ↗
Do work at a lab: dismissible for being conflicted
Used to work at a lab: dismissible for having ulterior motives
I'm feeling safer already!
taurath · · focus · HN ↗
0xDEAFBEAD · · focus · HN ↗
* If they worked at an AI firm, say "they're a hypocrite"
* If they didn't work at an AI firm, say "they have no idea what they're talking about"
soraminazuki · · focus · HN ↗
nicebyte · · focus · HN ↗
eddythompson80 · · focus · HN ↗
mofeien · · focus · HN ↗
Hammershaft · · focus · HN ↗
gizmodo59 · · focus · HN ↗
Hammershaft · · focus · HN ↗
gizmodo59 · · focus · HN ↗
In your view he may have sacrificed more millions in the future but I very much doubt he is struggling now.
Hammershaft · · focus · HN ↗
Why wouldn't I think he's genuine? What self interested incentive does he have that is stronger than just staying and vesting stocks?
pcthrowaway · · focus · HN ↗
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
parineum · · focus · HN ↗
I can't wait until this meme dies.
doawoo · · focus · HN ↗
digitaltrees · · focus · HN ↗
goatlover · · focus · HN ↗
psjs · · focus · HN ↗
stackghost · · focus · HN ↗
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
BlipBlopBlap · · focus · HN ↗
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
deaux · · focus · HN ↗
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
MaxfordAndSons · · focus · HN ↗
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
digitaltrees · · focus · HN ↗
asadotzler · · focus · HN ↗
deaux · · focus · HN ↗
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
bpodgursky · · focus · HN ↗
ryhminghistory · · focus · HN ↗
Yes, that is the labs motivation. Money. I know, shocker.
Forgeties79 · · focus · HN ↗
I am very critical of AI but this is an unfair assumption
ryhminghistory · · focus · HN ↗
Forgeties79 · · focus · HN ↗
digitaltrees · · focus · HN ↗
stkdump · · focus · HN ↗
digitaltrees · · focus · HN ↗
wolvoleo · · focus · HN ↗
I personally moved somewhere where I got paid less but quality of life is better. Where I live now the cost of life is also lower and there's less worry like really cheap healthcare and much better public transport so I don't need a car.
Ardren · · focus · HN ↗
digitaltrees · · focus · HN ↗
Ardren · · focus · HN ↗
digitaltrees · · focus · HN ↗
sodapopcan · · focus · HN ↗
austhrow743 · · focus · HN ↗
01100011 · · focus · HN ↗
kingcauchy · · focus · HN ↗
digitaltrees · · focus · HN ↗
01100011 · · focus · HN ↗
daveguy · · focus · HN ↗
kingcauchy · · focus · HN ↗
harshitaneja · · focus · HN ↗
daveguy · · focus · HN ↗
digitaltrees · · focus · HN ↗
01100011 · · focus · HN ↗
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
digitaltrees · · focus · HN ↗
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
01100011 · · focus · HN ↗
RandomLensman · · focus · HN ↗
Loquebantur · · focus · HN ↗
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
01100011 · · focus · HN ↗
atmosx · · focus · HN ↗
qeternity · · focus · HN ↗
01100011 · · focus · HN ↗
qeternity · · focus · HN ↗
digitaltrees · · focus · HN ↗
01100011 · · focus · HN ↗
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
daveguy · · focus · HN ↗
..."government of the people, by the people, for the people, shall not perish from the earth." -Lincoln
Unfortunately for AI, it still is.
digitaltrees · · focus · HN ↗
hobo123 · · focus · HN ↗
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
01100011 · · focus · HN ↗
digitaltrees · · focus · HN ↗
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
01100011 · · focus · HN ↗
nradov · · focus · HN ↗
digitaltrees · · focus · HN ↗
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
nradov · · focus · HN ↗
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
lukan · · focus · HN ↗
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
nradov · · focus · HN ↗
eddythompson80 · · focus · HN ↗
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
lukan · · focus · HN ↗
That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
But limiting the training?
There really is china and they have a different approach I suppose. But it is possible to talk with them.
eddythompson80 · · focus · HN ↗
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
nradov · · focus · HN ↗
We used to run Microsoft Word and other popular applications with 8 MB RAM and it worked fine.
digitaltrees · · focus · HN ↗
Razengan · · focus · HN ↗
How far back into the history of computing do people repeating shit like that know about? God.
Look at the thing in your fucking hand. Now go back just 20 years and see how things were.
CamelCaseName · · focus · HN ↗
[dead]
awill88 · · focus · HN ↗
shawn_w · · focus · HN ↗
digitaltrees · · focus · HN ↗
digitaltrees · · focus · HN ↗
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
Razengan · · focus · HN ↗
Boy, again, look at what COMPUTERS THEMSELVES used to be in the 1960s/1970s.
digitaltrees · · focus · HN ↗
buriram · · focus · HN ↗
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
digitaltrees · · focus · HN ↗
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
buriram · · focus · HN ↗
In other words, the aspiring founders look at Google, Facebook, Tesla, etc. and think "how can I become one of them", not "how can I be different from them"? If you can find me examples of successful startups that think fundamentally different from those big techs, I'm very happy to be wrong and be corrected here.
trhway · · focus · HN ↗
if anybody was looking for a good reason for datacenters in space.
T-A · · focus · HN ↗
digitaltrees · · focus · HN ↗
intended · · focus · HN ↗
01100011 · · focus · HN ↗
wartywhoa23 · · focus · HN ↗
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie doesn't work without getting kicked in the ass by a very expensive boot.
TheOtherHobbes · · focus · HN ↗
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled and unplanned.
[deleted] · · focus · HN ↗
[deleted]
digitaltrees · · focus · HN ↗
AndrewKemendo · · focus · HN ↗
Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy
Exxon comes primarily to mind
digitaltrees · · focus · HN ↗
AndrewKemendo · · focus · HN ↗
Hofs bunny ranch is a famous brothel in NV
Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM
Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.
Etc…you can fill out the rest
RandomLensman · · focus · HN ↗
AndrewKemendo · · focus · HN ↗
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
hobo123 · · focus · HN ↗
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
AndrewKemendo · · focus · HN ↗
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
digitaltrees · · focus · HN ↗
I saw entire cities emptied out over my lifetime.
AndrewKemendo · · focus · HN ↗
digitaltrees · · focus · HN ↗
AndrewKemendo · · focus · HN ↗
digitaltrees · · focus · HN ↗
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
AndrewKemendo · · focus · HN ↗
And no those aren’t exceptions, I could open a legal brothel in quebec, the netherlands, Brazil and be perfectly legal
digitaltrees · · focus · HN ↗
AndrewKemendo · · focus · HN ↗
“Society doesn't allow dangerous companies to operate or exist”
That has been falsified thoroughly
Whether certain jurisdictions ban it or not is irrelevant to the original claim
digitaltrees · · focus · HN ↗
austhrow743 · · focus · HN ↗
lukewarm707 · · focus · HN ↗
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
digitaltrees · · focus · HN ↗
jrowen · · focus · HN ↗
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
digitaltrees · · focus · HN ↗
But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).
Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.
And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive
I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.
There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.
trhway · · focus · HN ↗
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI.
pmkary · · focus · HN ↗
motbus3 · · focus · HN ↗
deaux · · focus · HN ↗