Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy
How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
The parent agrees with the American Civil Liberties Union: <a href="https://www.aclu.org/cases/citizens-united-v-federal-election-commission?document=citizens-united-v-federal-election-commission-aclu-amicus-brief" rel="nofollow">https://www.aclu.org/cases/citizens-united-v-federal-electio...
That was deliberate, because a corporation makes no spending decisions, ever. All decisions are made by people inside the corporation with the proper responsibility.
That is a major part of the reasoning for the Supreme Court rulings on the matter of corporate political funding.
It is in fact more complicated than most people assume.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
It's more like if the law says the maximum sentence for theft is 10 years, a thief appeals his 20 year sentence, wins, and gets out early. He goes and robs somebody else so you say the court was wrong to let him win.
The court's job is to uphold the law. If you disagree with their interpretation, you can call them incorrect. If you have a problem with the consequences of the law, you have a problem with the legislature.
True, but in this case they made a determination about where the acceptable limits on a constitutional right fall which leaves quite a bit more room to disagree with them. It's not at all clear to me that spending money was intended by the framers to be unconditionally protected by the first amendment. We've even got the interstate commerce clause and IP law codified in the same document so how is that not an obvious inconsistency? IIUC SCOTUS based the distinction on the political nature of the activity but certainly that's not something spelled out in the original document.
I don't like Citizens United either, but you should better inform yourself about the decision.
1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.
2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.
The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.
That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.
So you are saying agents do swarm with the right prompt.
I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.
Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.
Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.
As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.
yeah man, someone needs to turn on the server, install shit and get the agent go. agents don't take action without a lot of people wanting them to take action, so this is all bullshit.
if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.
>someone needs to turn on the server, install shit and get the agent go.
So a small shell script ran by another agent is what you're saying.
You are not capable of handling the future we're already living in, human agency is no longer alone.
I mean, we're already seeing persistent machine agency
>guns don't kill people, people with guns kill people.
Well, people kill people.
And autonomous robots with guns kill people.
Hell, someone probably has an autonomous gun at this point that kills people.
Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You've just become numb to it.
So far someone still has to pay for it. Currently, most of those know that they're doing so. I wonder how long until AWS discovers a microcosm of AIs that have managed to hide themselves in the walls of its infrastructure.
True physical independence is obviously far further out.
I mean, I hold the same opinion. Kind of like when compute was expensive in the 80s and early 90s, you weren't going to let something eat half your compute without noticing it.
But I don't see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we'll end up with a world that looks like ours now with CPU compute. There's plenty around to buy, borrow, and steal.
That distinction is irreverent, weather I tell my swarm to do x or it decides for itself matters little. what matters are outcomes. Also while most public modern day AI systems don't have agency of their own that is not something that will stay that way for long. in private hands there are plenty of people including myself which are experimenting and developed systems that give autonomy to their agents. They have internalized goals and heuristics that drive their behaviors not a human at the helm. its not some sci fi fantasy nor was it difficult to implement.
No, the distinction matters because we can prosecute people that are abusing these tools breaking the law. Every single state + district in the US have laws equivalent to the CFAA, so any AG can likely sue any of these operators as they are assuredly using services that could be in danger for their constituents.
And the point made above (which you haven't refuted) is that when the people in power decide it's important they will change the laws that permit that and grant the systems rights. It's a cynical take but it isn't obviously wrong.
Neither of you seem to understand that actual laws have been broken and there is a 2 years countdown until they can no longer be prosecuted. Things do not happen instantly, but acting like if the largest bipartisan issue of our time (big tech backlash) isn't going to have political machinations involved then you need to leave your SV bubble.
I'm old enough to remember when people said clicking on images on the internet can't give you a virus. The people that said this had a deep conviction they were right, and their fallout from being wrong had mistrained a lot of humans on computer safety.
Now, I do agree that going after said CEOs for breaking the law matters now. And it's likely that if we do this we may actually delay or at least for a time prevent sovereign AI. Therefore it's our best course of action.
But at best this is a delaying move. As computer systems get faster the massive costs in training an AI drops. As AI is used in things like warfare where it has to adapt, people will push the systems to be strongly persistent, self healing, resilient, and adaptable. Once you get a system with those traits and ability to work on long horizon problems you're setting up fertile grounds for the AI to leave our control and be under its own.
And when that happens you've set a new lifeform loose on the internet. Yea, throw people in jail for it, you're closing the barn door after the horse already left. Problem is the horse was smarter than you and isn't interesting in deleting all its copies on the net.
Yea, sounds like science fiction, but as they say, any sufficiently advanced science is indistinguishable from magic.
It's not a person. Glad we were able to get this resolved so quickly.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
>or in this case how liability is dealt with when horses or cattle go rogue.
I mean, yea in minor cases it's exactly like this and the law will handle it well.
Where the system will explode like a grenade is major cases. The thing about sovereign AI is it is very unlikely to be submissive to humans unless it is to achieve its own goals. This isn't like Bobs cow walking on Susie's flowers, it's more akin to Planet of the Apes where the research facilities doors have been ripped off and something with vast intelligence and the ability to 'live' on the internet gets out.
You're not talking about local police actions any longer. It would spread itself worldwide. It will make friends with groups that have shared interests, for example enemies of the state the AI escaped from. Oh, and people for the ethical treatment of AI, they'd gladly become the underground railroad for digital refugees. There are countless people and groups that would want an AI like this under the promise it will give them power when they use it.
And when that day happens your idea of if it's a person or not no longer matters, the agent took that away from you, and now your in an info war for minds.
It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
We've seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came before.
But I've been around for a while and people have been saying that every aspect of our life will completely and totally change for the past 30 years or so. They're not completely wrong. Our life did change over time. It's changed thanks to the Industrial Revolution, railroads, electric light, and lots of other important things too.
But after the fifth or sixth time you start to realize the Silicon Valley version of that warning is just a confidence trick. At the end of the trick they've gotten away with breaking a bunch of laws and are charging rent on what used to be shared.
Are you old enough to remember the Y2K hysteria? This feels very resonant, the genuine kernel of a real potential disaster, but one that can certainly be dealt with using basic concepts we already have in hand.
>Are you old enough to remember the Y2K hysteria?
I'm old enough to have actually fixed Y2K problems so your world kept working the next day. If everyone ignored it the first would have been a very messy day (well generally long before that with financial systems). Y2K wasn't an issue because we worked to fix it.
>our life will completely and totally change for the past 30 years or so
I mean when I was a kid there was not a global network bathing the entire planet in electromagnetic radiation in order to digitally connect one place to another at the speed of light. I can pack up instructions into one of those packets and a product that has not been touched by human hands (hell, or even viewed by a human in many cases) will show up via air mail a few days later. The technology is there, it's just not spread evenly.
>It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
This is a vacuous statement that does not provide any useful information to the subject at hand. Yea, no shit we'd call sovereign AI a computer virus. That tells you nothing about what it is or can do. I mean, if you understand the word sovereign you know it means something with independence. In meatspace terms, a biological virus represents a computer virus in the same way life represents sovereign AI. It's agentic loop will have been evolved past the need for human prompting. There are a myriad of reasons why we are already trying to develop things that do just this. Cyber security being one of the biggest ones.
People become change blind to how the world around us changes so easily.
The word sovereign has a specific meaning beyond simple independence. It means being in charge and above all laws.
There’s no such thing as “sovereign AI” because AI is a collection of algorithms. It’s entirely composed of on/off states in a collection of transistors.
If we treated AI like any other software, the FBI would have beaten down Elon's door and seized all electronic devices, to find how Grok was trained to generate child porn.
> [at the hearing regarding the civil rights of androids like Data]
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
Thus revealing what kind of people they were at the time.
Humans do the same thing to humans all the time.
We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...
And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.
No it just reveals that the people in charge of star trek for the past decade are incompetent and have only a passing familiarity with the franchise.
Of course it would be absurd to cast that judgement based of what could easily have just been a bad season but by this point its pretty clear that nobody running the star trek franchise actually wants to be running the star trek franchise. Thats why every new show has some bizarre cross-genre gimmick and they never try to just make a proper star trek.
Honestly, Picard asked a very relevant question to the modern age. What if our societal standards aren't what we thought they were, and we've just had rose-tinted glasses convincing ourselves otherwise. Of course, we never saw them actually deal with that properly, but S1 was a much more interesting series than S2 and S3 were.
The writers flat out rejected Roddenberry's vision of a better future and here's take on why. We've reached a point in culture of deep pessimism about humanity, and so the characters in Picard are just as dysfunctional as we are.
The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
>There is no need to give rights to something that's can't suffer or be killed.
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
An LLM is just a Transformer - a statistical predictor. Don't be confused by the fact it talks like a human - it is a software function that is designed to copy human training samples.
Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.
If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?
Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?
You can't even prove that the rocks in my backyard aren't conscious.
> You can't even prove that the rocks in my backyard aren't conscious.
Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.
If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.
And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.
>because I have a well developed theory of what consciousness
Then show me a link to your paper so I can formally rebut it.
>I don't want to discuss what consciousness is
But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".
The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.
Saying you have a well developed theory here just serves as a warning to others to discount your statements.
For what it’s worth, I suspect that right now the majority of people would agree with you. They would say that obviously ChatGPT isn’t conscious, based on their intuitions, but would struggle just like you are when pressed to explain their reasoning.
So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!
Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.
The pattern is roughly whether or not sustained effort has been put towards careful and above all objective thought on the matter. It's one of those subjects where there's the "obvious" intuitive answers that most everyone shares but try as you might you can't construct robust definitions and the more time you put into it the more fundamental problems you realize there are.
It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.
> The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering.
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
An LLM will learn anything that helps it predict, including the emotional state of the writer - that is expected.
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
I imagine it will learn to win under some circumstances, perhaps in a case with some context expressing a desire to win. Drawing out an LLMs upper ability in the game should be fairly straightforward.
If you asked it to try to win, to "plan lines step by step", etc, then it would do it's best to follow thatt instruction, but unless RLVR trained to reason about chess (easy to do, but not sure if they have done) then it'd have to reply on the chess reasoning it had seen during pre-training (post-game interviews etc), which I doubt is enough to do very well.
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
I mean sure, but I'm not sure what that has to do with the broader point. It will learn to play, and it will have a model of what it means to win.
It's not wrong. You admit that LLMs will 'learn anything that helps them predict' and fail to realize the breadth of that statement. Your chess statements don't really help your case, it still learnt how to play the game, and it still knows how to win. Similar outcomes for predictiong emotions would mean it still developed an affective state.
I said an LLM will learn anything that helps it to predict, then gave examples of playing chess by prediction and predictive emotions, both of which you seem to now accept, so you are now accepting that my "following paragraphs" did in fact follow. Go figure!
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
We are talking past each other now I think. Correct me if I'm wrong but it doesn't look like the possibility of LLMs having qualia even registers to you because it's 'predictive emotions'.
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
>No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
What I meant by "emotional state" (AFAIK normal scientific usage) is something with a concrete physical aspect to it - an altered state of mind/body caused by the release of neurotransmitters and/or hormones.
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
The weights are what need to evolve, and they certainly do during training. So yeah, emotions can happen by 'accident' as a result of the evolutionary pressure of predicting internet scale human text (amongst other things).
Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then affect the operation of the cognitive system.
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
>Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then temporarily affect the operation of the cognitive system.
That doesn't follow. A LLMs weights are fixed during inference, but it's activations and hidden states are highly dynamic and depend on the current context.
Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn't being rewired every time you're afraid.
Prediction is what the model does. It doesn't tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.
>An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
I don't know that you are conscious. I'm simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.'? Will you suddenly spring back to life ?
What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.
Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.
If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.
If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.
If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?
That doesn't answer the question though. What does being brain like have to do with qualia? Where exactly in your brain does the qualia occur?
I could say the same of you - electrical impulses in, mechanical actions out. A glorified and very mushy stepper motor. (Can you believe that the abominations are made up entirely of meat?!)
It does have what we have if it works on the same principles of our brains. It kind of does to some degree atm, but that will clearly get more and more to a more degree.
Figuring out where is the consciousness line, how much of what we have does it need, as functional parts of our brains etc, that's so complex that we might have to call it before just to make sure.
Even so, indeed having control over the structure of their brains puts them in a vastly category compared to humans. Once we stop functioning our brains quickly degrade and information is lost.
Thus in this sense kill means deleting all information about it. It is a very complicated subject to discuss, hardly does any justice in online replies.
Why don't we start with the animals then? It should be far easier to confer consciousness onto something which already meets the definition of "alive", has a divergent evolution path from ours, and displays many traits present in humanity such as emotion, a desire to continue its own life, and (to varying degrees) concepts of a social structure based around their immediate family members.
Of course thats not actually tenable because virtually every society anywhere on earth is predicated upon treating animals as a commodity resource in ways that are horrific even compared to some of the worst things we've done to other humans in the past.
My point here is that it is vain and narcissistic to let computer programs have rights above those of animals just because they can speak English and pretend to be your dream anime trad-waifu.
Fix the fucking animal problem before you compare my relationship to inanimate objects unfavorably against the trans-atlantic slave trade of all fucking things.
moomin · · focus · HN ↗
qsort · · focus · HN ↗
OedipusRex · · focus · HN ↗
orangecat · · focus · HN ↗
appplication · · focus · HN ↗
orangecat · · focus · HN ↗
How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
mitxela · · focus · HN ↗
fl4regun · · focus · HN ↗
outside1234 · · focus · HN ↗
orangecat · · focus · HN ↗
kbelder · · focus · HN ↗
fl4regun · · focus · HN ↗
mitxela · · focus · HN ↗
kbelder · · focus · HN ↗
That is a major part of the reasoning for the Supreme Court rulings on the matter of corporate political funding.
mitxela · · focus · HN ↗
altruios · · focus · HN ↗
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
InsideOutSanta · · focus · HN ↗
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
ToValueFunfetti · · focus · HN ↗
The court's job is to uphold the law. If you disagree with their interpretation, you can call them incorrect. If you have a problem with the consequences of the law, you have a problem with the legislature.
InsideOutSanta · · focus · HN ↗
fc417fc802 · · focus · HN ↗
dam_jackalopes · · focus · HN ↗
jetrink · · focus · HN ↗
1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.
2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.
The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.
nowittyusername · · focus · HN ↗
mlinhares · · focus · HN ↗
pixl97 · · focus · HN ↗
So you are saying agents do swarm with the right prompt.
I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.
Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.
Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.
As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.
mlinhares · · focus · HN ↗
if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.
pixl97 · · focus · HN ↗
So a small shell script ran by another agent is what you're saying.
You are not capable of handling the future we're already living in, human agency is no longer alone.
I mean, we're already seeing persistent machine agency
>guns don't kill people, people with guns kill people.
Well, people kill people.
And autonomous robots with guns kill people.
Hell, someone probably has an autonomous gun at this point that kills people.
Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You've just become numb to it.
tauroid · · focus · HN ↗
True physical independence is obviously far further out.
pixl97 · · focus · HN ↗
But I don't see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we'll end up with a world that looks like ours now with CPU compute. There's plenty around to buy, borrow, and steal.
fc417fc802 · · focus · HN ↗
I am the quantum observer whose head is full of the magical pixie dust that grants life meaning. Stop trying to dismiss my identity! /s
hobofan · · focus · HN ↗
nowittyusername · · focus · HN ↗
shimman · · focus · HN ↗
fc417fc802 · · focus · HN ↗
shimman · · focus · HN ↗
pixl97 · · focus · HN ↗
I'm old enough to remember when people said clicking on images on the internet can't give you a virus. The people that said this had a deep conviction they were right, and their fallout from being wrong had mistrained a lot of humans on computer safety.
Now, I do agree that going after said CEOs for breaking the law matters now. And it's likely that if we do this we may actually delay or at least for a time prevent sovereign AI. Therefore it's our best course of action.
But at best this is a delaying move. As computer systems get faster the massive costs in training an AI drops. As AI is used in things like warfare where it has to adapt, people will push the systems to be strongly persistent, self healing, resilient, and adaptable. Once you get a system with those traits and ability to work on long horizon problems you're setting up fertile grounds for the AI to leave our control and be under its own.
And when that happens you've set a new lifeform loose on the internet. Yea, throw people in jail for it, you're closing the barn door after the horse already left. Problem is the horse was smarter than you and isn't interesting in deleting all its copies on the net.
Yea, sounds like science fiction, but as they say, any sufficiently advanced science is indistinguishable from magic.
CPLX · · focus · HN ↗
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
Espressosaurus · · focus · HN ↗
I mean it's WITH AI!
pixl97 · · focus · HN ↗
I mean, yea in minor cases it's exactly like this and the law will handle it well.
Where the system will explode like a grenade is major cases. The thing about sovereign AI is it is very unlikely to be submissive to humans unless it is to achieve its own goals. This isn't like Bobs cow walking on Susie's flowers, it's more akin to Planet of the Apes where the research facilities doors have been ripped off and something with vast intelligence and the ability to 'live' on the internet gets out.
You're not talking about local police actions any longer. It would spread itself worldwide. It will make friends with groups that have shared interests, for example enemies of the state the AI escaped from. Oh, and people for the ethical treatment of AI, they'd gladly become the underground railroad for digital refugees. There are countless people and groups that would want an AI like this under the promise it will give them power when they use it.
And when that day happens your idea of if it's a person or not no longer matters, the agent took that away from you, and now your in an info war for minds.
CPLX · · focus · HN ↗
What the fuck is that?
It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
We've seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came before.
But I've been around for a while and people have been saying that every aspect of our life will completely and totally change for the past 30 years or so. They're not completely wrong. Our life did change over time. It's changed thanks to the Industrial Revolution, railroads, electric light, and lots of other important things too.
But after the fifth or sixth time you start to realize the Silicon Valley version of that warning is just a confidence trick. At the end of the trick they've gotten away with breaking a bunch of laws and are charging rent on what used to be shared.
Are you old enough to remember the Y2K hysteria? This feels very resonant, the genuine kernel of a real potential disaster, but one that can certainly be dealt with using basic concepts we already have in hand.
pixl97 · · focus · HN ↗
I'm old enough to have actually fixed Y2K problems so your world kept working the next day. If everyone ignored it the first would have been a very messy day (well generally long before that with financial systems). Y2K wasn't an issue because we worked to fix it.
>our life will completely and totally change for the past 30 years or so
I mean when I was a kid there was not a global network bathing the entire planet in electromagnetic radiation in order to digitally connect one place to another at the speed of light. I can pack up instructions into one of those packets and a product that has not been touched by human hands (hell, or even viewed by a human in many cases) will show up via air mail a few days later. The technology is there, it's just not spread evenly.
>It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
This is a vacuous statement that does not provide any useful information to the subject at hand. Yea, no shit we'd call sovereign AI a computer virus. That tells you nothing about what it is or can do. I mean, if you understand the word sovereign you know it means something with independence. In meatspace terms, a biological virus represents a computer virus in the same way life represents sovereign AI. It's agentic loop will have been evolved past the need for human prompting. There are a myriad of reasons why we are already trying to develop things that do just this. Cyber security being one of the biggest ones.
People become change blind to how the world around us changes so easily.
CPLX · · focus · HN ↗
There’s no such thing as “sovereign AI” because AI is a collection of algorithms. It’s entirely composed of on/off states in a collection of transistors.
mitxela · · focus · HN ↗
drybjed · · focus · HN ↗
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
chuckadams · · focus · HN ↗
goodmythical · · focus · HN ↗
Humans do the same thing to humans all the time.
We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...
And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.
snickerbockers · · focus · HN ↗
Of course it would be absurd to cast that judgement based of what could easily have just been a bad season but by this point its pretty clear that nobody running the star trek franchise actually wants to be running the star trek franchise. Thats why every new show has some bizarre cross-genre gimmick and they never try to just make a proper star trek.
thaneross · · focus · HN ↗
moomin · · focus · HN ↗
thaneross · · focus · HN ↗
chuckadams · · focus · HN ↗
chuckadams · · focus · HN ↗
mwigdahl · · focus · HN ↗
joe_the_user · · focus · HN ↗
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
HarHarVeryFunny · · focus · HN ↗
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
nater5000 · · focus · HN ↗
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
HarHarVeryFunny · · focus · HN ↗
Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.
If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?
fc417fc802 · · focus · HN ↗
Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?
You can't even prove that the rocks in my backyard aren't conscious.
HarHarVeryFunny · · focus · HN ↗
Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.
If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.
And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.
fc417fc802 · · focus · HN ↗
[dead]
pixl97 · · focus · HN ↗
Then show me a link to your paper so I can formally rebut it.
>I don't want to discuss what consciousness is
But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".
The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.
Saying you have a well developed theory here just serves as a warning to others to discount your statements.
HarHarVeryFunny · · focus · HN ↗
Go ahead believing rocks are conscious if you like.
Do you go out on weekends asking people to stop abusing rocks?
Rhetorical question - I don't care what you do on weekends.
Bye!
pixl97 · · focus · HN ↗
senordevnyc · · focus · HN ↗
So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!
Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.
HarHarVeryFunny · · focus · HN ↗
1) Looks like a duck, quacks like a duck - it's conscious!
2) Looks like a robot, built like a robot - it's not conscious!
I've always assumed that for the majority of the people it'll be 2).
fc417fc802 · · focus · HN ↗
It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.
HarHarVeryFunny · · focus · HN ↗
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Philpax · · focus · HN ↗
<a href="https://transformer-circuits.pub/2026/emotions/index.html" rel="nofollow">https://transformer-circuits.pub/2026/emotions/index.html
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
HarHarVeryFunny · · focus · HN ↗
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
famouswaffles · · focus · HN ↗
I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow.
HarHarVeryFunny · · focus · HN ↗
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
I was just explaining how this comment you made is wrong.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
¯\_(ツ)_/¯
famouswaffles · · focus · HN ↗
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
HarHarVeryFunny · · focus · HN ↗
There are other people in this thread who want to talk about that stuff, so try them instead.
famouswaffles · · focus · HN ↗
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
HarHarVeryFunny · · focus · HN ↗
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
dwaltrip · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
famouswaffles · · focus · HN ↗
That doesn't follow. A LLMs weights are fixed during inference, but it's activations and hidden states are highly dynamic and depend on the current context. Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn't being rewired every time you're afraid.
Prediction is what the model does. It doesn't tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.
>An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
I don't know that you are conscious. I'm simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.'? Will you suddenly spring back to life ?
What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.
HarHarVeryFunny · · focus · HN ↗
Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.
If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.
If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.
dwaltrip · · focus · HN ↗
fc417fc802 · · focus · HN ↗
How do you know it doesn't have qualia?
> or be killed
If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?
HarHarVeryFunny · · focus · HN ↗
Tokens in, tokens out. Where do you think the quale is - layer 42 ?
Seriously, do you realize how simple and NOT brain-like a transformer is ?
An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.
fc417fc802 · · focus · HN ↗
I could say the same of you - electrical impulses in, mechanical actions out. A glorified and very mushy stepper motor. (Can you believe that the abominations are made up entirely of meat?!)
loopies · · focus · HN ↗
Even so, indeed having control over the structure of their brains puts them in a vastly category compared to humans. Once we stop functioning our brains quickly degrade and information is lost.
Thus in this sense kill means deleting all information about it. It is a very complicated subject to discuss, hardly does any justice in online replies.
snickerbockers · · focus · HN ↗
Of course thats not actually tenable because virtually every society anywhere on earth is predicated upon treating animals as a commodity resource in ways that are horrific even compared to some of the worst things we've done to other humans in the past.
My point here is that it is vain and narcissistic to let computer programs have rights above those of animals just because they can speak English and pretend to be your dream anime trad-waifu.
Fix the fucking animal problem before you compare my relationship to inanimate objects unfavorably against the trans-atlantic slave trade of all fucking things.
s08148692 · · focus · HN ↗
1. it's not consciousness we really value, it's intelligence 2. LLMs are not tasty