Why is it a problem that the Chinese labs are just distilling down Anthropic’s models? Aren’t Anthropic’s models not just distilling down other people’s work?
Feels like Anthropic crying do as I say not as I do.
It ain't gonna happen. At work I have a dropdown menu in vscode with a dozen models to use interchangeably. They're all essentially commodities and will compete on price and squash almost all profit margin.
That's not their business model. They won't win on price, but they won't compete on price. Their business model is making the current state of the art.
If I'm a business and I need something done today, and bc Anthropic has the best model, there's a 99.9 chance it will be completed successfully for $1000. And using Deepseek there's a 70% chance it will, for $10 - you or me will go for the $10. Big businesses don't. Bc 1000 per task is nothing to them.
Yes, large corporations frequently pay orders of magnitude more for slightly better software. That's why Oracle produces the best stuff on the planet.
The real issue is that Deepseek has a 99.7% chance. So I can run it 10 times until it works and still pay 1/10 the money.
Actually opposite occurs. Big businesses are ok with a mediocre but cheaper result. Very few are willing to pay such cost. Just look at tech wages and the distributions
Except businesses are going the opposite direction here. The lack of stickiness makes the “premium” argument hard to play. Oracle won because swapping databases is a giant PITA. Swapping models requires almost no effort for most uses. And because of that enterprises are all building model marketplaces where providers have to compete on price performance.
Most folks I know can choose from any of the big labs or open weight models and they get billed internally for tokens against their budget. There’s little incentive to no switch to the lower cost closers.
This setup is a nightmare scenario for the big labs trying to execute the traditional enterprise sales plays. Those only work if your product is sticky and AI models are one of the least sticky things in the history of tech.
Big business doesn't pay more for better, but the do pay more for predictability, support and targeted outcomes. They will happily trade a chance at 100% better results for 10% less chance of unplanned outcomes
The business I work for is absolutely sensitive to 10 vs 1000, depending on the task. And it's a multi-billion dollar business. 1000/task may not be much on it's own, but there are a lot of tasks.
What Anthropic is doing requires way more resources than what the Chinese labs are doing. So their complaint is that they do 95% of the work and the Chinese labs do the last 5% and call it their own.
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
From one perspective, the 5% estimation is near-infinite orders of magnitude off, since they've trained on something approaching the sum-total of human knowledge.
More like google (when google just served content off of the website you were searching for), but also fed every form of media, turned up to 11, and given agency far beyond retrieval.
I get that angle but it’s a weak argument as Anthropic is doing the same to others. Also while there’s certainly a lot of computing power needed to do what Anthropic does, it’s increasingly clear there isn’t much secret sauce involved. Everyone knows how do to the core work it’s just a question of who wants to burn billions on compute to do it.
Anthropic’s anger here seems mostly rooted in their annoyance that this exposes they don’t really have core IP that’s not just easily replicated. And that’s clearly a problem for a deeply unprofitable company trying to convince people they’re worth $2 trillion.
Yeah, the whole thing seems like a human centipede of rug pulling. Probably the same as it's always been. Curating AI knowledge should be something that we put our best researchers towards, but realistically I think we wind up with 2-3 highly biased nationalistic models that are constantly copying off each other's notes.
Thanks, that’s the nature of any business. Founders see a way to take existing knowledge and expertise, combine it in some novel or interesting way, and produce a new product
Napster was a fantastic and disruptive product, the likes of which arguably has no equal to this day. But eventually the hammer came down from the courts and it was replaced by streaming services like Netflix, which pay to license materials from their creators.
SOTA models cost hundreds of millions to train. Did creating the contents of the text corpus they were trained on really cost an equivalent of 20x as much (~10 billions)? I honestly don’t know, but I could imagine it having been significantly less.
This isn’t meant as a moral argument, just musing about the relative cost comparison.
Pre vaccination smallpox killed hundreds of millions of people just in the twentieth century [0], the knowledge that allowed for the creation of just that vaccine is worth hundreds trillions of dollars in humans lives, let alone all of `the knowledge and experiences those people were involved in.
The knowledge that created the Haber-Bosch process [1] helps to sustain the majority of the world's populous, add another five hundred trillion dollars for that just to start with.
The creation of the printing press and all written information that allowed it to be built provided dissemination of knowledge beyond the ultra wealthy and is worth a non-finite amount of money.
LLM's are cool math, but they are less than a rounding error in comparison to even the tiniest sliver of human knowledge and technological output.
Shush, this is Hackernews. We all want to have our egos stoked that our industry is the most important in human history and that tech will transform and save us all. Go away with your historical analysis /s
do you also count eg published results of very expensive physics experiments? because once the costs of things like these are taken into account, we are way over 10 billions.
If you look at movies alone that would easily surpass 10s of billions. The cost of most books is probably more nebulous, but books, research, and more all have time and money spent to create them. I would guess the corpus of all media from the 20th century on would be minimally in the hundreds of billions of dollars.
I would argue that producing the complete written corpus on which they at least intend to train (even if some is still out of reach) cost literally everything to produce.
And the monetary cost doesn't even register when weighed against the blood, sweat and tears that went into capturing the authentic experiences of real human beings, whose honest expressions are now at least in some cases getting hoovered up, ingested, and then destroyed for all eternity, for fear that this specific work is the rounding error that might give an equally immoral competitor the edge in the bicycle-riding flamingo race that is currently consuming an absurd amount of the world's creativity and attention.
It probably cost vastly more than training the LLM. You need to consider the time people spent and perhaps weigh it in their hourly wage. Accumulate that across all the training data and it will be a mindboggling amount, compared to training the model.
But if you take it deeper, didn't most of those authors rely on the work of others? Most of human knowledge is small advancements of things we already knew. Often by reorganizing what we already knew.
Is that not what the foundation models are? A new reorganization of existing knowledge?
Often people ignore scale. N=1 is OK, therefore, N=1billion is OK. Same flawed argument as: "It's OK for one police officer to watch one street corner for the purpose of observing crime; therefore it's equally OK to have cameras recording every street corner in the city 24/7, for all purposes. Same thing!"
So then the problem is that Anthropic seems hypocritical when they knowingly insert themselves into this chain, and then complain about people down-chain from them.
To remedy the negative impressions (if they even care to do so) they should do 1 of 2 things:
1) stop complaining about it
2) stop distilling other people's work
They insert themselves into this chain for profit and complain about it. I really think that adds a thick layer to the hypocrisy that people, or at least me, feel is especially distasteful.
No LLM products would exist without the avalanche of largely non-consensual use of IP to create them, full stop. Any of these companies doing this and then turning around and complaining when their IP is "breached" are going to met with a chorus of tiny violins.
It's hypocritical and we shouldn't listen to their bullshit but I wouldn't expect anything else from them.
This generation of frontier models is "good enough." At some point, the Chinese labs will get to where the Americans are right now, they'll race to the bottom, and we will actually see what proliferation of AI looks like.
If Anthropic wants to be a trillion dollar company, they need to make revenue like Google or Apple do. Both of them have near monopolies, Anthropic is only getting further and further away as time goes on.
Of course they're gonna complain until they either figure out a better plan or accept a new valuation which is high but not spectacular.
It is an ancient practice, that when a human creates something, other humans will observe it and learn from it. Every group of humans living together has practiced this in some form for tens of thousands of years if not longer. Even animals do it. It's a natural assumption when making any form of art.
It is not a natural assumption that someone will digitize the artwork and use it to adjust a couple thousand matrix coefficients in a complex computer program. To most people that seems like copying with extra steps. The brain may in some ways resemble a computer, but what sets it apart is that we have always lived with brains. Everything a human does has already anticipated the presence of other brains, while etched circuits on ultrapure silicon crystals are something new.
Don't forget the mothers of all those original authors, as well as everyone who labored to build and sustain the societies which produced writers.
To a degree. The human produced knowledge is the product of all humanity (no human is an island).
A comparable idea could be that an encyclopedia or maths book is only distilling the things that other people did, and how dare they sell them. But the "only" is doing quite a bit of work. LLMs do not just spawn into existence. There is a body of work that they feed on, and then there is also very attributable work they do around and on top of that. All labs are struggling around the first order question: Is it okay to use prior work like this? The second order issue is still entirely reasonable to separately have and enforce rules about.
And the communal work of humanity is orders of magnitude more work than what anthropic pays for their scraping of content. I got no check from them for my contributions
Indeed, if it isn't a crime to train on humanity's data, it isn't a crime to train on capitalism arranged frontier LLM provider models. Is that bad for shareholders and capitalism? Meh, sounds like a suboptimal socioeconomic systems issue. Burn up all the capital the unsophisticated are willing to provide. “We are selling to willing buyers at the current fair market price.”
With my apologies to Brewster Kahle, "Universal Access to All Knowledge."
I use crime in the broad sense of "You shouldn't be allowed to do that" in this context. If you have a better word to capture that thought, let me know, I'll make the edit ("frowned upon" perhaps?). I don't have strong feelings other than "hah AI companies aren't going to be able to create a moat to capture the value they want to capture because we can collectively keep pulling it out of their models in perpetuity through ever improving model distillation methodologies". This is no different than Uber and DoorDash using VC dollars to subsidize services until they try to turn the knob to profitability once they've captured the market, except in this case, there are mechanisms to exfiltrate the model value into open models that can be distributed at very small marginal cost. They can never gate the golden goose money printer, they can only complain it isn't fair they aren't able to.
I have far less of a problem with the Chinese models if they even do this because they are making their models free, whereas the large Western LLM providers throw out a few bits but not their main work product. So not only are they hypocritical, I'm pretty sure that if the Chinese models were not released into the wild they would be making less noise.
Oh yeah, totally agree, I'd even rather pay those building the Chinese models if I didn't think I'd get thrown into a US gulag for felony contempt of business model.
It is kind of ironic that they scraped the web for publicly available data and used it freely to train their models and now their freely available models are being used to train other models.
I'm overall pro-Anthropic and pro-banning open-weights AI, but I agree with the parent commenter; distilling Claude models is not that different from pretraining on web data. It's all basically the same sort of thing.
I think a good litmus test here would be if Anthropic were to not care about distilling their models when the distillers keep the resulting models closed-source and sell tokens via an API. If they cared only about security concerns and not about people profiting off of their work, then they should be publicly fine with this and only protest against it going into open-weights models.
But that’s not more philosophical. It’s a perfect parallel! Enormous amounts of work, vacuumed up and resold. What’s the difference? If it’s ok to vacuum up all the knowledge in the world, then that includes knowledge of how to use all that to power an LLM.
> What Anthropic is doing requires way more resources than what the Chinese labs are doing.
Oh that’s very sad.
Meanwhile Anthropic made a product from the work effort of millions of people without compensating them, sell that product on tap and unless I am mistaken do not even have their competitors’ cover of having released any sort of meaningful open weights model.
They have taken from culture (including very specifically their most direct customers’ specific culture — our culture), turned it into a machine to make themselves rich, appear likely to predicate their valuation on permanently removing people from the workforce, then want to dump themselves onto pensions funds and ordinary savers to carry the bag.
It is, I agree, philosophical, because karma is a philosophy as well as a bitch.
I'd split it at 99.98% original, 0.015% Anthropic, 0.005% Chinese, and that's being exceedingly generous to the AI companies, there should be several more 9s and 0s in there.
Deepmind was indirectly distilling Claude 3, XAI was doing this to other models (with Musk shrugging it off like something unremarkable, which it is), it has nothing to do with nebulous stereotypes like East, West, China this, America that. It's mostly Amodei and Altman screeching over this fact.
Distillation doesn't "grab 95% of lab's work", that's ridiculous. At best it's tiny icing on top of the cake that's already there. It's not even necessarily done on a better model (e.g. GLM 4.7 distilled Gemini 2.5, a weaker model), I'm pretty sure A\ and OAI could do (or even do) the same with greater efficiency since they have access to logits, weights, and internal state of open models.
>An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
How is this philosophical? They should release the unsupervised pretrains, at the very least.
Yeah, you need a lot of resources in order to waste a lot of resources. Look at codex and Claude code - these trillion dollar companies 'with top talent' cannot build what open code and pi/oh-my-pi have built in the open for free? Both codex and Claude code, are slow, buggy pieces of shit (and I say that as someone who still heavily uses both for work, moved to open code and pi for personal stuff). The reality is these companies mostly focus on marketing and market capture through, non-competitive means - their services and software are unreliable, buggy trash.
> An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
Actually, it's an interesting argument to make. How many labour-hours went into creating the training data Anthropic has collected? Probably multiple billions of hours. How many labour-hours did it take them to setup the datacenters, scrapers, and training algorithms? A few thousands hours?
> What Anthropic is doing requires way more resources than what the Chinese labs are doing. So their complaint is that they do 95% of the work and the Chinese labs do the last 5% and call it their own.
Anthropic: The Chinese have broken the implicit ethical rule of honorable thievery. We did most of the heavy lifting in this global heist, and here they are taking our righteous loot. This is not right.
Because no one outside the AI scientists understand what distilling means. They probably all think about Mash and a vodka still, and a completely unrelated association.
cool, do you think these media representations are for you, or 99% of the people who would love China to be sanctioned because they're foreigners?
"That’s called competition. You’re allowed to test somebody else’s products all you want." - Jensen Huang <a href="https://x.com/wallstengine/status/2104604118937735553" rel="nofollow">https://x.com/wallstengine/status/2104604118937735553
Because cost of original training >> cost of distilling. It's the same thing that happens with Chinese knockoffs of physical products - it takes a lot of money and R&D time to design a new product, but it's basically free to buy the product, reverse engineer it, and resell it. All the data they originally trained on was available for free on the internet. If the original work was so valuable, it shouldn't be up on the internet for free in the first place imo.
It's different because you have to pay Anthropic and follow their TOS to get access to their model. If it was actually on the internet for free, anyone could do whatever they wanted with it.
Right, but the humans willingly released all those creations for free. I think that my issues with the "AI companies stole human creations" stance is that the information was freely available to everyone, and they put a lot of money and effort into transforming it into something useful.
>Aren’t Anthropic’s models not just distilling down other people’s work?
Can you elaborate on that? I mean my direct answer would be no, of course not. But why do you think frontier models are distilled? I think maybe there is an equivocation over the word “distillation.”
Frontier labs train on their own pretraining data, human feedback, synthetic data, and research. A distilled model is specifically optimized to reproduce another model's behavior.
Meanwhile R1-Distill-Qwen-32B was distilled from DeepSeek-R1.
If you want to say a frontier model is "distilled" from the world's data and R1-Distill-Qwen-32B is distilled from DeepSeek-R1 then you are equivocating two very different things.
The root comment was asking how it is a problem. If one considers it wrong, then it’s a problem regardless of whether Anthropic is complaining or not. Anthropic’s complaining or non-complaining should have no bearing on whether it’s considered a problem or not.
The difference is in the solution that would be proposed. I'm sure Anthropic wants to create some kind of IP protection regime for their model so it can't be distilled. I want their model to be public domain, since they trained on material that was not theirs to begin with.
The conversation here is mostly moral and ethical but the problem here seems to be financial.
Anthropic and OpenAi are spending a $$$$ to "distill" human output into an AI model, then others are spending $$ to distill their AI model into a near-equivalent model.
This is the same reason IP rights exist. On the surface, something like a patent feels ludicrious and even feels morally wrong. Some guy wrote down the recipe for arranging atoms or bits in a particular way, and now I can't!? However, it's designed to solve the same problem, figuring out and describing the process is much more costly than replicating it.
Anthropic and OpenAI are very happy to ignore the IP rights of others, so I'm not sure how they can ask for any kind of IP protection themselves. Live by the sword, die by the sword.
Model weights wouldn't be covered in a patent. You could patent a method of creating weights in a model, but you couldn't patent the weights themselves.
I wish people here could at least bother to inform themselves about the IP rights they are so quick to insist are abhorrent, when they seem to not even have a first clue as to what they actually cover.
AI training, if viewed through the capitalist mindset, is plain theft. Anthropic can't morally defend copying someone else's IP, but denouncing others copying Anthropic's stolen IP.
That doesn't mean they won't try, and that also doesn't mean they won't succeed.
cmiles8 · · focus · HN ↗
Feels like Anthropic crying do as I say not as I do.
[deleted] · · focus · HN ↗
[deleted]
jorblumesea · · focus · HN ↗
it's not complex. there's hundreds of billions of investor dollars counting on vendor lock in and walled gardens
2OEH8eoCRo0 · · focus · HN ↗
dpweb · · focus · HN ↗
If I'm a business and I need something done today, and bc Anthropic has the best model, there's a 99.9 chance it will be completed successfully for $1000. And using Deepseek there's a 70% chance it will, for $10 - you or me will go for the $10. Big businesses don't. Bc 1000 per task is nothing to them.
HWR_14 · · focus · HN ↗
The real issue is that Deepseek has a 99.7% chance. So I can run it 10 times until it works and still pay 1/10 the money.
bushbaba · · focus · HN ↗
thadt · · focus · HN ↗
Big businesses might pay $1000 vs $60 for certain tasks, but that won't work out well at scale.
cmiles8 · · focus · HN ↗
Most folks I know can choose from any of the big labs or open weight models and they get billed internally for tokens against their budget. There’s little incentive to no switch to the lower cost closers.
This setup is a nightmare scenario for the big labs trying to execute the traditional enterprise sales plays. Those only work if your product is sticky and AI models are one of the least sticky things in the history of tech.
andrew_lettuce · · focus · HN ↗
rootusrootus · · focus · HN ↗
Also, is it really 99.9% vs 70%, or 99.9% vs 99%?
teaearlgraycold · · focus · HN ↗
jorblumesea · · focus · HN ↗
why should sammie or darigold have the keys to the kingdom?
teaearlgraycold · · focus · HN ↗
nater5000 · · focus · HN ↗
[dead]
jedberg · · focus · HN ↗
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
mrwh · · focus · HN ↗
cyanydeez · · focus · HN ↗
mrwh · · focus · HN ↗
off_with_their_ · · focus · HN ↗
[dead]
watwut · · focus · HN ↗
ipsod · · focus · HN ↗
freejazz · · focus · HN ↗
that's an interesting way to describe reddit posts
Paradigma11 · · focus · HN ↗
ipsod · · focus · HN ↗
faangguyindia · · focus · HN ↗
jedberg · · focus · HN ↗
isolay · · focus · HN ↗
arctic-true · · focus · HN ↗
jedberg · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
AlexandrB · · focus · HN ↗
cmiles8 · · focus · HN ↗
Anthropic’s anger here seems mostly rooted in their annoyance that this exposes they don’t really have core IP that’s not just easily replicated. And that’s clearly a problem for a deeply unprofitable company trying to convince people they’re worth $2 trillion.
JackFr · · focus · HN ↗
The original authors of all the text, creators of the media and developers of the software did far more work than Anthropic.
jdonaldson · · focus · HN ↗
rubicon33 · · focus · HN ↗
mitthrowaway2 · · focus · HN ↗
tsunamifury · · focus · HN ↗
Did Anthropic put work in? Yes. Did they derive their value from Humanity being open with knowledge then try to sell it back? Also yes.
Did they even steal the tech? Also yes.
_DeadFred_ · · focus · HN ↗
nonethewiser · · focus · HN ↗
ihsw · · focus · HN ↗
[dead]
layer8 · · focus · HN ↗
This isn’t meant as a moral argument, just musing about the relative cost comparison.
Ar-Curunir · · focus · HN ↗
How is this even a question.
bmacho · · focus · HN ↗
mekael · · focus · HN ↗
The knowledge that created the Haber-Bosch process [1] helps to sustain the majority of the world's populous, add another five hundred trillion dollars for that just to start with.
The creation of the printing press and all written information that allowed it to be built provided dissemination of knowledge beyond the ultra wealthy and is worth a non-finite amount of money.
LLM's are cool math, but they are less than a rounding error in comparison to even the tiniest sliver of human knowledge and technological output.
[0] <a href="https://pubmed.ncbi.nlm.nih.gov/35143880/" rel="nofollow">https://pubmed.ncbi.nlm.nih.gov/35143880/ [1] <a href="https://cen.acs.org/food/agriculture/The-industrialization-Haber-Bosch-process/101/i26" rel="nofollow">https://cen.acs.org/food/agriculture/The-industrialization-H...
DirkH · · focus · HN ↗
Ar-Curunir · · focus · HN ↗
And also, humans have been doing a lot more work than just mathematics...
phamilton · · focus · HN ↗
A training set of 15 trillion tokens is 10 trillion words.
A penny a word is cheaper than the cheapest beginner freelance writer.
That makes a training set of 10 trillion words cost $100B.
Lots of assumptions there for sure, but we're certainly in the ballpark you are describing.
ToValueFunfetti · · focus · HN ↗
louiskottmann · · focus · HN ↗
The totality of the content on internet is worth several orders of magnitude more.
layer8 · · focus · HN ↗
tpm · · focus · HN ↗
allturtles · · focus · HN ↗
jonhohle · · focus · HN ↗
layer8 · · focus · HN ↗
Image/video models are, but those weren’t the topic.
Enginerrrd · · focus · HN ↗
rsingel · · focus · HN ↗
$500B for all kinds including TV and online
$300B for newsrooms including all staff
$140B for newsroom reporters only
So yeah, I think the price of the information ingested is way higher than training costs
Ohentis · · focus · HN ↗
wonnage · · focus · HN ↗
“bro like, what if we could price the sum total of human knowledge? That wouldn’t be that much, right?”
MathiasPius · · focus · HN ↗
And the monetary cost doesn't even register when weighed against the blood, sweat and tears that went into capturing the authentic experiences of real human beings, whose honest expressions are now at least in some cases getting hoovered up, ingested, and then destroyed for all eternity, for fear that this specific work is the rounding error that might give an equally immoral competitor the edge in the bicycle-riding flamingo race that is currently consuming an absurd amount of the world's creativity and attention.
zelphirkalt · · focus · HN ↗
jedberg · · focus · HN ↗
Is that not what the foundation models are? A new reorganization of existing knowledge?
xdavidliu · · focus · HN ↗
iAMkenough · · focus · HN ↗
kbelder · · focus · HN ↗
wonnage · · focus · HN ↗
ryandrake · · focus · HN ↗
gretch · · focus · HN ↗
So then the problem is that Anthropic seems hypocritical when they knowingly insert themselves into this chain, and then complain about people down-chain from them.
To remedy the negative impressions (if they even care to do so) they should do 1 of 2 things: 1) stop complaining about it 2) stop distilling other people's work
ToucanLoucan · · focus · HN ↗
No LLM products would exist without the avalanche of largely non-consensual use of IP to create them, full stop. Any of these companies doing this and then turning around and complaining when their IP is "breached" are going to met with a chorus of tiny violins.
SR2Z · · focus · HN ↗
This generation of frontier models is "good enough." At some point, the Chinese labs will get to where the Americans are right now, they'll race to the bottom, and we will actually see what proliferation of AI looks like.
If Anthropic wants to be a trillion dollar company, they need to make revenue like Google or Apple do. Both of them have near monopolies, Anthropic is only getting further and further away as time goes on.
Of course they're gonna complain until they either figure out a better plan or accept a new valuation which is high but not spectacular.
KolibriFly · · focus · HN ↗
[dead]
scythe · · focus · HN ↗
It is not a natural assumption that someone will digitize the artwork and use it to adjust a couple thousand matrix coefficients in a complex computer program. To most people that seems like copying with extra steps. The brain may in some ways resemble a computer, but what sets it apart is that we have always lived with brains. Everything a human does has already anticipated the presence of other brains, while etched circuits on ultrapure silicon crystals are something new.
agumonkey · · focus · HN ↗
marshray · · focus · HN ↗
eithed · · focus · HN ↗
dr_dshiv · · focus · HN ↗
briansm · · focus · HN ↗
mc32 · · focus · HN ↗
It’s like FTL. Until someone realizes it, it’s just talk.
jstummbillig · · focus · HN ↗
A comparable idea could be that an encyclopedia or maths book is only distilling the things that other people did, and how dare they sell them. But the "only" is doing quite a bit of work. LLMs do not just spawn into existence. There is a body of work that they feed on, and then there is also very attributable work they do around and on top of that. All labs are struggling around the first order question: Is it okay to use prior work like this? The second order issue is still entirely reasonable to separately have and enforce rules about.
OJFord · · focus · HN ↗
baxtr · · focus · HN ↗
jacquesm · · focus · HN ↗
bushbaba · · focus · HN ↗
toomuchtodo · · focus · HN ↗
With my apologies to Brewster Kahle, "Universal Access to All Knowledge."
<a href="https://www.youtube.com/watch?v=RV_ALlJGU_c" rel="nofollow">https://www.youtube.com/watch?v=RV_ALlJGU_c
abcthingx · · focus · HN ↗
toomuchtodo · · focus · HN ↗
"The Spice must flow."
voiceofchoice · · focus · HN ↗
jacquesm · · focus · HN ↗
toomuchtodo · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
jklinger410 · · focus · HN ↗
meowface · · focus · HN ↗
I think a good litmus test here would be if Anthropic were to not care about distilling their models when the distillers keep the resulting models closed-source and sell tokens via an API. If they cared only about security concerns and not about people profiting off of their work, then they should be publicly fine with this and only protest against it going into open-weights models.
OtherShrezzing · · focus · HN ↗
I see absolutely no distinction between the two, aside from minor technical approaches to gathering the content.
tene80i · · focus · HN ↗
koickong · · focus · HN ↗
[dead]
freejazz · · focus · HN ↗
And writing a book requires many more resources than what anthropic does
darkmighty · · focus · HN ↗
It sounds exactly the same, not more philosophical to me, except one is more inconvenient.
tsunamifury · · focus · HN ↗
dofm · · focus · HN ↗
Oh that’s very sad.
Meanwhile Anthropic made a product from the work effort of millions of people without compensating them, sell that product on tap and unless I am mistaken do not even have their competitors’ cover of having released any sort of meaningful open weights model.
They have taken from culture (including very specifically their most direct customers’ specific culture — our culture), turned it into a machine to make themselves rich, appear likely to predicate their valuation on permanently removing people from the workforce, then want to dump themselves onto pensions funds and ordinary savers to carry the bag.
It is, I agree, philosophical, because karma is a philosophy as well as a bitch.
dancemethis · · focus · HN ↗
AlexandrB · · focus · HN ↗
BigTTYGothGF · · focus · HN ↗
jasondigitized · · focus · HN ↗
orbital-decay · · focus · HN ↗
Andrex · · focus · HN ↗
Henchman21 · · focus · HN ↗
orbital-decay · · focus · HN ↗
>An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
How is this philosophical? They should release the unsupervised pretrains, at the very least.
aleqs · · focus · HN ↗
thrance · · focus · HN ↗
Actually, it's an interesting argument to make. How many labour-hours went into creating the training data Anthropic has collected? Probably multiple billions of hours. How many labour-hours did it take them to setup the datacenters, scrapers, and training algorithms? A few thousands hours?
KolibriFly · · focus · HN ↗
[dead]
yubblegum · · focus · HN ↗
Anthropic: The Chinese have broken the implicit ethical rule of honorable thievery. We did most of the heavy lifting in this global heist, and here they are taking our righteous loot. This is not right.
dominotw · · focus · HN ↗
[dead]
cmiles8 · · focus · HN ↗
cyanydeez · · focus · HN ↗
The word itself is the pivot, not anything else.
hn_throwaway_99 · · focus · HN ↗
I don't know anyone with even a passing understanding of how LLM training works that thinks that is the appropriate analogy.
cyanydeez · · focus · HN ↗
pdntspa · · focus · HN ↗
sergiotapia · · focus · HN ↗
jrflo · · focus · HN ↗
wonnage · · focus · HN ↗
jrflo · · focus · HN ↗
AlexandrB · · focus · HN ↗
ASalazarMX · · focus · HN ↗
You could say Anthropic distilled human knowledge and art.
jrflo · · focus · HN ↗
ASalazarMX · · focus · HN ↗
How can one answer this statement in good faith? AI companies literally violated IP by massively pirating works instead of legally licensing them.
nonethewiser · · focus · HN ↗
Can you elaborate on that? I mean my direct answer would be no, of course not. But why do you think frontier models are distilled? I think maybe there is an equivocation over the word “distillation.”
Frontier labs train on their own pretraining data, human feedback, synthetic data, and research. A distilled model is specifically optimized to reproduce another model's behavior.
Meanwhile R1-Distill-Qwen-32B was distilled from DeepSeek-R1.
If you want to say a frontier model is "distilled" from the world's data and R1-Distill-Qwen-32B is distilled from DeepSeek-R1 then you are equivocating two very different things.
nuancebydefault · · focus · HN ↗
bionhoward · · focus · HN ↗
layer8 · · focus · HN ↗
OneLessThing · · focus · HN ↗
layer8 · · focus · HN ↗
AlexandrB · · focus · HN ↗
seizethecheese · · focus · HN ↗
Anthropic and OpenAi are spending a $$$$ to "distill" human output into an AI model, then others are spending $$ to distill their AI model into a near-equivalent model.
This is the same reason IP rights exist. On the surface, something like a patent feels ludicrious and even feels morally wrong. Some guy wrote down the recipe for arranging atoms or bits in a particular way, and now I can't!? However, it's designed to solve the same problem, figuring out and describing the process is much more costly than replicating it.
failbuffer · · focus · HN ↗
AlexandrB · · focus · HN ↗
freejazz · · focus · HN ↗
I wish people here could at least bother to inform themselves about the IP rights they are so quick to insist are abhorrent, when they seem to not even have a first clue as to what they actually cover.
ASalazarMX · · focus · HN ↗
That doesn't mean they won't try, and that also doesn't mean they won't succeed.