Why is it a problem that the Chinese labs are just distilling down Anthropic’s models? Aren’t Anthropic’s models not just distilling down other people’s work?
Feels like Anthropic crying do as I say not as I do.
What Anthropic is doing requires way more resources than what the Chinese labs are doing. So their complaint is that they do 95% of the work and the Chinese labs do the last 5% and call it their own.
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
SOTA models cost hundreds of millions to train. Did creating the contents of the text corpus they were trained on really cost an equivalent of 20x as much (~10 billions)? I honestly don’t know, but I could imagine it having been significantly less.
This isn’t meant as a moral argument, just musing about the relative cost comparison.
Pre vaccination smallpox killed hundreds of millions of people just in the twentieth century [0], the knowledge that allowed for the creation of just that vaccine is worth hundreds trillions of dollars in humans lives, let alone all of `the knowledge and experiences those people were involved in.
The knowledge that created the Haber-Bosch process [1] helps to sustain the majority of the world's populous, add another five hundred trillion dollars for that just to start with.
The creation of the printing press and all written information that allowed it to be built provided dissemination of knowledge beyond the ultra wealthy and is worth a non-finite amount of money.
LLM's are cool math, but they are less than a rounding error in comparison to even the tiniest sliver of human knowledge and technological output.
Shush, this is Hackernews. We all want to have our egos stoked that our industry is the most important in human history and that tech will transform and save us all. Go away with your historical analysis /s
do you also count eg published results of very expensive physics experiments? because once the costs of things like these are taken into account, we are way over 10 billions.
If you look at movies alone that would easily surpass 10s of billions. The cost of most books is probably more nebulous, but books, research, and more all have time and money spent to create them. I would guess the corpus of all media from the 20th century on would be minimally in the hundreds of billions of dollars.
I would argue that producing the complete written corpus on which they at least intend to train (even if some is still out of reach) cost literally everything to produce.
And the monetary cost doesn't even register when weighed against the blood, sweat and tears that went into capturing the authentic experiences of real human beings, whose honest expressions are now at least in some cases getting hoovered up, ingested, and then destroyed for all eternity, for fear that this specific work is the rounding error that might give an equally immoral competitor the edge in the bicycle-riding flamingo race that is currently consuming an absurd amount of the world's creativity and attention.
It probably cost vastly more than training the LLM. You need to consider the time people spent and perhaps weigh it in their hourly wage. Accumulate that across all the training data and it will be a mindboggling amount, compared to training the model.
cmiles8 · · focus · HN ↗
Feels like Anthropic crying do as I say not as I do.
jedberg · · focus · HN ↗
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
JackFr · · focus · HN ↗
The original authors of all the text, creators of the media and developers of the software did far more work than Anthropic.
layer8 · · focus · HN ↗
This isn’t meant as a moral argument, just musing about the relative cost comparison.
Ar-Curunir · · focus · HN ↗
How is this even a question.
bmacho · · focus · HN ↗
mekael · · focus · HN ↗
The knowledge that created the Haber-Bosch process [1] helps to sustain the majority of the world's populous, add another five hundred trillion dollars for that just to start with.
The creation of the printing press and all written information that allowed it to be built provided dissemination of knowledge beyond the ultra wealthy and is worth a non-finite amount of money.
LLM's are cool math, but they are less than a rounding error in comparison to even the tiniest sliver of human knowledge and technological output.
[0] <a href="https://pubmed.ncbi.nlm.nih.gov/35143880/" rel="nofollow">https://pubmed.ncbi.nlm.nih.gov/35143880/ [1] <a href="https://cen.acs.org/food/agriculture/The-industrialization-Haber-Bosch-process/101/i26" rel="nofollow">https://cen.acs.org/food/agriculture/The-industrialization-H...
DirkH · · focus · HN ↗
Ar-Curunir · · focus · HN ↗
And also, humans have been doing a lot more work than just mathematics...
phamilton · · focus · HN ↗
A training set of 15 trillion tokens is 10 trillion words.
A penny a word is cheaper than the cheapest beginner freelance writer.
That makes a training set of 10 trillion words cost $100B.
Lots of assumptions there for sure, but we're certainly in the ballpark you are describing.
ToValueFunfetti · · focus · HN ↗
louiskottmann · · focus · HN ↗
The totality of the content on internet is worth several orders of magnitude more.
layer8 · · focus · HN ↗
tpm · · focus · HN ↗
allturtles · · focus · HN ↗
jonhohle · · focus · HN ↗
layer8 · · focus · HN ↗
Image/video models are, but those weren’t the topic.
Enginerrrd · · focus · HN ↗
rsingel · · focus · HN ↗
$500B for all kinds including TV and online
$300B for newsrooms including all staff
$140B for newsroom reporters only
So yeah, I think the price of the information ingested is way higher than training costs
Ohentis · · focus · HN ↗
wonnage · · focus · HN ↗
“bro like, what if we could price the sum total of human knowledge? That wouldn’t be that much, right?”
MathiasPius · · focus · HN ↗
And the monetary cost doesn't even register when weighed against the blood, sweat and tears that went into capturing the authentic experiences of real human beings, whose honest expressions are now at least in some cases getting hoovered up, ingested, and then destroyed for all eternity, for fear that this specific work is the rounding error that might give an equally immoral competitor the edge in the bicycle-riding flamingo race that is currently consuming an absurd amount of the world's creativity and attention.
zelphirkalt · · focus · HN ↗