7-8 figures annual spend will buy a hell of a lot of capable local inference hardware you can own, though it won't be at the absurd token/s rate, you'll be able to run almost anything on it... And it'll still have a good residual resale value after 4 years the way things are going now.
Feel like you could spend 6 figures building out a team and the rest renting compute for a whole year, and get the team to create a local inference solution with that kind of budget…
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
> There is no additional fee for using prompt caching. Input tokens, whether served from the cache or processed fresh, are billed at the standard input token rate for the respective model.
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
Yea i had some pretty meh results using gpt-oss-120b it in my evals where it should have benefited speed alot but it really under performed what i was expecting.
I kept having experiences with gpt-oss-120b on Cerebras where it would get stuck in a thinking block and then start endlessly saying things like "Running the command now." or "Making the changes now." and then simply repeating similar sentences like that forever instead of actually making the tool call. It made tool calls other times, so it wasn't an issue with tool calls being impossible, but it just wasn't doing a good job of using them for real instead of simply saying it would. So this was not an issue of it starting a tool call and then putting invalid syntax inside of it, it just would not make the tool call it was supposed to whatsoever. There's no automatic way to retry that.
Lighting my codebase on fire at the speed of light. Like microwaving the spaghetti.
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
Human code review? What is this, 2025? The modality today is write with one LLM, review by a different one, (important: two different model families will catch errors one series won't) then deploy right to production.
And if you tell the reviewer the author is a competitors model it becomes extra snarky and vigilant. Then give the review results to the author and tell it it's from the competition and it will also become slightly outraged.
I would be very curious to see how you explain it to your customers.
Is it going to sound similar to this?
> You see, our well-meaning AI-generated code has caused all your data to be permanently deleted. In case you are confused as to who to blame, we would like to clarify that we did not write, nor review the code. So we cannot possibly bear any responsibility for its mistakes. The responsibility lies with the LLMs, not us. We have already fired the LLM which did the coding and the review. And we are already using their main competitors. Hopefully that settles your concern with the quality of our service and we are looking to have you on board of our next products.
This seems to work well enough for people who deploy cloud instances without redundancy to us-east-1 then blame AWS when there's an outage. I say this somewhat unironically because if one pushes the "move fast and break things" slider all the way to the right then they're necessarily assuming that type of risk. Of course some people will try to have their cake and eat it too [1] as regards velocity and quality but that's a separate discussion.
[1] I've never understood this idiom because if one isn't in possession of their cake before eating it then they're eating stolen cake which is a decidedly anti-social activity and orthogonal to the point of the saying. It should be "eat their cake and keep it too" or something.
Yeah, idk. I work on serious things. Thats not how I do things. Not everything is webdev hobby projects. There's basically no instance where offloading your code review to an llm is acceptable behavior, except maybe for a one off tool you need personally.
No you dont need to code at 600 w/s BUT at those speeds, you can start doing things like asking multiple different agents the same question and picking the best solution each time without noticing the lag.
I prefer a fast model too, but you cannot get more done just because its faster. You just get to the human parts a bit faster. Code review, revision ect.
absolutely not ! it's not an "ohhh ill get 100x more coding done" it's definitely a preference/work style thing. I find it's hard to get in "the flow" when managing multiple agents, and if I just do one at a time with today's frontier models, I find myself waiting 10 minutes twiddling my thumbs while they do something all the time. Then needing to catch myself and turn back over to it when its done, all these micro switches between tasks is really difficult for me. If I had an instant agent I probably would be somewhat faster, but not zomg1000xunicornrockstar nonsense. I would be way happier though
Inco sucks. I tried their GLM 5.3 Flash and it was quantized to the point of hallucinating Chinese in the middle of English only agentic sessions. Never happened with any other provider.
bearjaws · · focus · HN ↗
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
ford · · focus · HN ↗
[0] <a href="https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise" rel="nofollow">https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
sharktheone · · focus · HN ↗
But it can apparently also run 5.6 Sol
eli · · focus · HN ↗
walrus01 · · focus · HN ↗
podocarp · · focus · HN ↗
jazzyjackson · · focus · HN ↗
scosman · · focus · HN ↗
RussianCow · · focus · HN ↗
scosman · · focus · HN ↗
LoganDark · · focus · HN ↗
eli · · focus · HN ↗
fakwandi_priv · · focus · HN ↗
So what he’s saying is correct, there is no separate cache pricing, which by normal standards should be 10% of the cost, which can become exceedingly expensive for anything other than single turn. The way they are stating this is of course strange..
shard972 · · focus · HN ↗
bearjaws · · focus · HN ↗
I have it retry on failure, but you should do that with any LLM really.
LoganDark · · focus · HN ↗
conception · · focus · HN ↗
Such a tiny model at that t/s is less impressive than it would have been four months ago.
physicallyIllfr · · focus · HN ↗
I genuinly only see these speeds being useful for customer service/transactional workflows. Of which much smaller models can do the job (but those dont make tons of money for companies like Cerebras that need to pay off massive amounts of debt).
Nobody needs to code at 600 words per second. Using a 100tps model for an hour or so will leave you with 4-8hrs of code review and revision work.
sbierwagen · · focus · HN ↗
podocarp · · focus · HN ↗
pbasista · · focus · HN ↗
And when things go south, you blame AI?
I would be very curious to see how you explain it to your customers.
Is it going to sound similar to this?
> You see, our well-meaning AI-generated code has caused all your data to be permanently deleted. In case you are confused as to who to blame, we would like to clarify that we did not write, nor review the code. So we cannot possibly bear any responsibility for its mistakes. The responsibility lies with the LLMs, not us. We have already fired the LLM which did the coding and the review. And we are already using their main competitors. Hopefully that settles your concern with the quality of our service and we are looking to have you on board of our next products.
alexjplant · · focus · HN ↗
This seems to work well enough for people who deploy cloud instances without redundancy to us-east-1 then blame AWS when there's an outage. I say this somewhat unironically because if one pushes the "move fast and break things" slider all the way to the right then they're necessarily assuming that type of risk. Of course some people will try to have their cake and eat it too [1] as regards velocity and quality but that's a separate discussion.
[1] I've never understood this idiom because if one isn't in possession of their cake before eating it then they're eating stolen cake which is a decidedly anti-social activity and orthogonal to the point of the saying. It should be "eat their cake and keep it too" or something.
physicallyIllfr · · focus · HN ↗
calgoo · · focus · HN ↗
physicallyIllfr · · focus · HN ↗
RugnirViking · · focus · HN ↗
I do. I used to use haiku for the speed. Now its just as slow as the rest. Speed is my #1 ranking of how good a model is
sitkack · · focus · HN ↗
What is your workflow?
physicallyIllfr · · focus · HN ↗
RugnirViking · · focus · HN ↗
lostmsu · · focus · HN ↗
verdverm · · focus · HN ↗
<a href="https://kamilstanuch.github.io/LLM-token-generation-simulator/" rel="nofollow">https://kamilstanuch.github.io/LLM-token-generation-simulato...
kylehotchkiss · · focus · HN ↗