This is the golden age of model training. Some days ago, I decided I wanted a local CPU only model that can perform exceptionally well for English to Bash translation (to avoid the googling for command syntax). I got a bunch of subagents to generate large amount of training data (140k+ samples), got the Qwen 3 0.6B base model, pointed Astra at it, and off to the races. It trained for 2 days (on and off) and I got a surprisingly good model for my task! The total active time I spent was a few hours. And it is still improving, what a time to be alive!
I didn't have a local GPU, so I asked it to go out and find hardware. It found a google TPU v6e which seemed reasonably priced. I gave it my google api key. I told it to use TPU only when training and bring it down afterwards. That's about it.
On the cloud side, nothing valuable existed, so the training couldn't ruin anything it didn't create. On the laptop side, I usually ask the agents to create named scripts for everything it needs to access, then those local script directory is green-lit with approve all. For cost, I kept giving it new budget in the 20-30 dollar increments.
I had to intervene a few times. For instance, as smart as the models are said to be (Astra), it would copy the full training run, train on the server, pull every checkpoint to the local machine, then run tests, update. So, the bandwidth bill was as high as training bill for the first 6 hours. It could have simply tested each checkpoint on the server, saved time and money, didn't occur to it until I said.
There’s a safer way to do this with nearly no added friction. Give it a read only API key. Then just ask it to write the API calls into a bash script and then read it and run it yourself. The agent can still inspect the live resources and diagnose and give you more commands to run. I do agree I wouldn’t give it create / write access.
I've done this sort of thing before but with Vast. Pre-deposited some money online, then let the LLM request and manage a training run on an allocation. Worked pretty well without risking bankruptcy.
I am thinking about opensourcing everything, although this is not my main domain or my main startup, so the overhead of huggingface etc seems a bit unnecessary
All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.
I’ve previously fine tuned (SFT, currently trying to distill) models using this <a href="https://huggingface.co/datasets/westenfelder/NL2SH-ALFA" rel="nofollow">https://huggingface.co/datasets/westenfelder/NL2SH-ALFA
That's a really impressive result. There are all kinds of small tasks like this I use an LLM for, but theoretically if you broke all the sub-use cases into local-only models, and had something lightweight that routed to the right model, you could have faster and cheaper workflows. E.g. something trained on the linux man pages for common commands, since it's usually quicker to ask an LLM for a specific command with flags than to consult the man pages.
The models he is using to generate training data are presumably commercial models. He is distilling their bash knowledge into a much smaller model he can run locally fast and cheap.
A very small, highly specialized model can use negligible resources (CPU, energy) to accomplish the same task.
For everyday work that happens frequently it's better to have a tiny specialized model instead of making billable API calls or turning your laptop into an 80W space heater for 20 seconds to run a general purpose model.
The large models can be used to generate synthetic training data. Tell them to make up 100,000 tasks paired with the resulting output as a 1-time cost. Then use that to train a small model.
Think of it as distillation, but focused on a specific task.
Given that they're just using it to avoid the googling for bash command syntax, I'm not sure they'll save in the end against the 140k training examples they generated.
I think OP's point remains, if you generate 140k pairs, your local model would need to run that many to offset having just used the generator (SOTA or not) model to begin with.
I wonder if another approach if latency is a concern is just to do a two shot pass with Jev (perhaps given small context you'd want one to match command, then one to match args of given command) would be an extremely fast, and cheap way to do it - rather than training your own.
Not quite. One, because you save on the initial query being sent multiple times. Two, because the reasoning will be very similar at the beginning; "user asked me to", "let's check what's in this project already", etc. you'll get similar actual output cost, but input and reasoning will be shorter.
If it's about the latency / flow disruption, spending a few hours once could easily be worth it if the result is actually good enough to skip googling/retries.
This example is not that niche. Lots of people use human to bash.
I'd probably use a small pre trained model if it was easy to use. I use a little script right now that calls a cheap model.
Actually.. $5 would probably will last me over a year so the only real benefit would be if I didn't have an internet connection.
You can buy a $10 subscription for a month to generate the training data, then cancel your subscription. The trained model is yours to use (and share with others) forever.
Yea specialized models could also be resold in a shareware style, like even if it cost 50$ to produce you'd just have to sell 10 copies of it to people for 5$.
Which 5$ is a pretty easy sell if its useful in any way, It's pretty easy to justify a purchase if its yours forever and doesn't use much CPU so is easy to run I mean people were spending 1000$+ on mac mini setups to run local llms or run remote agents.
It wouldn't cost $10 for their lifetime use if they used cheap models instead. Mistral Small 3 24B is probably substantially better than their Qwen 3 0.6B trained model and would give you about 500,000–2,000,000 English to Bash translations. If they're a heavy user, they might use $0.30 in total, ever.
They also used Astra for the coding, which they can't get on a $10 subscription. And then there's the actual training cost.
This is so cool - I'm aware of this in a vague way. Can you write a little tutorial or give some good links. I want this to be the next new things I do :)
I feel like we need a good index for these kinds of specialized models, especially if you plan to open them up. The downside is a new bash version means potentially new training.
I took it more to mean “do standard Unix command-line stuff” rather than purely emitting Bash, and while e.g. POSIX does receive updates, the basic stuff using the main tools should also work in the future!
The same Google that drowns you in low-quality AI responses, ads, and “organic” traffic stuffed with ads? I’d prefer the private, local, homegrown specialized English-to-bash translator…
If it’s one of thing that you want just for English to bash shell commands, I will create AST, it is deterministic, exceptionally fast, no tokens so no need to fine tune existing model, please let me know your thoughts.
You can go quite far using a human language to Bash grammar based setup but at some point the input prompts are harder to translate. The OP has existing projects that work with AST quite deeply so I assume they know about that already.
I am building a natural language to CSV/Excel commands for a "wrangler" type desktop app. Same issues. The MVP is being built with parsers of sorts, entirely code generated. Then I want to fine-tune a tiny model at some point.
Golden age before the age that ends humanity. Not talking about any "rogue AI", just the known statistical models of what is coming due to climate change.
I'm literally working on context/harness engineering right now (a set of opencode plugins)
Aside on the aside, I welcome this new era of really personal software. Not Ai's being sycophants, rather being able to easily and quickly change, adapt, or extend software I am not familiar with.
Seriously, I'm using a Qwen 3.8 27B on the homelab, distilled from supposed Fable traces. Regardless, the difference is notable, less thinking, better output. Distilled / heavy quant is better than the original (imv)
Here's a similar project for those who want to replicate: <a href="https://github.com/ThorOdinson246/whatisit-nl2sh" rel="nofollow">https://github.com/ThorOdinson246/whatisit-nl2sh
I have been trying a mix of fine-tuning and I am amazed that most people do not see this coming.
A tiny, smaller than 1b parameter model, fine-tuned, can kick ass for constrained work. I do not have a lot of budget, I fine-tune only on a 16GB M4 Mac Mini. But that also tells me the potential is wild. Progress has been slow since I moonlight on this.
I have been trying to build a set of models + agents for full-stack development, where each model does only a small piece, like take user prompt and break into backend/frontend tasks. Then a Rust+Diesel model, a Rust+Auxum model, a Solid+Router model and so on. I know this is wild but this is just theory - can 5 or 6 Qwen 3.5 0.8b models do full-stack web development? My hunch says they can, better than what most people expect. Heck, with a good harness, it might beat all the cheaper models for the specific task, like Haiku or Luna.
I think a sub-set of people see this coming, I also think it isn’t just fine tuning open weight LLM models. A few people I know who are thinking along the same lines with architectures like BERT etc.
That being said it’s much easier at the moment to continue to use the frontier providers for most general tasks, that is the argument I’ve heard.
For creating these types of fine tuned local models, on constrained hardware for inference, I do think this is the way to go for specific tasks too!
The issue I find with this is that the frontier models still outperform the small finetuned model on its specific task. So much so that the ROI on doing fine tunes is likely negative. I would love to hear some specific example where it did provide value though, if any has any. That would be helpful to start being able to find similar cases.
There are lots of distilled models on huggingface that are much smaller than, say, Opus. They are distilled from Opus or Fable and show clear improvements. I do not have the budget to fine-tune a 30b or more parameter model but from my tiny model experiments, the results are quite clear. Again, I have only a couple small tests.
Have you actually fine-tuned yourself? Email categorization comes to mind and there are tons of non-LLM approaches even that will give fantastic results. How did spam filters work before LLM?
I think LLMs just made us think that is the only way. It is not.
Maybe we're getting into a world where our general models can make their own separate fine-tuned models as utilities just like they would a bash/python script. If making a fine-tune is cheap and fast then it can also be throw-away and constantly improved.
I have a single command that fires up llama.cpp on cpu only using gemma4 e2b, answers a single question from the command line and exits. This takes about 3 seconds to load from an SSD, and is smart enough to solve exactly these "remind me of the syntax" scenarios if you dont wanna switch to a browser.
I have no experience doing this, and I dont mean this to be snarky: what is the power (and thermal) usage of running your CPU only LLM?
When I do something "heavy", my 9800x3D will kick on the fans and start making lot of heat. This is fine when I'm intentionally doing say a transcode, but if I'm just querying syntax it could get pretty annoying. Do you "feel" it? I know that will be pretty machine dependent.
I vaguely recall a project from a while back that did something similar without LLMs.
I’m really pushing my recall, but I want to say it was written in Ruby and stored pre-configured commands that it just did traditional search over.
I vaguely recall it working okay because 99.99% of the questions people asked were the same (“tar command to gzip a directory and strip the prefix” is something I google like once a month).
GodelNumbering · · focus · HN ↗
shriphani · · focus · HN ↗
GodelNumbering · · focus · HN ↗
shriphani · · focus · HN ↗
otterley · · focus · HN ↗
GodelNumbering · · focus · HN ↗
I had to intervene a few times. For instance, as smart as the models are said to be (Astra), it would copy the full training run, train on the server, pull every checkpoint to the local machine, then run tests, update. So, the bandwidth bill was as high as training bill for the first 6 hours. It could have simply tested each checkpoint on the server, saved time and money, didn't occur to it until I said.
otterley · · focus · HN ↗
GodelNumbering · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
varispeed · · focus · HN ↗
I wouldn't put my house on it. Brave.
libria · · focus · HN ↗
This is the part where the narrator looks at the camera and says "Don't try this at home, kids!"
bitpush · · focus · HN ↗
Are you confusing this with an OAuth token or something?
raizer88 · · focus · HN ↗
verdverm · · focus · HN ↗
<a href="https://docs.cloud.google.com/billing/docs/how-to/budgets-spend-caps" rel="nofollow">https://docs.cloud.google.com/billing/docs/how-to/budgets-sp...
degamad · · focus · HN ↗
steve_adams_86 · · focus · HN ↗
edot · · focus · HN ↗
MisterMunchkin · · focus · HN ↗
[Search: Can I refund Google cloud?]
It looks like we’re not able to ask for a refund since we did actually use all of that compute intentionally.
Would you like me to write you a pleading email to send to the support team?
hgoel · · focus · HN ↗
lopsotronic · · focus · HN ↗
PEe9bB7D · · focus · HN ↗
GodelNumbering · · focus · HN ↗
Edit: will do as soon as possible
jack_pp · · focus · HN ↗
genxy · · focus · HN ↗
equinumerous · · focus · HN ↗
jjice · · focus · HN ↗
atombender · · focus · HN ↗
onel · · focus · HN ↗
luisfmh · · focus · HN ↗
I ask cause would this be a kind of model distillation?
I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.
GodelNumbering · · focus · HN ↗
verdverm · · focus · HN ↗
edit: others have asked any you have replied "soon (tm)", looking forward for that day
jeremyjh · · focus · HN ↗
mtud · · focus · HN ↗
toasty228 · · focus · HN ↗
equinumerous · · focus · HN ↗
dominotw · · focus · HN ↗
we dont know what the result is and how its impressive.
amelius · · focus · HN ↗
Or are the subagents generating your training data using a closed/paid model?
computerex · · focus · HN ↗
Aurornis · · focus · HN ↗
For everyday work that happens frequently it's better to have a tiny specialized model instead of making billable API calls or turning your laptop into an 80W space heater for 20 seconds to run a general purpose model.
The large models can be used to generate synthetic training data. Tell them to make up 100,000 tasks paired with the resulting output as a 1-time cost. Then use that to train a small model.
Think of it as distillation, but focused on a specific task.
nearbuy · · focus · HN ↗
verdverm · · focus · HN ↗
liquicity · · focus · HN ↗
I think OP's point remains, if you generate 140k pairs, your local model would need to run that many to offset having just used the generator (SOTA or not) model to begin with.
I wonder if another approach if latency is a concern is just to do a two shot pass with Jev (perhaps given small context you'd want one to match command, then one to match args of given command) would be an extremely fast, and cheap way to do it - rather than training your own.
viraptor · · focus · HN ↗
Not quite. One, because you save on the initial query being sent multiple times. Two, because the reasoning will be very similar at the beginning; "user asked me to", "let's check what's in this project already", etc. you'll get similar actual output cost, but input and reasoning will be shorter.
kubb · · focus · HN ↗
computably · · focus · HN ↗
Lalabadie · · focus · HN ↗
This particular example is maybe a niche, but 1400 people can use a few hundred queries in a reasonable amount of time.
8n4vidtmkvmk · · focus · HN ↗
selcuka · · focus · HN ↗
carsoon · · focus · HN ↗
Which 5$ is a pretty easy sell if its useful in any way, It's pretty easy to justify a purchase if its yours forever and doesn't use much CPU so is easy to run I mean people were spending 1000$+ on mac mini setups to run local llms or run remote agents.
nearbuy · · focus · HN ↗
They also used Astra for the coding, which they can't get on a $10 subscription. And then there's the actual training cost.
jamienk · · focus · HN ↗
newswasboring · · focus · HN ↗
tomrod · · focus · HN ↗
msdz · · focus · HN ↗
askl · · focus · HN ↗
edschofield · · focus · HN ↗
lp92 · · focus · HN ↗
teeskay · · focus · HN ↗
brainless · · focus · HN ↗
I am building a natural language to CSV/Excel commands for a "wrangler" type desktop app. Same issues. The MVP is being built with parsers of sorts, entirely code generated. Then I want to fine-tune a tiny model at some point.
<a href="https://github.com/brainless/baho" rel="nofollow">https://github.com/brainless/baho
okamiueru · · focus · HN ↗
verdverm · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
xhevahir · · focus · HN ↗
It's good to hear you're enjoying yourself, but I suggest retiring that expression. It's really beginning to grate.
willy_k · · focus · HN ↗
JSR_FDED · · focus · HN ↗
torginus · · focus · HN ↗
oDot · · focus · HN ↗
verdverm · · focus · HN ↗
Aside on the aside, I welcome this new era of really personal software. Not Ai's being sycophants, rather being able to easily and quickly change, adapt, or extend software I am not familiar with.
verdverm · · focus · HN ↗
<a href="https://huggingface.co/vwdubb/Qwen3.8-27B-Fable-Distill-NVFP4/tree/main" rel="nofollow">https://huggingface.co/vwdubb/Qwen3.8-27B-Fable-Distill-NVFP...
side quest, are fable distillations only wrong when it's another country?
amrrs · · focus · HN ↗
yashthakker · · focus · HN ↗
[dead]
soundworlds · · focus · HN ↗
ByteOfWood · · focus · HN ↗
Not my project
peab · · focus · HN ↗
brainless · · focus · HN ↗
A tiny, smaller than 1b parameter model, fine-tuned, can kick ass for constrained work. I do not have a lot of budget, I fine-tune only on a 16GB M4 Mac Mini. But that also tells me the potential is wild. Progress has been slow since I moonlight on this.
I have been trying to build a set of models + agents for full-stack development, where each model does only a small piece, like take user prompt and break into backend/frontend tasks. Then a Rust+Diesel model, a Rust+Auxum model, a Solid+Router model and so on. I know this is wild but this is just theory - can 5 or 6 Qwen 3.5 0.8b models do full-stack web development? My hunch says they can, better than what most people expect. Heck, with a good harness, it might beat all the cheaper models for the specific task, like Haiku or Luna.
RugnirViking · · focus · HN ↗
jordz · · focus · HN ↗
That being said it’s much easier at the moment to continue to use the frontier providers for most general tasks, that is the argument I’ve heard.
For creating these types of fine tuned local models, on constrained hardware for inference, I do think this is the way to go for specific tasks too!
pitched · · focus · HN ↗
brainless · · focus · HN ↗
Have you actually fine-tuned yourself? Email categorization comes to mind and there are tons of non-LLM approaches even that will give fantastic results. How did spam filters work before LLM?
I think LLMs just made us think that is the only way. It is not.
williamse · · focus · HN ↗
[dead]
awwaiid · · focus · HN ↗
onel · · focus · HN ↗
busfahrer · · focus · HN ↗
I have a single command that fires up llama.cpp on cpu only using gemma4 e2b, answers a single question from the command line and exits. This takes about 3 seconds to load from an SSD, and is smart enough to solve exactly these "remind me of the syntax" scenarios if you dont wanna switch to a browser.
GCUMstlyHarmls · · focus · HN ↗
When I do something "heavy", my 9800x3D will kick on the fans and start making lot of heat. This is fine when I'm intentionally doing say a transcode, but if I'm just querying syntax it could get pretty annoying. Do you "feel" it? I know that will be pretty machine dependent.
everforward · · focus · HN ↗
I’m really pushing my recall, but I want to say it was written in Ruby and stored pre-configured commands that it just did traditional search over.
I vaguely recall it working okay because 99.99% of the questions people asked were the same (“tar command to gzip a directory and strip the prefix” is something I google like once a month).
ijidak · · focus · HN ↗
Can you (or your agent) please write a tutorial or share some good links.
(Namely the fine tuning part.)
I'd like to learn how to do this as well.