‹ BackHN Continuity

Thread

Ember-1

589 points · 249 comments · gmays

  1. GodelNumbering · · focus · HN ↗
    This is the golden age of model training. Some days ago, I decided I wanted a local CPU only model that can perform exceptionally well for English to Bash translation (to avoid the googling for command syntax). I got a bunch of subagents to generate large amount of training data (140k+ samples), got the Qwen 3 0.6B base model, pointed Astra at it, and off to the races. It trained for 2 days (on and off) and I got a surprisingly good model for my task! The total active time I spent was a few hours. And it is still improving, what a time to be alive!
    1. luisfmh · · focus · HN ↗
      Curious about how you generated the training data? Was it just asking an existing model to generate a bunch of examples?

      I ask cause would this be a kind of model distillation?

      I have a small model I'm looking to train on some data, and I have some real live data but I'd love to be able to extend it.

      1. GodelNumbering · · focus · HN ↗
        All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.
        1. verdverm · · focus · HN ↗
          Do you have a write-up or git repo for this? Would love to learn more and/or dig into the guts

          edit: others have asked any you have replied "soon (tm)", looking forward for that day

        2. jeremyjh · · focus · HN ↗
          It would be awesome to share your training set on hugging face if it’s easy to de-personalize it. The largest I could find was only 800 rows.
          1. mtud · · focus · HN ↗
            I’ve previously fine tuned (SFT, currently trying to distill) models using this <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;westenfelder&#x2F;NL2SH-ALFA" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;westenfelder&#x2F;NL2SH-ALFA
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.