‹ BackHN Continuity

Thread

Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

575 points · 225 comments · firelex

  1. pan_lid · · focus · HN ↗
    30ms for a 0.8B decision model is crazy fast. Most of my inference pipelines struggle to hit that with smaller models.
    1. prodigycorp · · focus · HN ↗
      how many ms is it for a 32k token prefill?
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.