‹ BackHN Continuity

Thread

Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

575 points · 225 comments · firelex

  1. pan_lid · · focus · HN ↗
    30ms for a 0.8B decision model is crazy fast. Most of my inference pipelines struggle to hit that with smaller models.
    1. prodigycorp · · focus · HN ↗
      how many ms is it for a 32k token prefill?
    2. anvuong · · focus · HN ↗
      Is it though? I got sub 10ms for 200M params vision model way back in 2020 with TensorRT optimization running on RTX 2080Ti. 30ms for 0.8B classifier model doesn't sound special to me.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.