Thread

Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

575 points · 225 comments · firelex

hadlock · 1790654014 · focus · HN ↗
If you're looking for a more impressive doom example, I put together laya-duum which uses open source micropython implementation of doom (duum) and freeware freedoom1.wad. It uses the standard jev api and will play through the first two levels to completion: <a href="https://github.com/Hadlock/laya-duum" rel="nofollow">https://github.com/Hadlock/laya-duum
1. sunbum · 1790664229 · focus · HN ↗
  Why is this more impressive?
  1. hadlock · 1790664370 · focus · HN ↗
    It's not just shooting at static monsters in an empty room, this will go after monsters, make decisions about prioritizing health, changing goals, and finish the level, etc. It's not unique, it's just more impressive than the demo they're using.
    runeks · 1790668810 · focus · HN ↗
    Their demo could also change goals, e.g. avoid monsters instead of shooting them, by changing some variable. I'm sure that could also be used to change goals.
    mosselman · 1790678080 · focus · HN ↗
    But it doesn’t. That is why that other person is claiming their demo is more impressive.
    What you are saying is like me losing 13kg is as impressive as having the potential to do so if only I would stop eating ice cream and too many snacks every day.
    trencedamp · 1790686513 · focus · HN ↗
    You're confusing demo with technology.
    The potential to lose weight, in this context, is the original post because there was potential but it wasn't shown
    This commenter is the person who lost 13kg.
    The technology is not different but the demonstration is more impressive
2. antman · 1790667634 · focus · HN ↗
  - Laya receives semantic snapshots, not the framebuffer.
  What does tjis mean? Another generative model?
  1. ovi256 · 1790676660 · focus · HN ↗
    Laya is the model this uses. Like Jev, it's blind - it can't take an image as an input. So how can it play Doom, or Atari games, or do other visual tasks, as people have shown it to do? They write a bit of app-specific adapter code that transforms the game state into a structured piece of data (here, the "semantic snapshot"). And that is what these models take as input
3. prodigycorp · 1790669233 · focus · HN ↗
  laya sucks. it's a dead end and for some treat it as a peer of jev or even the qwen jevs. base model doesnt have enough world knowledge to generalize.
  1. earino · 1790672710 · focus · HN ↗
    The base-model criticism is fair. Laya is weak zero-shot and doesn't have the world knowledge of a much larger pretrained model. Their own site puts the base models at around 0.35 on the typed-decisions benchmark, which is close to random.
    I don't think that makes it a dead end. Laya is designed to be fine-tuned for a specific task, and the site reports the fine-tuning gains. The 0.766 number comes from fine-tuning on the benchmark's train split, not from the base checkpoint. They also report that fitting a single temperature scalar per question type cuts expected calibration error from 0.466 to 0.081. That's a large gain, and it only shows up after you specialize the model.
    "Peer" is doing a lot of work in that comment. Jev can take on a new task without retraining because it starts with far more knowledge; Laya trades that away to stay small and trainable for a fixed task. So I wouldn't compare base Laya to Jev and stop there. Compare Jev to fine-tuned Laya on the same task and test set, then look at accuracy, latency, cost, calibration, and robustness, depending on which of those matter for the deployment.
    prodigycorp · 1790674204 · focus · HN ↗
    btw your comment was immediately marked as flagged/dead. Odd for a high quality comment. I vouched for it.
    My main point of disagreement would be that I fundamentally see a different use for a jev sort of model (generalism is applealing), but if you're finetuning, a bert base is not bad.
    earino · 1790675367 · focus · HN ↗
    Thanks for the vouching. I don't really understand what I did to anger the algorithms! But yeah, I think that that's the real sticking point. If the value prop of jev is "plug it in anywhere and it generally does smart things" you're absolutely right that Laya is Not A Thing. I've spent a bunch of time working on fine tuning models (i teach a class on it! <a href="https://earino.github.io/applied-deep-learning/" rel="nofollow">https://earino.github.io/applied-deep-learning/ using the same ModernBERT stuff!) and so for me, I naturally went: "wait a second, so how good is Jev compared to fixing the problem domain."
    Anyways, thanks for the vouching!
    prodigycorp · 1790676713 · focus · HN ↗
    Great course, thanks for sharing, going through your slides and papers right now.
    fwip · 1790688932 · focus · HN ↗
    My guess is, it's because you sounded like an LLM in a few phrases there.
    earino · 1790693262 · focus · HN ↗
    I think that's a pretty good guess. I move around different countries a lot so I also assume I can look like a bot farm from multiple source IPs.
    However at this point I talk to LLMs more than anyone except probably my wife. As a multiple times immigrant, I can absolutely believe I'm adjusting my speech patterns to its vernacular.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.