Show HN: Pac-Bench – How well can models one-shot a Pac-Man game?
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
Show HN: Pac-Bench – How well can models one-shot a Pac-Man game?
Unofficial Hacker News client; not affiliated with Y Combinator.
weitendorf · · focus · HN ↗
Clearly some new RLAAS/dataset/env is being used for this now (it doesn't even seem that complicated, you have one LLM judge whether gameplay is recognizable as the original game or not and another trying to implement a logically/semantically identical version of the game). It's why the performance improvement on this workload has been so dramatic.
Everything is going to go from 0->1 on this benchmark in short order because of that.