Thread

Analyzing Frontier Model Progress with My Favourite Game: Prince of Persia

63 points · 43 comments · msephton

Loading the complete thread in the background. This saved snapshot is available now. Refresh

smokel · 1790435692 · focus · HN ↗
The prompts provided are atrocious. It's amazing that the LLMs actually built something useful.
> My first prompt was simple: "in this original code there 6502 assembly code for prince of persia, use the save level files and try to do it in c# console."
Do what in the what now in the console?
1. gcanyon · 1790435995 · focus · HN ↗
  "now char coming but movement everything wrong ... prince is not in floor"
  I'm assuming a language barrier on the part of the author. I wonder if the models would do better being prompted in the author's main language.
  1. cavemandaveman · 1790439079 · focus · HN ↗
    Or having another model proofread the prompts and write clearer instructions
2. koala_man · 1790438579 · focus · HN ↗
  Unsurprisingly it was buggy without the obligatory "Make no mistakes."
3. daemonologist · 1790439890 · focus · HN ↗
  Yeah, I'm thinking the nigh-unreadable AI-speak we get these days makes a lot more sense if this is what they're training on. Or maybe the author has translated the prompts from another language?
4. InsideOutSanta · 1790440393 · focus · HN ↗
  It's so confusing how the actual article is in English, but the prompts are just gibberish. But LLMs are pretty good at deciphering gibberish; I often put our CEO's absolutely atrocious E-Mails into ChatGPT and tell it to explain wtf he wants from me.
  Also, I feel like the LLMs would have done better if they had started from scratch each time, rather than being burdened by the output from the previous attempt.
5. aksss · 1790449326 · focus · HN ↗
  The prompts were awful and really distracted from an otherwise cool project concept. Also, testing the next model by iterating on the first model’s crappy foundation. Why not start from the original each time?
6. opiotrek · 1790485362 · focus · HN ↗
  That was my same thought, these are barely coherent sentences in English.
criemen · 1790435711 · focus · HN ↗
A German magazine (c't) ran a Asteroids programming contest in 2008 - create a client with access to the emulator output of the game that provides keyboard inputs to control the game.
The highest scoring submission that won the contest had a high score of around 137k. Last week, I had GPT-6 Astra, Sol and Luna implement and hill-climb on this task, as I wanted to see how big the difference in smartness is. Luna implemented something, but never exceeded ca. 20k points, with a large variance. Sol got something in the area of the humans implementation.
Astra, which finished fastest, had a highscore of around 1.7Mm when the game seemed to fairly reliably crash. On the way, it disassembled parts of the ROM to extract information about the game.
I didn't do a ton work to document and measure the specifics, but it was very impressive.
1. looperhacks · 1790437797 · focus · HN ↗
  If I remember correctly, back then all "good" entries in the contest solved the game by syncing with the RNG and could basically predict where and how new asteroids would spawn. The difference between entries was planning ahead movement and residence against control issues due to the contest setup (like inputs getting delayed). I wonder how Astra managed so much better, I thought the problem was already solved
  1. criemen · 1790440399 · focus · HN ↗
    Thanks! I went back, and had a look at both the contest rules and the highest scoring submission again. Turns out I benchmarked against infinite games, whereas the contest measured a 5min time limit. So, Astra solved a different problem.
    Comparing strategies, though, Astra does something pretty different from the highest scoring submission. It doesn't predict the RNG, instead it reacts to the visuals on-screen, planning ahead by estimating velocity of all objects on screen. Unlike the winning solution, it actively flies the ship, whereas the winning solution basically only rotates and teleports. So Astra behaves more like a regular "perfect" player would, rather than one that breaks the PRNG.
    suddenlybananas · 1790445335 · focus · HN ↗
    It is a bit frustrating that you reported it as being so much better than human made solutions despite not checking even the most basic conditions.
    criemen · 1790445599 · focus · HN ↗
    My apologies, and it's too late to edit the parent now.
    To be fair, though, I mainly ran this experiment to distinguish models, not to see if they were better than humans or not, so for that purpose it didn't really matter if the basic conditions were the same or not.
2. NitpickLawyer · 1790443264 · focus · HN ↗
  > On the way, it disassembled parts of the ROM
  You should check the logs, there's probably a bunch of retro gaming forums that got hacked behind the scenes :D
rhipitr · 1790436783 · focus · HN ↗
Off tangent but the latest prince of Persia game is really fun
1. pawelduda · 1790437920 · focus · HN ↗
  I played The Lost Crown and it's top tier, I assume you're talking about The Rogue?
2. Kuyawa · 1790437992 · focus · HN ↗
  The Lost Crown? <a href="https://www.youtube.com/watch?v=n4vaxKwZ5rQ" rel="nofollow">https://www.youtube.com/watch?v=n4vaxKwZ5rQ
  That looks really amazing
  1. dofm · 1790439149 · focus · HN ↗
    It looks like one of those spam shooty game ads you get between Youtube Shorts.
    Prince of Persia is a piece of contemplative, subtle, beautifully, artistically minimal motion puzzle art. God knows what that is.
  2. dayvid · 1790440464 · focus · HN ↗
    It's one of the best recent metroidvanias. It deserves a lot more attention. I was surprised by how good it was.
  3. 7734128 · 1790442049 · focus · HN ↗
    However, it's Ubisoft and therefore tainted.
Kuyawa · 1790437535 · focus · HN ↗
[dead]
Kuyawa · 1790438381 · focus · HN ↗
I am going to try to build the original Prince of Persia using Swift as it is the best language for building a macOS game. Being a classic 2D cinematic platformer, Apple's native frameworks provide exactly what I need without the overhead of a massive cross-platform engine, and I love simplicity
So here goes my weekend, flag this and get a life
1. andsoitis · 1790439447 · focus · HN ↗
  Share it when you're done! Even if it isn't complete complete.
  1. Kuyawa · 1790453906 · focus · HN ↗
    Already started and will be ready soon, check back in an hour
    <a href="https://github.com/kuyawa/prince" rel="nofollow">https://github.com/kuyawa/prince
    Kuyawa · 1790460400 · focus · HN ↗
    Sorry about the delay, some bugs to swat
    2 hours later and $2 in tokens, Prince of Persia in Swift 6 for MacOS is ready
    Not fully tested by me (AI tested it fully on every milestone, ~340 tests) but for sure tonight I'll do it from beginning to end
    Let me know what you think, feedback is always welcome
    And btw, cmd+1..5 can change the resolution to a bigger canvas, I prefer it at cmd+5 almost full screen. Haven't tested retina display as I use a TV as monitor and resolution is not optimal
    Enjoy!
    Kuyawa · 1790463519 · focus · HN ↗
    Adding respawn and saving last level so restart takes you there and not to the beginning
    I forgot where the sword is, can't pass the first guard, hmm
dofm · 1790439065 · focus · HN ↗
Just one thing you could do is not drag Prince of Persia into this.
Please.
hapless · 1790443592 · focus · HN ↗
when the author mentioned playing prince of persia on a pc-xt, i assumed this was some kind of ai-generated nonsense
but no
they actually did port the game to the lowest-end hardware available in 1989. you could actually play prince of persia on an 8088 with a cga card!
1. brandonmenc · 1790445121 · focus · HN ↗
  [delayed]
  1. hapless · 1790453275 · focus · HN ↗
    The XT came out in 1983, it was a real piece of shit by 1989 standards
    Not to mention, the PC was a pretty, uh, "challenging" platform for video games. It amazes me someone managed to port something as graphically sophisticated as PoP to an 8088 with CGA.
    brandonmenc · 1790472871 · focus · HN ↗
    [delayed]
    InsideOutSanta · 1790496838 · focus · HN ↗
    PCs were incredibly expensive at the time, though, and barely useful. Most people didn't buy a new one every two years, despite the fast progress in hardware development. So many people, especially kids, were stuck with "ancient" hardware. I had friends who had an 8088 PC or an Apple II in the late 90s.
jessechili · 1790447445 · focus · HN ↗
[dead]
tehlike · 1790451065 · focus · HN ↗
Instead of fixing a broken version, maybe each model should have started from scratch
1. jablongo · 1790460122 · focus · HN ↗
  yea for a real comparison between models this would have been better, but I have a feeling he just wanted to play the best possible version of PoP.
  1. tehlike · 1790468609 · focus · HN ↗
    That may still be a new model. Having an existing broken engine might bias the new model in iterative design vs figuring out what's wrong in the first place.
2. msephton · 1790474356 · focus · HN ↗
  I agree this would have been more interesting.
oynqr · 1790451978 · focus · HN ↗
This is one of the worst articles I have ever read that was posted on this site. I will not elaborate.
1. jackblack11 · 1790452211 · focus · HN ↗
  This site is full of corporate simps and ai douches for the past 5 years, what do you expect?
jackblack11 · 1790452243 · focus · HN ↗
Corporate simps and ai douches will love this ai blog spam!
gambiting · 1790457970 · focus · HN ↗
The real miracle of AI is that it could work with these prompts at all
"now char coming but movement everything wrong ... prince is not in floor"
I'm a games developer and I have no idea what OP is asking for here.
jablongo · 1790460073 · focus · HN ↗
This isn't really comparing the models; he's using successive models to improve his rebuild of Prince of Persia. Either way though, it will be interesting to see which games, if any, resist implementation via LLM. I recently started a Starcraft 2 like real time strategy game with Fable managing many Opus agents and it got to a playable 3d in 3 days.
duhhhhh1212 · 1790460596 · focus · HN ↗
<a href="https://www.pangram.com/history/09ae8c73-2d40-432b-aea7-c764e7739326?ucc=t8c6BkzVOh8" rel="nofollow">https://www.pangram.com/history/09ae8c73-2d40-432b-aea7-c764...
Don't waste your time.
1. opiotrek · 1790485475 · focus · HN ↗
  There's a massive discrepancy between the English in this post and barely coherent prompts in it
rothron · 1790461299 · focus · HN ↗
I see a lot of these attempts at getting a model to replicate a classic game. I'm sure this is a use case with some interest, not everyone wants to spin up an emulator, but it strikes me as odd. Rather than English being the new high level language for programming, the LLMs is handed a finished game as a specification and asked to copy it.
That gets you past the problem of the lack of specificity in English, but requires you to already have the thing you want to get made. At least tokens are cheap.
1. OakNinja · 1790469243 · focus · HN ↗
  English has been the high level language for programming since Common Business-Oriented Language - COBOL.
  Programming has always been defining the problem you have. Nothing has changed from that perspective. Sometimes that problem is recreating something that already exists :)
keeda · 1790466174 · focus · HN ↗
I see your Prince of Persia and raise you seventeen games fully or partially reversed and ported to the web by one very capable guy in a few weeks:
<a href="https://georgzoeller.com/blog/posts/what-reverse-engineering-and-modernising-an-old-war-game-tells-us-about-the-econ/" rel="nofollow">https://georgzoeller.com/blog/posts/what-reverse-engineering...
He also did this thing where he “spliced” different game concepts and turned WoW into an FPS: <a href="https://news.ycombinator.com/item?id=49652089">https://news.ycombinator.com/item?id=49652089
In case you mistake him for an “AI booster”, you should see this guy’s other posts, he is extremely critical of the AI frontier labs and Tech Bros and this whole AI bubble.
But he is also a very experienced game developer, and under his control the latest models have pulled off these crazy feats. What you are reading is the analysis of a domain expert and what it means for the industry.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.