Thread

1. wk_end · 1789928618 · focus · HN ↗

   From a preservation perspective, this is one of the least useful decomps ever, given how committed Capcom is to making sure RE4 is ported absolutely everywhere (kidding!)

   Made using the leaked debug build and its symbols, which shows how meaningful the work the game preservation community that acquires and distributes these things is.

   OTOH this is pretty off-putting:

   > Where the compiler needed a particular source shape to reproduce a register choice or a schedule and no natural spelling was found, the construct is marked with a // COMPILER-DIFF: comment (644 of them: dead tests, empty asm("") launders and anchors, register T x asm("rN") pins, padding statements).

   To me the value of a decomp isn't reproducing the original bytes per se (we already have the original bytes after all) - it's about reconstructing the understanding of the original game as represented by human-readable source code; getting byte-for-byte is just an indication that you've gotten it right. Needing to add a bunch of slop to force the compiler to match the original output is actually just an indication that you've gotten it wrong - and it's a demonstration of the danger of Goodhart's law, especially as it applies to AI.

   1. Pannoniae · 1789929053 · focus · HN ↗

      Correct, this is just hacks for stuff you didn't manage to match exactly. Either that or your build environment isn't the same. Sadly, there are things which aren't really possible to reproduce in a byte-identical manner, things like exact file layout or variable declaration order, compilation order and that kinda stuff. And they might cause small but equivalent changes like different inlining/optimisation decisions, so it's really tricky to get it byte-exact.

      1. mitxela · 1789937794 · focus · HN ↗

         Why wouldn't those be possible to reproduce?

         1. Pannoniae · 1789939861 · focus · HN ↗

            "assign all these pointers to the correct types, a wrong guess leads to different COMDAT folding"

            "get the order of local variables in this function right, otherwise the register allocation doesn't match. Oh and there's 150 local variables just in this function, good luck trying them all"

            "Find out the translation unit boundaries exactly (assume there's no pdb otherwise this is trivial) and after doing so, figure out the order they were compiled in, otherwise it won't match"

            "brute force the compilation flags for the project and if you're done, also bruteforce it for the CRT or any other middleware which usually came prebuilt so it doesn't match the main game"

            Should I continue;)

            1. mitxela · 1789943008 · focus · HN ↗

               Indeed it's a shitload of work. It's also possible and has been done. You don't have to only use global brute-force - you could also reverse engineer the compiler. The OOT/MM decomps achieved completeness without //COMPILERDIFF.

               1. Pannoniae · 1789943691 · focus · HN ↗

                  Correct me if I'm wrong (I'm not very well-versed in game decomp scenes) but aren't all those bytematched decomps from 90s or at the latest early 2000s games? They didn't have global optimisation (MSVC introduced it in VS .NET or 2003 I think and many games didn't use it until later)

                  So these are mostly problems with more advanced compilers yk

                  1. dezgeg · 1789977119 · focus · HN ↗

                     Correct, N64 and PS1 era projects can essentially always be strictly tackled one function at a time. There are very minor things (like string literal sharing) where some earlier function can affect later function, but the workarounds are comparatively simple.

                     On the other hand, I have heard even old MSVC is a nightmare. Various things are affected by hash table ordering (so the names of variables matter in some situations), stuff like order of #includes mattering, etc.

                     1. Pannoniae · 1789977896 · focus · HN ↗

                        Yuuuup exactly. I guess this is masked because old 90s games are often more culturally significant so people haven't found these problems yet but yes, these are serious challenges.
   2. toast0 · 1789929918 · focus · HN ↗

      Certainly, it's better if you don't have these. But if I was working on something like this, I think you release when you have something that covers most of the territory, and then refine it over time.

      Maybe someone else comes and helps out here and there.

   3. brandonpelfrey · 1789931569 · focus · HN ↗

      You might find it interesting that I am working on AI-driven decomp of a PS1 game not by matching bytes but by having agents produce C code and test code. Agents submit a C proposal to the harness, the harness compiles their C proposal for the original function, then the test suite provided by the implementer runs against the original machine code and the compiled C code version. The line and branch coverage of both must be 100% and given identical inputs and starting RAM, the function return value and RAM state and RAM/MMIO read write sequence must be identical. This is done with a small MIPS simulator which can run all the tests extremely fast.

      The reason I’m finding this is much faster than a traditional decomp is that while it’d be nice for the bytes to match, finding the perfect blend of compiler version, compiler args, permitting variables etc to try and find the perfect register assignments, etc is all very time consuming. My ultimate goal is not a byte for byte match, that’s just one way to ensure correctness. I’ve found agents are much faster and effective at reading the original assembly and understanding what’s going on then writing semantically equivalent C.

      1. Tiberium · 1789932039 · focus · HN ↗

         That's a great approach, I think byte matching is just popular because it's extremely easy to test in the end: are the bytes the same? While your approach requires putting far more trust into the tests.
   4. ErroneousBosh · 1789932709 · focus · HN ↗

      > given how committed Capcom is to making sure RE4 is ported absolutely everywhere

      Heh. I was just looking at the RE family on Steam. Nothing over a fiver.

      I guess they really did make it more convenient than piracy.

   5. jchw · 1789933459 · focus · HN ↗

      Making a function fully matching by virtue of hacks like this is mostly not harmful to the ability to understand the code, but is a useful tool in ensuring that the code as a whole really is matching and identical to the original. Otherwise, it's difficult to prove.

      There is also value in decomps beyond just understanding and general interest. They can also be used to make more advanced mods, better translation patches, etc. The fidelity here matters, like having asm code accessing structures makes it hard to modify structures, but any fidelity improvement beyond pure asm is very welcome.

      Of course I still prefer to try to recover the original code that caused the compiler to do what it did, but it's a really challenging problem sometimes. I've been working on decompiling code from old versions of MSVC for literally years now and you accumulate some knowledge of what things impact register allocation or the order of symbols but some of it comes from things that get fully erased from the source. Like for example, debug builds generally seem to retain symbols that aren't actually referenced anywhere, but those symbols only actually make it into an object file if they are. For functions that were only ever inlined and not actually referenced anywhere... They still wind up in the object files and thus in debug builds, despite nothing referencing them. They are also COMDAT any'd because they can appear in multiple objects legally, which means the exact object that winds up retaining it in the final linked executable is arbitrary (and the compilation flags of the object containing it, too - I bet that was fun for developers to debug.) This is incredibly useful but very challenging, needless to say. It may even be feasible to construct examples that would be legitimately infeasible to simply guess back to equivalent source, which I suspect is a major reason why until it was finally shown to be possible in larger scale projects many people wrote fully matching decomps off as a fool's errand..

      1. Pannoniae · 1789939353 · focus · HN ↗

         With /LTCG /GL I don't think it's possible to get matching (or at least it's a very tall order), the codegen is wayy too volatile for an exact match and since inlining and reg alloc work on heuristics with thresholds it really cascades. Even stuff like what order you declare your locals in or the exact frontend syntax can mess things up...

         1. jchw · 1789940449 · focus · HN ↗

            Most people are working on builds where debug information was either inadvertently or sometimes intentionally included (e.g. for beta releases sometimes debug builds with debug info would ship for sake of making things easier.) These builds usually don't have all of the normal release optimizations on. That makes it much more likely to get a match.

            I haven't tried this, but I also suspect that once you have a lot of code fully matching, it might make it possible to ratchet your way up further into builds that you don't have debug information for, that may have more aggressive compilation options. I am not sure if you would manage to get /LTCG builds fully matching even with this advantage, but it's going to be the best shot at it. You're possibly 90% of the way there already.

            1. Pannoniae · 1789941583 · focus · HN ↗

               Hey I'm not working in matching something with LTCG atm :) I'm just saying in general.

               And yes if you have a pdb / an Od build then things are much easier, I was assuming arbitrary game i.e. release binaries.

               The "knowledge laundering" approach you describe might help in reconstructing headers, class layouts and function names which is a godsend although I don't think it would be enough to get a match. Getting functionally equivalent code is muuuch easier (although there's the problem of "how do you verify that without running every function")

               1. jchw · 1789947148 · focus · HN ↗

                  Yeah, this is probably true. I've done non-matching decompilations of modern software up to a few hundred kilobytes worth of code - it is challenging but doable. I have no idea how hard it would be to get to matching with LTCG no matter where you start from. If it was genuinely not practically possible for computational reasons I would be unsurprised.

                  AI is pretty powerful for decompilation, especially because you can also just have an LLM go and start reverse engineering bits of the linker and compiler if you want. (I suspect this decomp is AI assisted if the Clauded out README is any indication.) Maybe future models will be able to come up with clever and novel ways to reduce the number of possibilities and converge faster on possible matching source codes. Or maybe not; I think Astra is the best LLMs have ever been at decompilation and yet I find LLMs frustrating and prone to getting deeply stuck in local maxima in my experimentation.

                  1. Pannoniae · 1789977813 · focus · HN ↗

                     Yeah IMO the next big thing will be a way of verifying equivalence while filtering out the "noise" differences in optimisation. Otherwise you can't really decide whether you got it correct or not.
      2. StilesCrisis · 1789960101 · focus · HN ↗

         Working on a decomp hobby project now, and "100% matching" on an individual function can be so deceptive! These two functions could look identical in assembly:

         void foo() { bar(); }

         int foo(int x, float y) { return bar(x, y); }

         This is because parameter-passing and return-value handling can sometimes be done in zero instructions in PPC, if the values are already in the right registers. So you need to continually go back and reassess old functions once you learn the signatures of newer ones!

         1. jchw · 1789961796 · focus · HN ↗

            Oh yeah - absolutely. This isn't even just a PowerPC thing - it happens more generally, because usually if you're forwarding both arguments and return values for a function that has the same calling conventions, you don't really have to do much adaptation, everything was already set up for you. It hit me a ton on x86 when I first started my journey into trying to reverse engineer. It still occasionally hits me, especially when I'm relying too hard on decompiler output and not paying enough attention.

            There are plenty of examples where two very semantically different source codes can have the same output, which is really counter-intuitive when combined with the difficulty that often comes with trying to find a single source code that does match.

            This is just another reason why debug information is such a godsend; having symbol names for C++ code will usually give you most of the function signature, and type information will give you the rest. That greatly enriches the disassembly, and the automated decompilation output, and no doubt constrains the number of possible source codes that could match both the output and the debug information, somewhat alleviating this issue.

            1. StilesCrisis · 1789993341 · focus · HN ↗

               Yup. Unfortunately the project I'm working on does not have any debug info. There are not even lingering __FILE__ strings anywhere. Just picking out the TU bounds has been a chore!

               It's a huge benefit that things like the OS and SDK are known and predictable. This got me a foothold. Otherwise I would have gotten nowhere.

               I'm still impressed that this decomp has macros (fully lost in the assembly) and nearly-100% meaningful variable names and struct layouts. That's not easy!

               1. jchw · 1789994426 · focus · HN ↗

                  How in the world are you finding reasonable TU boundaries, without debug info? I've always been puzzled by this in particular.

                  1. StilesCrisis · 1789995902 · focus · HN ↗

                     Float constants are stored as data which is deduplicated on a per-TU basis. So if my TU has 1.0 and 0.0 in it, every function in that TU which uses those numbers will load from the same address. The next TU will get its own dedicated 1.0 and 0.0 constants.

                     1. jchw · 1789998751 · focus · HN ↗

                        That is pretty clever. In a particularly C++ heavy program I was able to find decent splits by starting around where the vtables were stored and searching from there, which was helpful since they also often pointed into the relevant part of the .text section. The repeating floats thing definitely seems to ring true though. For old MSVC it seems like it will emit floats even if they aren't directly referenced, if they happen to be referenced by inline functions that are not used. That adds another challenge if you want to get really close to the actual original source: figuring out what those inline functions actually are instead of just dropping a dummy that references them and is never used :) But I'd say that's extra credit, if you ever got to the point where that was the only remaining debt...

                        1. StilesCrisis · 1790001411 · focus · HN ↗

                           For sure, actually mirroring the shape of inlined code properly rather than just emitting code N times is already extra credit as far as I'm concerned. I've got enough loose ends without chasing that sort of thing!
   6. mitxela · 1789937736 · focus · HN ↗

      The OOT and MM decomp had a bruteforcer tool that would reorder lines until the register allocation matched.

      1. Sesse__ · 1789999133 · focus · HN ↗

         You&#x27;re probably thinking of <a href="https:&#x2F;&#x2F;github.com&#x2F;simonlindholm&#x2F;decomp-permuter" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;simonlindholm&#x2F;decomp-permuter, which is used across many more decomp projects (and can do a lot more than swap lines). It&#x27;s a huge help, but it&#x27;s by no means enough for all regalloc differences; there&#x27;s lots of stuff it cannot do.

Unofficial Hacker News client; not affiliated with Y Combinator.