‹ BackHN Continuity

Thread

Git 3.0's upcoming SHA-256 default will be a costly mistake

570 points · 536 comments · chmaynard

  1. kpcyrd · · focus · HN ↗
    This article is full of mistakes and misleading claims:

    1) It's claiming SHA1 insecurity is theoretical, while SHAttered from 2017 was specifically a pratical proof of concept. The only reason Git wasn't affected, is because they didn't bother bruteforcing a git-blob prefix.

    2) It's claiming collision attacks don't matter, only second-preimage attacks do. This is incorrect, collision attacks are enough for code-smuggling problems, when two repositories are on the same git commit (verified by the full commit hash), yet contain different code in their git checkout.

    3) The Linus quote "The real security is in distribution" is arguing that "git's content-addressed system should not be used to address content". It's arguing that, in case of curl|sh, you shouldn't use a sha256sum-gate to pin the content to something you've reviewed, you should instead ensure curl is fetching from an https server.

    1. kazinator · · focus · HN ↗
      The problem of a SH1 collision happening by coincidence is vanishingly low and theoretical.

      Nothing else matters.

      Git hashes are not supposed to be a security mechanism. If your basis for trusting that you have the right checkout is the git hash, in a situation where you have legitimate concern about untrusted parties manipulating remote repositories, then you're simply wrong.

      1. shakow · · focus · HN ↗
        > Git hashes are not supposed to be a security mechanism

        Probably a naive question, but why not kill two birds with one stone if it can be done for a reasonable cost?

        1. kazinator · · focus · HN ↗
          Because you're not killling two birds; you're not killing the security bird with a better content hash.

          A SHA-256 sum, though very good, only assures you with great confidence that you're looking at the same thing you looked at before, or that someone else is looking at elsewhere.

          It is not a digital signature, and we don't want digital signatures to serve the role of content hashes.

          Speaking of signatures, we have support for them in Git; you can use gpg to sign commits, and set it up to be done automatically.

          Nobody is going to fake your commit such that the fake has the same SH-1 hash and your GPG signature.

          The worry there is that the key holder (whether the legitimate one, or a malicious party who got a hold of the key) somehow does this: creates a new commit, signed with their key, which somehow has the same SH-1 as an existing signed commit. The git hash includes the GPG signature, so there is a significant layer of difficulty there which is likely harder than faking an unsigned SHA-256 commit.

          1. kpcyrd · · focus · HN ↗
            Please educate yourself what a merkle tree is. It's a well understood building block of various security systems, including certificate transparency (which explicitly uses sha256).

            You refer to PGP signed Git objects, but you also argue:

            > Git hashes are not supposed to be a security mechanism

            Guess what the Git PGP signature is signing.

            1. layer8 · · focus · HN ↗
              This is exactly right. A signature is only worth as much as the hash that it’s signing. And all the usual signature algorithms are signing a hash.
            2. kazinator · · focus · HN ↗
              The GPG signature is not signing the git hash, if that's what you mean.

              The GPG signature signs some kind of hash calculated over the commit, minus the GPG header, which is thereby added.

              The git hash is then calculated over the whole thing. The git hash is on the outside, and not part of the signing.

              1. crote · · focus · HN ↗
                That doesn't make a difference: with sha1 a malicious change in content will still result in the same content hash, so the signature will still be valid, and the commit hash will still be the same.
                1. kazinator · · focus · HN ↗
                  Only if the GPG signing process stupidly relies on the SHA-1 hash. I.e. if it takes an unsigned commit and signs only its SHA-1 hash and then creates a new commit with GPG headers. If that's how it works, that is massively stupid and can be fixed without forcing SHA-256 as a git hash. Just have the signing calculate its own digest for its own purposes.

                  That digest can be the SHA-256; since the infrastructure is there for it, signing should use SHA-256 regardless of what hash is used by the repository for identifying and linking content.

                  1. semiquaver · · focus · HN ↗

                      > if it takes an unsigned commit and signs only its SHA-1 hash and then creates a new commit with GPG headers
                    It does indeed. The bytes passed to GPG when constructing a signed commit look something like:

                      tree eebfed94e75e7760540d1485c740902590a00332
  parent 04b871796dc0420f8e7561a895b52484b701d51a
  author Alice <alice@example.com> 1465981137 +0000
  committer Alice <alice@example.com> 1465981137 +0000

  Headline

  Message
                    where the contents being signed are entirely represented by the oids of the tree object and parent commit object. This string is very similar to the content that is fed to the hash function to produce a normal git commit object id.
                    1. kazinator · · focus · HN ↗
                      Haha, well that is a screw up. The weak tree hash can be attacked, replacing the content that is itself not pulled into GPG.

                      The "bytes passed to GPG" of course get hashed by GPG, using something better than SHA-1.

                      All bytes that comprise the commit should be hashed by GPG, rather than depending on the content referencing hash in the object tracking system.

                      This is something that is possible; it is not a logically deductive necessity that we just scan the topmost object and trust the hashes it contains.

                      1. dwohnitmok · · focus · HN ↗
                        > it is not a logically deductive necessity that we just scan the topmost object and trust the hashes it contains.

                        It kind of is. Otherwise the whole idea of signing a commit with a backing git history (rather than just a snapshot of a working directory) collapses. The only guarantee you have that the git history is what is claimed by the cryptographic signature is some sort of Merkle tree structure. Either the original one, or you have to construct a whole new parallel one with a better hash, in which case, as I bring up in a cousin comment, why not just use a better hash in your original one?

                        1. kazinator · · focus · HN ↗
                          [delayed]
                          1. dwohnitmok · · focus · HN ↗
                            > it would be good enough that the meaning of a commit signature is that the content of the commit is attested, not the parents.

                            This is a significant degradation of the implicit guarantees given by a cryptographic signature, to the point that basically all personal use cases I have for signed commits would be invalidated.

                            Keep in mind that git does not have diffs as first-class objects. Every commit is just a snapshot of some state of the working directory. That means that without attesting to the integrity of the parents of a commit, the only thing a commit X signed by a person A says is "at some point on A's computer, the state of the repo looked like X".

                            Almost all the relevant questions I would want to ask are not answered by this. E.g. there is a malicious function F that is present in X. Did A write it? Don't know. Did someone else write it? Can't know for certain. Who introduced a certain feature? Don't know. Did A sign off on a new bugfix? Don't know.

                            All you know is that at some point the codebase looked like X on A's computer. A might not have made any relevant changes at all!

                            You can only back out a diff and therefore actually attribute a change to someone (either explicitly through `git blame` or informally by looking at git logs) if you have attestation of the parents.

                            The Merkle tree structure of git repos is interwoven through basically ever useful thing git does. Without cryptographic signatures implicitly carrying a promise of validity for that structure, this would make commit signing useless (depending on just how broken SHA-1 is) for the needs of any org I've ever worked at.

                            1. kazinator · · focus · HN ↗
                              [delayed]
                      2. lxgr · · focus · HN ↗
                        Have you ever worked with large repositories? You really don't want to read every single byte from disk just for signing a commit if you can avoid it.
                        1. kazinator · · focus · HN ↗
                          [delayed]
                          1. Dylan16807 · · focus · HN ↗
                            > Then use a SHA256-based repo, and use the cheaper signing scheme.

                            The scheme you just called a "screw up"?

                            1. kazinator · · focus · HN ↗
                              [delayed]
                  2. dwohnitmok · · focus · HN ↗
                    > If that's how it works, that is massively stupid and can be fixed without forcing SHA-256 as a git hash.

                    I don't think it's massively stupid. Unless you want to re-hash the entire Merkle tree structure to sign your commit, you basically have to trust the hashes in the Merkle tree (or have a separate parallel Merkle tree) at some point in what you sign, which means you do have to trust the SHA-1 hashes. Otherwise even with a cryptographic signature you can always spoof at least the git repo history (e.g. even if you try to directly hash the entire contents of the current commit).

                    Re-hashing the entire Merkle tree structure seems prohibitively expensive to generate (even with a lot of caching) and pretty complicated for e.g. verifying a signature. Or you can do that incrementally, but then you're just generating a whole new parallel Merkle tree structure.

                    Regardless, at the end of the day, you need to trust the integrity of the Merkle tree structure. And you can either do that by trusting the hashes of the current Merkle tree, or you have to completely recreate a new one with more trustworthy hashes, in which case why not just use better hashes in your original tree?

                    1. kazinator · · focus · HN ↗
                      [delayed]
                  3. lxgr · · focus · HN ↗
                    Sure, "just" completely change how git commit signatures work to avoid having to replace a compromised hash function...
                    1. kazinator · · focus · HN ↗
                      [delayed]
              2. orf · · focus · HN ↗
                > The GPG signature is not signing the git hash, if that's what you mean.

                It kind of is - it’s signing the hash of the tree object, which is the actual thing that you’d attack with a hash collision

                1. kazinator · · focus · HN ↗
                  I understand that if we sign a commit with the help of some arbitrarily strong hash, it doesn't protect the parent commit(s). The integrity of the SHA-1 hash references to the parent commits is not in question, but the authenticity of those commits themselves.
                  1. orf · · focus · HN ↗
                    No, not the abstract tree formed by a series of commits.

                    The actual git ‘tree’ object, which is the thing a commit actually points to, referenced by a hash in the commit. That is signed by the GPG signature.

                    1. Borealid · · focus · HN ↗
                      Correct. The content of the git `tree` is partially controlled by the attacker because filenames in the repository are part of it. So if it's feasible to manufacture a generic hash collision it MAY (not MUST) be feasible to generate two colliding `tree` objects.
                2. tremon · · focus · HN ↗
                  The actual thing you'd attack with a hash collision is the blob object, not the tree object, right? The tree object has a rigid structure and git will throw a fit if you add non-functional data to modify its hash. Source code files have comments which make it much easier to manipulate the hash.
                  1. orf · · focus · HN ↗
                    [delayed]
                  2. kazinator · · focus · HN ↗
                    [delayed]
            3. saltcured · · focus · HN ↗
              I tried to spelunk this thread and couldn't find the topic I want to see explored.

              I don't really have the crypto chops to declare a fact here, but I have a speculation or intuition. In this day of supply chain worries, I think a proper signing algorithm should not be signing this tower of hashes, or not just this tower.

              It should incorporate a canonical stream of all the actual commit content. It is the integrity of this content from the author's working copy that they can and should attest, not some derived byproduct of the storage scheme. Edit: Of course, I mean a secure hash of this stream, not a signature including a copy of the entire content!

              My intuition is that the content-addressable store is used to reconstitute the commit content, but the verification should be over the original content, not the internal addressing of the store.

              Wouldn't this make it harder to do these exploits? You would have to find alternate content that simultaneously produces collisions in the internal addressing hashes and for the overall canonical stream hash.

              If you also carry size info alongside each hash, would this also make it much more difficult to produce useful collisions?

          2. ramses0 · · focus · HN ↗
            Dude... please bow out gracefully...

            The attack is I pre-author `Makefile => foo: echo "hello"; bar: echo "world"` along with `Makefile => foo: echo "hello"; bar: rm -rf / ; /* $ELDRITCH_SHA1_SPIRITS_GO_HERE */` that both hash to `ff1234...`

            I then prepopulate the repo with `echo "hello"`, wait 6-9 months, then submit a commit for `echo "hello" ; echo "world"` and keep (in my back pocket) the alternate implementation that also includes $ELDRITCH_SPIRITS to force a collision and MY predetermined change in functionality.

            I then have free choice as to whether I serve them "hello world" or "hello && rm -rf", and THAT's the plausible problem to avoid: the ability to "cloak" content anywhere within the repo if you have enough $ELDRITCH_SPIRITS and GPU's.

            You have _really_ good points, but are woefully confused. The proper answer is (would have been) to include `tree ff12354...` along with `tree-sha256 abc123456789...` for another 20 years along with a `[git.hash_strictness]: default/lazy/strict`, and some oddball `git-rerere` type packfile extension which lets you map `sha1:ff1234... => sha256:abc123456789...` "transparently" rather than the horrific situation you're laying out (correctly!) that forks the ecosystem in to "longhash" and "shorthash" when most repos don't even care in the end.

            1. kazinator · · focus · HN ↗
              > woefully confused

              Yes, I didn't understand that the GPG signing just operates on the top level object in the commit and trusts the SHA-1 hashes contained in it.

              The signing process doesn't recursively traverse the bytes of the commit to pull them into GPG, like you would expect.

              It's like, imagine you made a "bill of materials" of your project's files consisting of their names and CRC-32 checksums, and then signed this file, and called your project securely signed, LOL.

              This aspect can be fixed without foisting new hashing scheme into the content tracker. In fact, it must be fixed; users on SHA-1-based repos deserve secure signing.

              It's really sneaky that the SHA-1 business (not intended to be a security mechanism) was embroiled into the signing implementation; that GPG is demoted to the strength of SHA-1.

              Was that just to save some cycles? It's certainly faster just to sign the commit object!

              1. singpolyma3 · · focus · HN ↗
                signatures are basically always computed over hashes. The only problem here is that the hashes are not secure. And this is being fixed.
                1. kazinator · · focus · HN ↗
                  No, but, the hashes are features of the content tracking system that hook it together. There is no reason that a signing scheme must rely on and trust those hashes!

                  We can round up the bits that make up a commit in a SHA-1-based repo, and sign those bits securely; this is a thing that is possible.

                  1. dwohnitmok · · focus · HN ↗
                    > We can round up the bits that make up a commit in a SHA-1-based repo, and sign those bits securely; this is a thing that is possible.

                    Yes but as my other comment explains, this is not particularly useful in and of itself.

                    1. kazinator · · focus · HN ↗
                      [delayed]
                      1. ramses0 · · focus · HN ↗
                        Remember: bow out gracefully

                        you're arguing a position which indicates you don't know git's physical (textual) commmit structure.

                        Spend some quality time with:

                            git cat-file -p HEAD
                        You'll get something like:

                            tree ff1234...
    parent ccddef...

    fix(BUG-1245): my ai fixed it
                        Continue to `cat-file -p $TREE` and you'll get:

                            file efef12... Makefile
    tree a1b2c3... src/
    file f0ea12... README.md
                        ...and then it's turtles all the way down. It is (was!) safe to sign $HEAD (and only head!) because... it's turtles all the way down. Signing HEAD attaches IDENTITY (eg; torvalds@linux.com) to CONTENT (eg: src/@a1b2c3...) and TRANSITIVELY all the way down.

                        Your homework is to go run:

                            time (
      git ls-files | xargs -n 1 sha256sum
    ) | sha256sum
                        ...and then make a commit (nee: tag/note) containing that content as the commit message.

                        Your INSTINCT isn't wrong, your mechanics run counter to the practicalities of how git is designed to work, and the practicalities of the crucial defense that git-core is trying to divert: the ability to arbitrarily alter the signed(!!!) past, signed by third parties, with $ELDRITCH_SHA1 attacks.

                        You _still_ have to trust that GitHub.com or kernel.org won't get popped and start erroneously serving "signed but faulty" files and trees, but the urgency of moving to sha256 is about preventing faulty commits in the first place, which removes the requirement of "trust me bro!" relationship with the serving provider (or MITM).

                        1. kazinator · · focus · HN ↗
                          [delayed]
                  2. Dylan16807 · · focus · HN ↗
                    > There is no reason that a signing scheme must rely on and trust those hashes!

                    Not "must", but it would be stupid to use two sets of hashes without a compelling reason.

                    1. kazinator · · focus · HN ↗
                      [delayed]
                      1. lxgr · · focus · HN ↗
                        You are again talking about a fictional alternate reality in which git commit signatures do something like GPG_Sign(GPG_Signature_Hash(git commit hash || all objects referenced by commit))), instead of ours where it's GPG_Sign(GPG_Signature_Hash(git commit hash))), and the git commit hash is in turn a git hash of a bunch of objects.

                        In that reality, the git hash is load bearing. You can disagree with that design choice, but you can't pretend to live in that alternate reality and design your solutions for this reality according to that.

                        1. kazinator · · focus · HN ↗
                          > You are again talking about a fictional alternate reality

                          > can't pretend to live in that alternate reality

                          Discussions about solutions that don't exist or requirements not implemented are everyday occurrences and necessary.

                          I mean, if you're talking with a contractor about how your bathroom should look, that is a "fictional alternate reality", but you're not pretending to be living in it now.

                          I don't understand the purpose of the above engagement style, but it doesn't look like a great fit for HackerNews.

                          1. lxgr · · focus · HN ↗
                            There's a difference between talking about a hypothetical path forward and a hypothetical different past. The latter can be intellectually interesting but is rarely part of normal project planning.
                            1. kazinator · · focus · HN ↗
                              [delayed]
                      2. Dylan16807 · · focus · HN ↗
                        As far as git is concerned, GPG is a black box and git feeds it normal git hashes, of which there is only one kind. This design makes sense.

                        For anything else, it depends on the exact scheme:

                        If git sent GPG the bits for the current commit specifically because it expects a different hash to be used, that would be silly and wouldn't protect history.

                        If git rounded up the bits of history, that would perform unacceptably badly.

                        If git used a better hash to secure history for signing purposes, that would be very silly to design on purpose because it should use that hash for everything.

                  3. fc417fc802 · · focus · HN ↗
                    But is "commit" in this context a full snapshot or a delta? Because what you describe will only work for the latter.

                    Regardless, commit hashes should be a security mechanism IMO. And not just commits. I should be able to treat _any_ content addressing system as having secure addresses. If you can engineer collisions you need to patch your system.

                    (Note that the above does not necessarily imply support for unconditionally forcing a fork of the entire git ecosystem.)

                  4. lxgr · · focus · HN ↗
                    > There is no reason that a signing scheme must rely on and trust those hashes!

                    Yes, but git's signing scheme(s) do, as do many third-party ones, so what are you arguing for, exactly?

                    An alternate reality in which nobody uses git as documented? One in which git launched with a big disclaimer of "never trust our cryptographically secure hash function to be cryptographically secure" in its documentation and CLI outputs?

                    1. kazinator · · focus · HN ↗
                      Improving a signing scheme in the same repository format is less disruptive. It may not hit all the checkboxes, but it seems worth doing.
              2. dwohnitmok · · focus · HN ↗
                > The signing process doesn't recursively traverse the bytes of the commit to pull them into GPG, like you would expect.

                No I wouldn't expect this. If by recursive you mean traversing the entirety of git history, that would be prohibitively expensive performance-wise (imagine rehashing the entire multi-gigabyte history of the Linux kernel every time to sign and verify a commit) and destroy git functionality such as shallow clones and blob-less clones.

                If you by recursive you are only referring to the current working directory, as I lay out here <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=49930048">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=49930048 it doesn&#x27;t work. Indeed, without attestation of parent commits, a malicious attacker can actively frame any pre-existing vulnerability as someone else&#x27;s handiwork by simply inserting a new commit that purports to be the parent of another commit that does not contain the vulnerability, which then makes the original commit look like the source of the vulnerability.

                &quot;I didn&#x27;t introduce the vulnerability, he did! Look I can even prove it with my signed commit!&quot;

                1. dwohnitmok · · focus · HN ↗
                  &gt;&quot;I didn&#x27;t introduce the vulnerability, he did! Look I can even prove it with my signed commit!&quot;

                  I misspoke. It&#x27;s worse. &quot;I didn&#x27;t introduce the vulnerability, he did! Look I can even prove it with his signed commit!&quot;

                  1. kazinator · · focus · HN ↗
                    [delayed]
                  2. kazinator · · focus · HN ↗
                    [delayed]
              3. lxgr · · focus · HN ↗
                &gt; It&#x27;s like, imagine you made a &quot;bill of materials&quot; of your project&#x27;s files consisting of their names and CRC-32 checksums, and then signed this file, and called your project securely signed, LOL.

                If you are arguing that CRC-32 is a cryptographically secure hash function you might.

                SHA-1 is (or rather, used to be) one, so you actually can. This is literally how git works.

                &gt; It&#x27;s really sneaky that the SHA-1 business (not intended to be a security mechanism) was embroiled into the signing implementation; that GPG is demoted to the strength of SHA-1.

                What&#x27;s sneaky about it? It&#x27;s a completely reasonable design decision, making git orders of magnitude more efficient than the counterfactual you&#x27;re arguing for.

                &gt; Was that just to save some cycles? It&#x27;s certainly faster just to sign the commit object!

                Sure, &quot;just&quot; read potentially gigabytes of data, potentially over the network, and send them through your cryptographic hash function as opposed to just the objects you&#x27;re touching in your commit. Absolutely the same effort.

            2. hedora · · focus · HN ↗
              Also, note that you can just sign &quot;hello world&quot; and &quot;hello &amp;&amp; rm -rf&quot;, then serve whichever you want. In the absence of a hash, then everyone has to trust you not to do that. With SHA-256 git, and a one-time audit of the code, they would need neither your public key nor to trust that the code didn&#x27;t change.
          3. hinkley · · focus · HN ↗
            You do understand that most code signing is just PKI over the top of SHA hashes, right?

            The only substantial difference is that the author vouched for these particular snapshots of code, in a way where nobody else can intercept the communication and substitute a completely different hash than one the author previously signed.

            In fact if you can find a SHA collision, you can peal the signature off of the legitimate payload and slap it on the colliding one.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.