Thread

You can run Git on object storage if you re-make packfiles

145 points · 35 comments · evacchi

Loading the complete thread in the background. This saved snapshot is available now. Refresh

mmastrac · 1789796247 · focus · HN ↗
Nice. I saw some git-on-durable-objects projects but I'd so much prefer a straight object storage option.
mgrandl · 1789798246 · focus · HN ↗
Sweet. It’s kinda wild to me that this hasn’t been done. Would be so much nicer if gitlab/forgejo/gitea/… supported object storage for everything. Currently you always still need a filesystem for git repos.
1. throwaway7356 · 1789806043 · focus · HN ↗
  It has been done. It is no longer done because it is not efficient.
  Yes, it was not "object storage", but Git can be served from a dumb http server as static files. Now someone figured out that object storage can also serve static files via http. Wow!
2. eru · 1789808283 · focus · HN ↗
  On Linux, Fuse is your friend. See eg <a href="https://github.com/matthiasgoergens/git-snap-fs" rel="nofollow">https://github.com/matthiasgoergens/git-snap-fs
3. baalimago · 1789808545 · focus · HN ↗
  > It’s kinda wild to me that this hasn’t been done
  <a href="https://github.com/awslabs/git-remote-s3" rel="nofollow">https://github.com/awslabs/git-remote-s3
  This works seamlessly so you can run `git remote add origin s3://my-git-bucket/my-repo`
4. everforward · 1789836434 · focus · HN ↗
  Doesn’t Git store data in sparse files? My impression is that git’s on-disk layout is exactly the kind of thing you’d avoid on object storage. Interactions involve a ton of small reads, resulting in poor performance and enormous bills.
  It only really makes sense to tarball repos into cold storage on S3 or Glacier, but short of GitLab cloud I suspect there aren’t enough repos cold enough to be worth the dev costs.
  S3 that doesn’t have wild costs for git objects sounds like it’s just NFS
  1. masklinn · 1789842683 · focus · HN ↗
    Git does not use sparse files no, however in packs objects are delta-ified: any object can be either literal or a combination of literal segments and range copies from a base object, and that base object can be the same, recursively. So to resolve any object you need to jump around a fair bit in the pack file. The normal GC creates delta chain of length 10, aggressive defaults to 50, the maximum is 4095.
5. shados · 1789859241 · focus · HN ↗
  There's a bunch. I got pissed during one of the last GitHub outages and had an AI agent one shot a clone with PR, stacks, actions, and full compatibility with the git client running on Cloudflare workers and R2. Only limitations is it was https only because CF workers don't have tcp to implement and without an enterprise account.
  Works great to this day.
jauntywundrkind · 1789798590 · focus · HN ↗
After Git At Any Scale (<a href="https://news.ycombinator.com/item?id=49348141">https://news.ycombinator.com/item?id=49348141) there's been a lot of these!
<a href="https://github.com/Nickforall/walgit-elixir" rel="nofollow">https://github.com/Nickforall/walgit-elixir <a href="https://github.com/tobi/walgit" rel="nofollow">https://github.com/tobi/walgit (<a href="https://news.ycombinator.com/item?id=49420598">https://news.ycombinator.com/item?id=49420598) <a href="https://github.com/tuist/micelio/" rel="nofollow">https://github.com/tuist/micelio/
jiggawatts · 1789799677 · focus · HN ↗
Next: replace whole-file hashes with a Merkle Patricia tree hash and enable BitTorrent-like distribution protocols on top, not to mention the deduplication advantages!
1. notpushkin · 1789818358 · focus · HN ↗
  History repeats itself: <a href="https://github.com/cjb/GitTorrent" rel="nofollow">https://github.com/cjb/GitTorrent
2. JoshTriplett · 1789825888 · focus · HN ↗
  > replace whole-file hashes
  BLAKE3 has a whole mechanism for this, and as an added bonus it'd be substantially faster.
hanwenn · 1789801006 · focus · HN ↗
"Finally delta objects should be stored as their own object in the packfile instead of slapped onto the end of the object it’s a delta of so that you don’t have to read the object and its deltas to read the object in the first place."
Author thinks that delta encoding is there for no reason. Good luck, kid.
0bytes · 1789801485 · focus · HN ↗
4th submission in 3 days. Is this a marketing spam campaign?
1. gjvc · 1789804182 · focus · HN ↗
  it's a shit website with awful fonts, certainly
  1. jamietanna · 1789818640 · focus · HN ↗
    Which fonts are you seeing not looking good? It seems perfectly readable sans-serif and monospace wise, but could be system preferences on what fonts to use?
2. bryanrasmussen · 1789810622 · focus · HN ↗
  It seems like the kind of thing I would naturally post if I came across it, so a bunch of submissions over 3 days makes me think a bunch of people saw it and when they saw it said huh, I will submit to HN.
  I think that makes more sense than some spam campaign for this.
  1. notpushkin · 1789818173 · focus · HN ↗
    I’m wondering how this is possible. Usually when you submit something that was already posted recently, HN will upvote the existing post instead.
    bryanrasmussen · 1789828914 · focus · HN ↗
    it depends on how much interest was shown, I've definitely posted things and then they get posted again less than 24 hours later and it is that second post that takes off.
3. xena · 1789914373 · focus · HN ↗
  I'm the author and content lead at Tigris. My policy for the team is "do not self post to hacker news".
uptownhr · 1789805010 · focus · HN ↗
I created my own as well and powers agentgit.co
1. scoopertrooper · 1789806291 · focus · HN ↗
  That Claude written page is impenetrable. You really need to write yourself or use a better model.
andrewaylett · 1789810319 · focus · HN ↗
`git update-server-info` is your friend; it generates the auxiliary files needed to allow clients to make range requests against packs.
For example: <a href="https://andrewaylett.github.io/rgitweb/" rel="nofollow">https://andrewaylett.github.io/rgitweb/ is a repository viewer that runs against a static git repo. It doesn't care what you're storing the pack in, just that the data is there to make the right range requests.
flowingfocus · 1789811574 · focus · HN ↗
I recently found <a href="https://github.com/enroute-sh/enroute" rel="nofollow">https://github.com/enroute-sh/enroute which also does git on object storage
slightly off topic but I found the idea of a git-proxy interesting where you start with proxying an existing forge and then later adopt the object storage backend
1. podviaznikov · 1789855646 · focus · HN ↗
  yes! I tried to do exactly this here <a href="https://porcelain.sh/" rel="nofollow">https://porcelain.sh/
  never finished, because stopped having personal use case, but had exactly same idea(and a bit more features)
alizainf · 1789817144 · focus · HN ↗
You can also run git on postgres - <a href="https://github.com/usecontextlayer/pggit" rel="nofollow">https://github.com/usecontextlayer/pggit
1. NetOpWibby · 1789825429 · focus · HN ↗
  Oh this is exactly what I need, fantastic! Thanks for sharing.
delf · 1789819275 · focus · HN ↗
This is exactly what GitSocial does: <a href="https://gitsocial.org/" rel="nofollow">https://gitsocial.org/. Plus it can generate static sites for repositories, letting you serve it all from a bucket.
1. Difwif · 1789838675 · focus · HN ↗
  This seems incredible and I love the idea. Anyone using this for large projects? Any clear downsides or tradeoffs?
  1. delf · 1789839998 · focus · HN ↗
    Not that I know of as I don't have a way track this. I did test importing and mirroring everything for Ghostty and Meshtastic projects, found out that initial import and push steps may take a while.
    Great to have issues, PRs, etc in git, but requires each participant to set up a fork to participate.
    No CI though, but an extension to capture job logs is on the roadmap.
  2. adastra22 · 1789844699 · focus · HN ↗
    Radicule does something similar. Also git-bug.
pierrebarre · 1789827094 · focus · HN ↗
Or just use <a href="https://github.com/Barre/ZeroFS" rel="nofollow">https://github.com/Barre/ZeroFS
Works pretty awesomely with git workloads, even Linux sized trees.
1. jeremyjh · 1789841041 · focus · HN ↗
  Do you have any benchmark data you can share?
  What you are saying sounds like an assertion that the problem described here[1] - by Vicent Marti - that packfiles aren't contiguous blocks of data and so every fetch fans out to more fetches from the client - is not actually a problem? Or is there an innovation in zerofs that somehow solves this for git? From what I can see it offers NFS and NBD interfaces and GitHub tried and rejected both of those very early on. Probably they did this on a more traditional remote filesystem backed by SAN or local SSD but their problem was network chatter, not storage contention.
  The problem he described seems consistent with known patterns from datalake storage architectures, its a big part of why columnar storage is the only game in town for most use cases.
  This is why TFA has a lot of interest here - he has changed the storage layout of packfiles to a columnar model to address this challenge.
  [1] <a href="https://cursor.com/blog/git-at-any-scale#git-without-packfiles" rel="nofollow">https://cursor.com/blog/git-at-any-scale#git-without-packfil...
  1. pierrebarre · 1789843348 · focus · HN ↗
    [dead]
sisuo30390 · 1789830905 · focus · HN ↗
[dead]
colechristensen · 1789838857 · focus · HN ↗
I reimplemented git in elixir with additional backends, it's working in production for my own git code forge but definitely has some rough edges.
<a href="https://anvil.fangorn.io/fangorn/ex_git_objectstore" rel="nofollow">https://anvil.fangorn.io/fangorn/ex_git_objectstore
LoganDark · 1789862613 · focus · HN ↗
> (I’ll call this "dotgit" so I don’t have to write as many backticks)
And then "dotgit" was not seen ever again :)

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.