Thread

GitLab Outage

69 points · 36 comments · kunley

Loading the complete thread in the background. This saved snapshot is available now. Refresh

nabeards · 1790291604 · focus · HN ↗
Just went to push code and found this out.
1. nicholasjarnold · 1790291714 · focus · HN ↗
  Code review for me... Hit the 503. The status page lagged the event by ~10 mins or so it seems.
  git+ssh also seems affected, as I'm unable to push/pull right now.
b0afc375b5 · 1790291912 · focus · HN ↗
Interesting that the status page was updated immediately. Is this the norm now? I remember back in the day when status pages showed all green even when servers were clearly down for a long time already.
1. thesh4d0w · 1790292058 · focus · HN ↗
  It took at least 5 minutes, closer to 10. I think the outage is so bad, it got pushed fast.
2. stack_framer · 1790292119 · focus · HN ↗
  It was not updated immediately. I was seeing 503s for a solid 15 minutes while the status page was all green. This is also their second outage today:
  <a href="https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7" rel="nofollow">https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
  1. nicholasjarnold · 1790292189 · focus · HN ↗
    Yep, status page was certainly all green when I started noticing it.
3. seabrookmx · 1790292295 · focus · HN ↗
  As others mentioned, it wasn't that quick.
  It's also not accurate. After nearly a half hour the page shows "git operations" as green but I'm not able to push/pull from my repos with SSH or HTTPS (and haven't been able to the whole time).
4. kunley · 1790334647 · focus · HN ↗
  It was updated at least 5 minutes after I noticed the outage first. Guess it's typical organisation lag
smalltorch · 1790291959 · focus · HN ↗
Gasp! immediately builds a complete git alternative to never expirence a single second of downtime ever again
1. JonChesterfield · 1790295005 · focus · HN ↗
  Well, git is already distributed, so you could use _any other computer_ instead of gitlab.com...
  1. smalltorch · 1790296134 · focus · HN ↗
    I'm only kidding, although it could be a good excersise to learn. I think a pure TUI git front end / back end project would be fun to make.
  2. gblosser · 1790298285 · focus · HN ↗
    or go back to Mercurial...
    Do you remember Mercurial?
    Pepperidge Farm remembers.
    mitxela · 1790301778 · focus · HN ↗
    Isn't it just git but harder to use?
davb · 1790292483 · focus · HN ↗
Coincidentally (?) we’ve seen a huge uptick in failed login attempts to our MFA-protected admin accounts on our self-hosted Gitlab CE instance today. They’re coming from thousands of IPs across hundreds of networks (VPS providers and, apparently, residential proxies).
1. deckar01 · 1790292737 · focus · HN ↗
  There was a relative lull in GitHub bot activity the last few weeks, but they seem to be back today. I assume the promo token flood gates opened for the new models.
  1. physicallyIllfr · 1790295146 · focus · HN ↗
    I dont think bots need bonus/trials. There are lots and lots of ways to get free, or virtually free tokens. Hell half the vibe coded AI chrome extensions send store their openrouter keys client side.
2. TZubiri · 1790298301 · focus · HN ↗
  It's probably related to the issue published by aikidolabs where leaked per-user gitlab issued emails were being used as a sort of credential.
  <a href="https://www.aikido.dev/blog/gitlab-email-push-to-main" rel="nofollow">https://www.aikido.dev/blog/gitlab-email-push-to-main
lemonish97 · 1790296292 · focus · HN ↗
Right, on to another alternative now
1. Towaway69 · 1790341473 · focus · HN ↗
  [delayed]
delf · 1790296679 · focus · HN ↗
You can use GitSocial to mirror to your own bucket for situations like this: `gitsocial mirror <a href="https://gitlab.com/owner/repo" rel="nofollow">https://gitlab.com/owner/repo s3://s3.example.com/mybucket/repo`
<a href="https://gitsocial.org/" rel="nofollow">https://gitsocial.org/
1. PhilippGille · 1790333440 · focus · HN ↗
  It would be nice to disclose that you're the creator of the tool which you are promoting here.
  1. delf · 1790344246 · focus · HN ↗
    Will do it going forward. Thank you for the feedback!
lampcord · 1790296839 · focus · HN ↗
Another day, another reminder why we self-host our GitLab instance. Cloud convenience always has these kinds of trade-offs.
1. zer0x4d · 1790297225 · focus · HN ↗
  We self host ours too. 100% uptime ;)
  1. TZubiri · 1790298811 · focus · HN ↗
    what host are you using that gets 100%uptime? A hyperscaler like AWS?
    zer0x4d · 1790300243 · focus · HN ↗
    I run it on Digital Ocean but AWS should be fine too.
    mitxela · 1790301755 · focus · HN ↗
    Just means it hasn't crashed yet.
    firesteelrain · 1790301839 · focus · HN ↗
    VMWare Cloud Foundation with Dell R7725s
  2. gib444 · 1790318169 · focus · HN ↗
    [delayed]
ChrisArchitect · 1790298913 · focus · HN ↗
Incident link: <a href="https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6ab5ac13c927d205640922b7" rel="nofollow">https://status.gitlab.com/pages/incident/5b36dc6502d06804c08...
hk1337 · 1790300014 · focus · HN ↗
I like what tangled seems like it could be but it's still too new to really use. It'd be nice if it was more of an aggregator for git servers where still access a git server whether tangled is up or not.
1. mitxela · 1790301735 · focus · HN ↗
  You don't really need one of these systems. You can just host git.
  1. hk1337 · 1790323471 · focus · HN ↗
    True, my thought was a site that just makes it easier to see all the various users. Like GitHub but links to users and orgs actually go to each person’s server.
tibbon · 1790300550 · focus · HN ↗
Seems they can compete with GitHub after all! /s
1. Towaway69 · 1790341558 · focus · HN ↗
  [delayed]
theplumber · 1790324708 · focus · HN ↗
Maybe Sam or Dario is playing hide and seek again….

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.