Thread

GitHub Actions and Pages are experiencing degraded availability

140 points · 69 comments · Footkerchief

KyleTheDev · 1786031990 · focus · HN ↗
Incident number 6 for the month of August, notably on August 6th. This pattern does not bode well.
1. paularmstrong · 1786034352 · focus · HN ↗
  Four of them were regarding AI/Copilot. So that's really just one long running incident and will probably continue to be 50/50 chance of being down.
Topfi · 1786032040 · focus · HN ↗
Very frustrating, nothing more I can add at this point.
dlcarrier · 1786032143 · focus · HN ↗
So, it's a normal Thursday?
presbyterian · 1786033263 · focus · HN ↗
The speed at which GitHub is tanking their reputation is fascinating
Insanity · 1786033408 · focus · HN ↗
Soon we can start posting "GitHub {x} is up!" instead, that's becoming more news-worthy lol. I know they like to blame AI/scaling, but really they are owned by one of the main hyperscalers, so I find that excuse kind of weak..
1. [deleted] · 1786033747 · focus · HN ↗
  [deleted]
newtonianrules · 1786033707 · focus · HN ↗
Did Microsoft fire all the people working on the GitHub features people actually use?
1. Topfi · 1786033971 · focus · HN ↗
  I mean, Github doesn't have its own CEO anymore, so arguably.
  1. sharts · 1786041735 · focus · HN ↗
    It sucked with a CEO as well
2. WorldMaker · 1786043499 · focus · HN ↗
  You don't think GitHub working on seven different interfaces to copilot and vibe coding PR page changes filled with bugs are a good use of engineering time?~
h2aichat · 1786033853 · focus · HN ↗
I am having their issues, also... Anyway this guys are great!
homeonthemtn · 1786033998 · focus · HN ↗
Ah. So that's why.
dinny · 1786034169 · focus · HN ↗
man.. again?
peterldowns · 1786034590 · focus · HN ↗
Can't wait for the Graphite/Cursor alternative (Origin) to become available. Actions being down for an hour in the middle of the US East workday is crazy. It's super frustrating to be an enterprise customer of Github, seems like we don't get any dedicated resources or additional stability. At this point our only option is self-host github enterprise or move away.
1. stephenway · 1786040660 · focus · HN ↗
  Would you move hosting/CI before Origin is available, or are you basically stuck on GitHub until then?
denysvitali · 1786034661 · focus · HN ↗
Was? Still is
whateveracct · 1786034983 · focus · HN ↗
any 9s left?
1. theamk · 1786035426 · focus · HN ↗
  there is one 9 of uptime, but the dashboard hasn't been updated yet: <a href="https://mrshu.github.io/github-statuses/" rel="nofollow">https://mrshu.github.io/github-statuses/
  Once this goes in, I'd expect to see 89%, which is zero nines. (I'd like to say, "a new low!", but sadly we've had this before)
2. ch4s3 · 1786035462 · focus · HN ↗
  They're running 9 8s of uptime.
3. pbkompasz · 1786036613 · focus · HN ↗
  nein nein nein
andrewmcwatters · 1786035116 · focus · HN ↗
[dead]
dboreham · 1786035502 · focus · HN ↗
Someone needs to work on improving the way Actions deals with resource exhaustion like this. Today it exploded all the CI jobs with "failed" and an inscrutable message about "The job was not acquired by Runner of type hosted even after multiple attempts" then "Internal server error". It should say "Timed out due to no runner available, our apologies, try again later, or pay $$$ here to get priority for runners".
1. [deleted] · 1786035633 · focus · HN ↗
  [deleted]
arandomhuman · 1786035573 · focus · HN ↗
I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
1. x3n0ph3n3 · 1786036795 · focus · HN ↗
  Microsoft is not a serious company anymore.
  1. hinkley · 1786038908 · focus · HN ↗
    They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
    Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
    They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
    I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
  2. sharts · 1786041692 · focus · HN ↗
    One might question your perspective if you ever thought they were serious.
2. vaneri2007 · 1786036800 · focus · HN ↗
  Maybe switching to <a href="https://github.com/gitlabhq" rel="nofollow">https://github.com/gitlabhq :)
3. duped · 1786038068 · focus · HN ↗
  I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
  If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
  1. hinkley · 1786038416 · focus · HN ↗
    The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
    They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
  2. packetlost · 1786038629 · focus · HN ↗
    That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
    jacobgold · 1786039160 · focus · HN ↗
    It's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
    packetlost · 1786039395 · focus · HN ↗
    The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
    duped · 1786040947 · focus · HN ↗
    The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
    I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.
    As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
    A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
    packetlost · 1786042590 · focus · HN ↗
    > The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
    Sure, but the part that actually schedules where a 'job' gets run is based on a relatively simplistic tag system. Reading the yaml and plopping some job metadata into a queue-like system isn't where I would expect their issues to be, but at their scale I'm sure everything becomes fragile and inscrutable.
    > imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
    You don't need to. GitHub Actions runners, and most CI runners that I've interacted with appear to have a pull-based model where they ask for work that matches their declared tags/shape (usually platform/runtime/OS/etc.). This probably amounts to a database query, but who knows.
    > A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
    I highly doubt it's a simple message queue. My issue is git repos and their CI infrastructure have very low coupling to other repos or entities in most circumstances, at least conceptually, so parts of the system (ie. regions, shards, etc.) should be able to function even when others are down (ie. it shouldn't break for everyone). There's clearly centralization and coupling that isn't obvious from an outside perspective, which sorta tells me it's incidental.
4. inigyou · 1786038849 · focus · HN ↗
  GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
  This is no surprise given standard Microsoft operating procedure - <a href="https://news.ycombinator.com/item?id=47616242">https://news.ycombinator.com/item?id=47616242
  1. exochrono · 1786039159 · focus · HN ↗
    they walked that back for now at least - self-hosted gh runners are still free: <a href="https://docs.github.com/en/billing/concepts/product-billing/github-actions" rel="nofollow">https://docs.github.com/en/billing/concepts/product-billing/...
    inigyou · 1786039575 · focus · HN ↗
    Well, they're still implemented that way.
  2. arandomhuman · 1786042074 · focus · HN ↗
    Oh that is pretty gross, thanks for sharing. I was not aware of that process, that explains the cost they attempted to roll out for runners too a few months back.
5. cyanydeez · 1786038922 · focus · HN ↗
  gitlab-ce can run all the integration tests in docker et al.
  Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.
Arubis · 1786036096 · focus · HN ↗
Github Actions are rapidly becoming an argument against dogfooding into your critical path.
rvz · 1786036162 · focus · HN ↗
Do we need to say this again? GitHub appears that they are never going to improve. Just look at how GitHub spark has deprecated, now we have this.
After 6 years of this nonsense of "centralizing everything on GitHub", it is not a good idea at all.
You might as well self host like I said before [0].
[0] <a href="https://news.ycombinator.com/item?id=22868406">https://news.ycombinator.com/item?id=22868406
classictraffic · 1786036623 · focus · HN ↗
This has been preventing me from merging a PR for over an hour. I'm really tired of the constant outages GitHub has
ptx · 1786036874 · focus · HN ↗
Are there any other CI providers that offer free cloud-hosted Windows runners?
Or has Microsoft made sure (through Windows licensing terms and pricing) that it's not possible to compete with their own CI offering?
Edit: CircleCI seems to offer 750 minutes/month (whereas GitHub offers 1000 minutes/month).
1. inigyou · 1786038982 · focus · HN ↗
  You can always pay for it, use your own machine, use a VM on your own machine, or if you are developing non-slop FOSS, ask Codeberg.
  1. ptx · 1786041775 · focus · HN ↗
    Does Codeberg support Windows runners at all? Forgejo only seems to have unofficial third-party builds of the Forgejo Actions runner for Windows.
    inigyou · 1786042596 · focus · HN ↗
    Good question. Even if it turns out they don't support it, I imagine they wish to support it.
opwizardx · 1786036937 · focus · HN ↗
How did they get Github Pages into “degraded performance”? Aren’t those just CDN stored static pages?
1. classictraffic · 1786037261 · focus · HN ↗
  I believe Pages uses Actions under the hood for publishing, so in my head it sort of makes sense for these things to be affected in the same outage
  1. wulfmann · 1786039135 · focus · HN ↗
    Well hopefully reads aren't affected then...
  2. opwizardx · 1786039403 · focus · HN ↗
    You are right, didn’t know that.
    Seems like the only reliable way to run GHA jobs is to not use their runners. Hope they at least didn’t break self-hosted runners operations
2. jubilanti · 1786040277 · focus · HN ↗
  Already built pages are still being served, but I've been waiting for 3 hours for an action to rebuild a single fix to a broken link.
fir3pho3nixx · 1786038253 · focus · HN ↗
Ever since Microsoft has taken this over are we even surprised by the downtime anymore? Piss up ... brewery ...
fir3pho3nixx · 1786038320 · focus · HN ↗
Are we even surprised anymore? Since microsoft (the IBM of its time) have taken this over all I can say is ... piss up ... brewery.
player_piano · 1786039180 · focus · HN ↗
So, Github Actions is currently at two nines of uptime?
1. steve-atx-7600 · 1786039399 · focus · HN ↗
  are you sure its actually two nines and not zero?
2. PhilippGille · 1786039930 · focus · HN ↗
  98.33 according to <a href="https://mrshu.github.io/github-statuses/" rel="nofollow">https://mrshu.github.io/github-statuses/
steve-atx-7600 · 1786039313 · focus · HN ↗
WTF. Outage every month if not multiple. Even for paying enterprise plans.
Also, doesn't even have RAG offering.
homeonthemtn · 1786039332 · focus · HN ↗
Still twiddling thumbs over here
baggachipz · 1786039659 · focus · HN ↗
There's a reason I'm browsing HN right now. Dead in the water.
1. hinkley · 1786040465 · focus · HN ↗
  There are three other things I could be doing instead, but I noticed it was down when I pushed the last of my GH todo list for the day and my brain is stuck in closure-seeking.
  I should eat lunch.
timetraveller26 · 1786039795 · focus · HN ↗
If anybody is looking for alternatives, self host Woodpecker is really easy and it's a very capable CI/CD <a href="https://woodpecker-ci.org/docs/intro" rel="nofollow">https://woodpecker-ci.org/docs/intro
axod · 1786039917 · focus · HN ↗
Broken for almost 4 hours.
niwtsol · 1786039971 · focus · HN ↗
Nothing like the old "everything is normal" to only be corrected 8 minutes later it is actually still down and then 2 hours later they are still working on it.
Aug 06, 2026 - 16:27 UTC - Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 06, 2026 - 16:19 UTC - Update - Pages is operating normally.
rwz · 1786040339 · focus · HN ↗
[dead]
alamsterdam · 1786040754 · focus · HN ↗
It's been hours :(
I have sympathy for the on-call team trying to resolve it, most of us have been there done that.
But seems something is systematically going wrong at GH
1. yashap · 1786041018 · focus · HN ↗
  It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd.
  Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.
  1. alamsterdam · 1786041264 · focus · HN ↗
    Mon and Dad (MS) are (generally) pretty solid with uptime.
    What is happening at GH?
    Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?
    yashap · 1786041466 · focus · HN ↗
    Yeah who knows, would be interesting to hear an inside take if any readers here are also GH devs!
    They're at 93.91% uptime over the past 90 days, according to <a href="https://mrshu.github.io/github-statuses/" rel="nofollow">https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.
    A glorious one nine of reliability.
    marcprux · 1786043772 · focus · HN ↗
    [delayed]
2. delduca · 1786042850 · focus · HN ↗
  > But seems something is systematically going wrong at GH
  Yes, we call it: Microslop.
canadiantim · 1786040991 · focus · HN ↗
Ah that checks out, was wondering why my PR's were piling up.
jjice · 1786041366 · focus · HN ↗
On other occasions, I'd take this as a time to have a walk because I'm blocked. Unfortunately, I need to get some stuff out for a customer quickly. Can't do anything about that though. GitHub is the only product in it's size class that I use that has this kind of incredibly poor uptime.
I'd love to know what the most common root causes for these outages are.
1. throwaway613746 · 1786041599 · focus · HN ↗
  [dead]
2. sharts · 1786041942 · focus · HN ↗
  They are known are engineers pushing features that bloated product management folks keep pushing so they can justify their jobs on LinkedIn.
  I'm not sure why this particular industry is so abysmal at making things even semi-reliable after decades of research, educated workforces, and loads of cash.
  1. rochak · 1786043765 · focus · HN ↗
    Corporate greed and late stage capitalism. Welcome to the future. We know you’ll love it.
[deleted] · 1786042140 · focus · HN ↗
[deleted]
[deleted] · 1786042260 · focus · HN ↗
[deleted]
[deleted] · 1786042289 · focus · HN ↗
[deleted]
earthpyy · 1786043780 · focus · HN ↗
That's why my CI on main branch didn't triggered.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.