‹ BackHN Continuity

Thread

Salesforce Global Outage

280 points · 184 comments · mabil

  1. stmw · · focus · HN ↗
    Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.
    1. Anon1096 · · focus · HN ↗
      Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.
      1. ocdtrekkie · · focus · HN ↗
        > which belies deep misunderstanding

        I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach.

        Smaller is, in fact, better, because it's easier to operate reliably.

        1. toomuchtodo · · focus · HN ↗
          Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale).

          "Simplicity is the ultimate sophistication." -- Da Vinci

          1. a_conservative · · focus · HN ↗
            Hidden in this discussion around self-hosting reliability are other options as well.

            Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted.

            Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something

            There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.

            1. toomuchtodo · · focus · HN ↗
              That was my intent, pull the thread.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.