‹ BackHN Continuity

Thread

Salesforce Global Outage

280 points · 184 comments · mabil

  1. mergy · · focus · HN ↗
    Unplanned outage timing is never good but this is really not good.

    <a href="https:&#x2F;&#x2F;www.salesforce.com&#x2F;dreamforce&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.salesforce.com&#x2F;dreamforce&#x2F;

    Sept 15-17

    1. ramesh31 · · focus · HN ↗
      Probably not a coincidence
      1. dehrmann · · focus · HN ↗
        Most places at this scale have code freezes in place well before conferences. The most likely issues are some launch couldn&#x27;t handle the scale or periodic deployments have been saving them from some sort of long-standing leak bug, and pausing going into Dreamforce meant some service hasn&#x27;t been restarted in a week. Historically, Salesforce sharded by customer, so that goes against both of these, unless it&#x27;s in a routing layer.
        1. SaucyWrong · · focus · HN ↗
          I&#x27;m an ex-Salesforce, and yes, at the time I left, there was a huge change freeze surrounding Dreamforce. Unless a demo of an announced feature was coming in really at the buzzer, change velocity would have been low since a few weeks ago. I worked in a sub-cloud, so I can&#x27;t even speculate as to the reason for the failure.

          Something I wonder about is whether SRE responses were delayed due to having to be emergency-change-approved because Dreamforce was on. I don&#x27;t recall a global outage ever occurring during a change freeze when I worked there, so &#x2F;shrug.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.