‹ BackHN Continuity

Thread

How Uber Protects Against Retry Storms

120 points · 49 comments · iscmt

  1. prologic · · focus · HN ↗
    So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.

    This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)

    1. BobbyTables2 · · focus · HN ↗
      I had a former colleague who would put a “retry 10x with sleeps” in each of A, B, C, D.

      None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.

      1. prologic · · focus · HN ↗
        LOL
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.