I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
aftbit · · focus · HN ↗
tregoning · · focus · HN ↗
sroussey · · focus · HN ↗
Also, a simple signal status server or queue system helps to keep global state such that everyone doesn’t retry all at once.
If you have a central error rate server you can skip your retry based on the error rate (100% error rate, don’t retry, etc).