Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it.
> We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.
At least it didn't fix the problem so they can actually start finding the real cause.
> We're no longer pursuing restarts as a path to remediation.
Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the proposed clumsy fix (it could make things worse).
Rebooting is the first step: If it fixes it you don't have a problem. If it doesn't, you know more about the problem.
It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again?
The requirement is to get customers out of impact as #1 priority. If there's suspicions around memory/thread states a restart makes a lot of sense. Digging through logs and flight records takes a lot of time, customers are losing business in that time. If you're afraid to restart your service you need to work on your telemetry.
raffraffraff · · focus · HN ↗
> We're no longer pursuing restarts as a path to remediation.
Oh you have
cube00 · · focus · HN ↗
> We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.
At least it didn't fix the problem so they can actually start finding the real cause.
> We're no longer pursuing restarts as a path to remediation.
Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
b112 · · focus · HN ↗
So hopefully it's not done often.
CoffeeOnWrite · · focus · HN ↗
b112 · · focus · HN ↗
ocdtrekkie · · focus · HN ↗
It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again?
Rebooting it a second time fixed it.
wookmaster · · focus · HN ↗