Thread

1. cube00 · 1790017251 · focus · HN ↗

   > When it went to flip into the backup, we discovered that the backup fiber had a break

   Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

   I wonder how long it was down? Days, weeks, months?

   1. zoomablemind · 1790019009 · focus · HN ↗

      That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.

      1. pixl97 · 1790019357 · focus · HN ↗

         You actually have to ensure that someone isn't just checking boxes the tests passed and the tests actually passed.

         I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core.

         Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.

         1. lovich · 1790021864 · focus · HN ↗

            And auditing that is expensive and you always get given hell from the assholes going “why am I paying for this if nothing has gone wrong”

            The amount of shit I was given everytime I went through a checklist working at a fedramp certified company working with emergency alerts on the phone system made me prematurely gray.

            I could feel the barely contained seething rage everytime I told the execs that the reason this release will take 3 days and not be instantaneous like your friends releases at a faang and that information made you embarrassed at your dinner party, is because you agreed to this process contractually years ago and now it’s a crime if I just sign off on it being ok without actually checking that it’s ok.

            1. pixl97 · 1790023256 · focus · HN ↗

               Yea, fangers are def not used to working on life critical systems. I personally have avoided it as much as possible, but not completely, for all the reasons you listed above. No need for me to accept that much responsibility for other peoples lives when I've been able to earn a living just fine not doing so.

               1. hubbahubbahubba · 1790132006 · focus · HN ↗

                  It’s not just faang. Oracle software pre-y2k had disclaimers against being used in nuclear power plants.

Unofficial Hacker News client; not affiliated with Y Combinator.