Thread

1. cube00 · 1790017251 · focus · HN ↗

   > When it went to flip into the backup, we discovered that the backup fiber had a break

   Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

   I wonder how long it was down? Days, weeks, months?

   1. toast0 · 1790018021 · focus · HN ↗

      Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.

      1. unethical_ban · 1790020443 · focus · HN ↗

         That is negligence, practical if not contractual.

         Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.

         So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.

         1. Jtsummers · 1790020771 · focus · HN ↗

            > validates that they are geographically separate up until the connections

            This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.

            1. hdgvhicv · 1790038992 · focus · HN ↗

               Yes the probe here is a lack of monitoring.

               You can get two separate failures in short succession, where the first one hasn’t been fixed. For critical devices that’s why you have tertiary backups, ideally on different technology compel you. For one major event I had diverse fibre and satelite and microwave. Lost microwave and satelite to electronic warfare but the fibres were ok. Ofcom were really pissed about it, but nothing they could do.

               1. justsomehnguy · 1790047774 · focus · HN ↗

                  > Yes the probe here is a lack of monitoring.

                  No?

                  Like, okay, monitoring reports what there is no link.

                  Now what? Somebody gets a hi vis vest, a hard hat and goes along the cable to investigate the reason?

                  There is nothing monitoring could help, especially if the cable was cut recent enough.

                  1. iso1631 · 1790063413 · focus · HN ↗

                     If I lose a link anywhere in the world then BGP reroutes within a second or so, and we get a notification in seconds or maybe minutes on slack depending on how busy slack is being.

                     If it's down for more than 15 minutes in country or 2 hours internationally it gets flagged for manual attention. Shorter outages are logged but only looked at monthly as part of the operational report process.

                     Now sure you can have two independent faults on the same day, but the reports are that they didn't know the backup line was down.

                     1. justsomehnguy · 1790166017 · focus · HN ↗

                        Meh, I knew people would just clutch to the monitoring ignoring everything else.

                        In your case you already have way more than monitoring. You have the infrastructure designed for resilience. You have that design implemented. You have the protocols to do if the things go north. You have automation to disregard minor events and to bring to the attention more serious things. You have way more than monitoring alone.

Unofficial Hacker News client; not affiliated with Y Combinator.