Thread

1. cube00 · 1790017251 · focus · HN ↗

   > When it went to flip into the backup, we discovered that the backup fiber had a break

   Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

   I wonder how long it was down? Days, weeks, months?

   1. toast0 · 1790018021 · focus · HN ↗

      Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.

      1. koolba · 1790018279 · focus · HN ↗

         Put all your backups in one basket, and then pray that nobody crushes the basket.

         1. oasisbob · 1790019180 · focus · HN ↗

            This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.

            1. jacquesm · 1790026533 · focus · HN ↗

               In NL this often happens because of waterways. All starts off with good intentions, multiple providers, totally different fibers exiting the building in different directions. And then they have to cross a canal or a river. It's a 50/50 at that point whether they converge on the same conduit across the water.
            2. hdgvhicv · 1790038789 · focus · HN ↗

               I find intonation it’s better to select two competing provides as they are less likely to use the same ducts.

               Had a few issues in some lamdlpcked counties on occasion - Kabul had all traffic routing via the same route into Pakistan for a few years, Kathmandu had an earthquake knock out both ISPs within 30 seconds of each other.

               Backups via starlink global roaming work well now in particularly challenging environments.

Unofficial Hacker News client; not affiliated with Y Combinator.