‹ BackHN Continuity

Thread

US halts flights at busy East Coast airports, says fiber line cut

239 points · 150 comments · allanbreyes

  1. cube00 · · focus · HN ↗
    > When it went to flip into the backup, we discovered that the backup fiber had a break

    Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

    I wonder how long it was down? Days, weeks, months?

    1. readthenotes1 · · focus · HN ↗
      A lot of people don't check their backups until they need to restore.

      A lot of people are incompetent.

      1. thewebguyd · · focus · HN ↗
        A backup without a restore test isn't a backup at all
        1. bonoboTP · · focus · HN ↗
          You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
          1. lovich · · focus · HN ↗
            No, even professional companies with critical data balk at this.

            I worked at a company worth a few billion and the leadership balked when they told engineering they wanted a near instantaneous failover system and our department informed them that would require paying for a second environment that could be rolled over to.

            It is rare to find leaders who can accept the cost of redundant infrastructure that is there for emergency backup.

            What puzzles me is why they can’t accept it when they are perfectly fine with insurance costs and I can’t see much of a difference between the two when looking at a spreadsheet of costs other than possibly tax differences between the type of expenditure.

            1. 486sx33 · · focus · HN ↗

              [dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.