‹ BackHN Continuity

Thread

Telstra outage: The night a network decided the year was 2006

93 points · 34 comments · TMWNN

  1. speedgoose · · focus · HN ↗
    Human made summary: A telecom company had time synchronisation issues in a location of their infrastructure. They eventually used a single hardware GPS clock that fixed the issues but became their only authoritative clock in this location.

    One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.

    Apparently a GPS clock can keep track of the time if it’s up and running when the week counter overflows, otherwise it has to take a wild guess. Apparently their hardware GPS clock used a hardcoded start time from its firmware instead of trusting a less reliable existing clock.

    The article finishes with some AI looking suggestions to prevent such an issue to happen again.

    Mine would be to have bought one or two more GPS clocks and not from the same provider.

    1. skissane · · focus · HN ↗
      > One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.

      As I pointed out in my other comment below, this is only true of the old L1 signal, not the newer L2C signal. If they had a newer GPS card, this would never have happened.

      > Mine would be to have bought one or two more GPS clocks and not from the same provider.

      I think it would be more important to have a newer one that doesn't have this problem, than two old ones which both do.

      1. echoangle · · focus · HN ↗
        I don't think every GPS receiver necessarily has to have this problem, even with the old signal. It's not crazy to keep a week count in persistent memory and use that as a lower bound on startup so you only get a wrong time if the receiver isn't online for 1024 weeks.
        1. myself248 · · focus · HN ↗
          Exactly. The sane ones just ask you what year it is at startup, and use that to determine the initial epoch. Then they handle rollover automatically since then.

          It's been widely opined that 1024 weeks (~20 years) is the worst possible interval. Either rollovers should've happened VERY frequently (say, 128 weeks) so receivers would be FORCED to deal with it, or extremely infrequently (16384 weeks?), so it's simply never an issue.

          The unhappy medium is long enough that developers feel justified in saying "naaaah, our receiver won't still be in use then, we can ignore that!", but in practice it's very likely to happen.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.