On LinkedIn
This week the world’s timekeepers vote on ending the leap second for good, because it breaks software. On 25 October, 02:30 happens twice in Europe. Nobody votes on that.
The second was never the problem. We just treat time as a given.
I have lost count of the outages, incidents and even attacks where the clock made the investigation harder, or impossible. Wrong timezones on devices are the happy flow: once you spot them, which is a challenge in itself, you can convert. Drift is the nightmare. Moments refuse to line up, and suddenly you have no idea what is cause, what is effect, what is related and what is just noise.
At the peak of that frustration I met Ralph Braun at Wegener ICT/K. He took me deep, with real passion, into NTP, strata and clock drift. Into the elegant tools that compensated for it back when nobody had an atomic clock on tap. And into his frustration with people who were sloppy with the specs. I had been using all of it without a thought. After Ralph, with a lot more respect.
It matters more every year, because logging has exploded. IOPS tells you how many times per second a system reads or writes data. When I look at ours at ON2IT on a “quiet” moment, the numbers are obscene. At that volume the smallest deviation wrecks your fact-finding, your analysis and with it your ability to solve anything.
So keep the second. Earth needs time, and Earth never signed the RFC. If we cope with an hour that happens twice, we can cope with a second. Count from epoch, let NTP slew the clock a little faster or slower instead of jumping it, and you lose no events. What we can’t cope with is a clock nobody checks.
Take your last incident. Could you prove which event came first?
What’s the worst clock you ever had to untangle?
The sources: the Nature study by Zhang and Dumberry on how Earth’s core speeds up and slows down our days (via Gizmodo), and this week’s vote at the 28th CGPM in Versailles.
The part nobody talks about: Earth is spinning faster, so the next one could be a negative leap second. A second that gets skipped. That has never happened, so almost no code has ever been tested for it.
Five-minute check for Monday: is every device logging in UTC, and does anything alert you when a clock drifts?
(Ralph, if this ever reaches you: you were right about the specs.)