Thanks for sharing that I'm very interested to read the auditor's results
Edit: That's it!? It's also not clear on whether it was a configuration or a firmware issue (or both). It sounds like maybe there was some battery backed ram or something that reset maybe?
Edit number 2: Ok I re-read it and it is better than I thought, still many questions though.
They kept the LCD dot matrix character display, the dual-color status LED, and the chassis. All of the other original guts remain in-place; it's completely reversible and no new holes were made.
I agree that rebuilding is an awkward fit here. Repurposing fits better in some ways and worse in others.
edit: Maybe retromod works. New stuff in an old box, with continuation of parts of the original vibe.
In the spirit of a 1990s time server is I also got Timelord + Tardis working (time service over AppleTalk, tested on a Mac SE/30, courtesy of netatalk) and tested multiple NTPv1, NTPv3, and NTPv4 clients through this (and soon an Xserve G5).
It's less about getting the 1995 hardware working, and more about rebuilding the actual time server to serve the original purpose, for the next 5-10 years at least.
I'm planning on using this chassis to build up an NTP Stratum 1 server for the global NTP pool, hopefully later this year. I thought it would be a lot more fun than letting the box go to e-waste.
And leaving the guts completely untouched means I can do a full restoration someday, if I want.
The two main things I'd need to do:
1. Modify the RF chain so I can use normal GPS antennas that don't down-convert the frequency.
2. Modify the DC bias voltage so it doesn't fry GPS antennas.
After that, I could work on patching firmware, but I'd probably just use the original guts as a frequency reference and leave time to other hardware.
I could even run the Pi and the old hardware simultaneously. The old hardware providing IRIG-B, 10 MHz, and 1 PPS references that are a little more accurate (it has a TCXO, haven't measured its timing accuracy yet). And the Pi could provide PTP and NTP, using a more modern GPS chip which even supports dual band and survey-in for more accuracy.
I'm spitballing a bit, but it looks like an absolutely-correct antenna for your clock might be a Symmetricom 140-615. There's a couple (literally two) on the ebay right now that look minty-fresh; they're not the cheapest things, but they're cheap compared to many other things in the low-volume RF world. :)
If that's the right antenna (just because I think it is doesn't mean that you should agree), then that would certainly simplify the RF side of the project. No hardware mods required at all. It takes 12v and does internal downconversion to the right frequencies.
After that, you only have the daunting task of patching firmware -- if it turns out (from testing) that it patching is even required. (IIRC, you speculated in the video that it might already be patched and I don't disagree with that outlook at all.)
Both legs of this thing represent interesting projects. I find that time is strangely rewarding to get exactly-right even when that doesn't really matter for the application at-hand. It's a good kind of weird.
(I really need to fire up my own little GPS-sourced Pi Zero NTP server some day soon. Chasing temperature-based clock deviations on that was actually kind of fun.)
I thought it was a fun video (the first I've watched of yours, though I read the blog). And I'd honestly forgotten about the Telstra outage even though it was news here
Yeah, I got recommended this video and felt very clickbaited and stopped watching when he mentioned that he's just replacing the innards with a Pi. Incredibly lame.
After the Telstra NTS issue and preparing for the upcoming summer storms and power outages, I set up a chrony mesh on three devices, each with a different upstream stratum 1 NTS. I seriously considered a GPS reference too, but I live on the bottom floor of apartment blocks so reception is probably quite poor. After blocking the firewall ports, the mesh stays linked and drift is well within sensible boundaries for the typical length of internet and power outages of a few hours. Although, a man with a watch knows the time but man with two watches cannot be certain.
I have a GPS based PTP server, the antenna is indoors 15 feet away from a south facing window, it locks on pretty well (though this will probably depend on the surrounding buildings)
It's surprising how often I can get an okay-ish GPS signal indoors... it seems to have a lot more to do with building materials than anything else.
Multipath reflections means the location (and thus timing accuracy) wanders a lot, but it's still great down to microseconds if you can get 5+ satellites in view.
This sounds pretty interesting - what are you working on where it's necessary to maintain this level of time precision? I've worked on positioning systems where it's obviously important, but it never mattered much at home
Nothing special - just a homelab. It's not so much the precision that matters at home as for planning when the home becomes an island, in 'orphan mode' as chrony calls it [1], without a time reference at all. Time really is a 'sovereign function' per Telstra's report [2], especially if you have internal HTTPS, etc.
Last summer an 8+ hour mains power outage drained my home CWWK OPNsense router's UPS then internal battery (it must have been losing voltage). When mains power came up the router was unable to connect to the ISP because the internal time was completely wrong and I had to manually set it via console. However, the internal batteries and clocks on other home network infrastructure (HP NAS and Lenovo Tiny) were fine.
The chrony.conf orphan mode:
server 192.168.1.1
server 192.168.1.2
local stratum X orphan (higher X determines who is master in orphan mode, usually minimum 10)
Annoyingly, the OPNsense chronyd plug-in does not (yet) implement the "local orphan" mode [3], so I have not yet confirmed that it does accept the non-reference master orphan time to bootstrap itself... should test before the summer storms start again.
If we're going to be stupid with it, I should write up an article on my ultra-precise 16MHz Arduino that runs at cock on 16000000Hz. The crystal is replaced by a GPS-disciplined VCO.
On Telstra, they brought in an external auditor and the report was recently release:
* Post: https://www.telstra.com.au/exchange/what-we-ve-learned-from-...
* Report: https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr...
* https://news.ycombinator.com/item?id=49549656
Thanks for sharing that I'm very interested to read the auditor's results
Edit: That's it!? It's also not clear on whether it was a configuration or a firmware issue (or both). It sounds like maybe there was some battery backed ram or something that reset maybe?
Edit number 2: Ok I re-read it and it is better than I thought, still many questions though.
The author is not rebuilding anything, they just stuck a Raspberry Pi in the case.
They kept the LCD dot matrix character display, the dual-color status LED, and the chassis. All of the other original guts remain in-place; it's completely reversible and no new holes were made.
I agree that rebuilding is an awkward fit here. Repurposing fits better in some ways and worse in others.
edit: Maybe retromod works. New stuff in an old box, with continuation of parts of the original vibe.
In the spirit of a 1990s time server is I also got Timelord + Tardis working (time service over AppleTalk, tested on a Mac SE/30, courtesy of netatalk) and tested multiple NTPv1, NTPv3, and NTPv4 clients through this (and soon an Xserve G5).
It's less about getting the 1995 hardware working, and more about rebuilding the actual time server to serve the original purpose, for the next 5-10 years at least.
I'm planning on using this chassis to build up an NTP Stratum 1 server for the global NTP pool, hopefully later this year. I thought it would be a lot more fun than letting the box go to e-waste.
And leaving the guts completely untouched means I can do a full restoration someday, if I want.
The two main things I'd need to do:
1. Modify the RF chain so I can use normal GPS antennas that don't down-convert the frequency.
2. Modify the DC bias voltage so it doesn't fry GPS antennas.
After that, I could work on patching firmware, but I'd probably just use the original guts as a frequency reference and leave time to other hardware.
I could even run the Pi and the old hardware simultaneously. The old hardware providing IRIG-B, 10 MHz, and 1 PPS references that are a little more accurate (it has a TCXO, haven't measured its timing accuracy yet). And the Pi could provide PTP and NTP, using a more modern GPS chip which even supports dual band and survey-in for more accuracy.
Or option C?
I'm spitballing a bit, but it looks like an absolutely-correct antenna for your clock might be a Symmetricom 140-615. There's a couple (literally two) on the ebay right now that look minty-fresh; they're not the cheapest things, but they're cheap compared to many other things in the low-volume RF world. :)
If that's the right antenna (just because I think it is doesn't mean that you should agree), then that would certainly simplify the RF side of the project. No hardware mods required at all. It takes 12v and does internal downconversion to the right frequencies.
After that, you only have the daunting task of patching firmware -- if it turns out (from testing) that it patching is even required. (IIRC, you speculated in the video that it might already be patched and I don't disagree with that outlook at all.)
Both legs of this thing represent interesting projects. I find that time is strangely rewarding to get exactly-right even when that doesn't really matter for the application at-hand. It's a good kind of weird.
(I really need to fire up my own little GPS-sourced Pi Zero NTP server some day soon. Chasing temperature-based clock deviations on that was actually kind of fun.)
I thought it was a fun video (the first I've watched of yours, though I read the blog). And I'd honestly forgotten about the Telstra outage even though it was news here
Yeah, I got recommended this video and felt very clickbaited and stopped watching when he mentioned that he's just replacing the innards with a Pi. Incredibly lame.
Kinda agreed. Except I continued watching because it was kind of ...pleasant?
Jeff: please do better! :)
After the Telstra NTS issue and preparing for the upcoming summer storms and power outages, I set up a chrony mesh on three devices, each with a different upstream stratum 1 NTS. I seriously considered a GPS reference too, but I live on the bottom floor of apartment blocks so reception is probably quite poor. After blocking the firewall ports, the mesh stays linked and drift is well within sensible boundaries for the typical length of internet and power outages of a few hours. Although, a man with a watch knows the time but man with two watches cannot be certain.
I have a GPS based PTP server, the antenna is indoors 15 feet away from a south facing window, it locks on pretty well (though this will probably depend on the surrounding buildings)
It's surprising how often I can get an okay-ish GPS signal indoors... it seems to have a lot more to do with building materials than anything else.
Multipath reflections means the location (and thus timing accuracy) wanders a lot, but it's still great down to microseconds if you can get 5+ satellites in view.
This sounds pretty interesting - what are you working on where it's necessary to maintain this level of time precision? I've worked on positioning systems where it's obviously important, but it never mattered much at home
Nothing special - just a homelab. It's not so much the precision that matters at home as for planning when the home becomes an island, in 'orphan mode' as chrony calls it [1], without a time reference at all. Time really is a 'sovereign function' per Telstra's report [2], especially if you have internal HTTPS, etc.
Last summer an 8+ hour mains power outage drained my home CWWK OPNsense router's UPS then internal battery (it must have been losing voltage). When mains power came up the router was unable to connect to the ISP because the internal time was completely wrong and I had to manually set it via console. However, the internal batteries and clocks on other home network infrastructure (HP NAS and Lenovo Tiny) were fine.
The chrony.conf orphan mode:
server 192.168.1.1
server 192.168.1.2
local stratum X orphan (higher X determines who is master in orphan mode, usually minimum 10)
Annoyingly, the OPNsense chronyd plug-in does not (yet) implement the "local orphan" mode [3], so I have not yet confirmed that it does accept the non-reference master orphan time to bootstrap itself... should test before the summer storms start again.
1. <https://chrony-project.org/doc/latest/chrony.conf.html>
2. <https://www.telstra.com.au/exchange/what-we-ve-learned-from-...>
3. <https://github.com/opnsense/plugins/issues/5115>
If we're going to be stupid with it, I should write up an article on my ultra-precise 16MHz Arduino that runs at cock on 16000000Hz. The crystal is replaced by a GPS-disciplined VCO.
Why?
Well, I wasn't doing anything else much fun...
This is not a great idea. Pre-2000 GPS time server firmwares do no handle leap seconds properly.