It's quite interesting - this isn't ethernet as we know it. Instead of each NIC using its own free-running clock, all the physical layers are sync'ed to each other at layer 1. (note that gigabit ethernet, which is what it uses, sends data at all times - when idle it sends the idle symbol)
Haven't looked into this in depth but sub-nanosecond sync for systems up to 10km apart is interesting since 10km is about 33 light microseconds. There is some trickery going on.
It's totally possible to achieve synchronization better than light transmission time. For the purposes of synchronization, the speed of light delay, and any other delay are indistinguishable, and need not be distinguished.
The gravity well time dilation is about 3.5 nanoseconds per meter per year near the surface of the earth. (time changes rate with altitude in a gravity well)
Sub-nanosecond synchronization is getting into the relativity is measurable realm.
Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.
It's the physics of cooling the beasts and the communication delays that make those plans ludicrous.
To turn your assertion on its head, the fact that the supporters don't seem to be able (or willing) to do the math to fact check these proposals is not an indicator that the plans will work.
As a starting point for comparison, the total power budget of the ISS is under 100kW and a single supercomputer rack dissipates about 4x that. What changes to the ISS can be made to get 100x more power and dissipate 100x more heat?
Oh I have done the math. There are multiple ways to get cooling to work in space:
1. There is no real size limit to radiators in space, especially when in solar orbit.
2. High temperature chip architectures can be used, operating at 600K.
3. Heat pipes can bring the temperature even higher, such as to 1200K.
4. Special 3D radiator geometries can be used to optimize heat escape.
5. Metamaterials can be used to optimize photon emission in the best directions.
Together, these will shrink the required radiator area dramatically. Beyond these standard ideas, other exotic approaches exist at the edge of viability.
The ISS in contrast is restricted considering it has to sit in Earth orbit.
Hmm one would expect heat expansion to change the length of fiber over tens of kilometers. Does it also affect light speed in the fiber? I think consumer fiber is not buried very deep on average, but maybe for these use cases you use something hefty anyway.
In our lab tests phase lock jitter between WR client and master is about 10ps (picoseconds) over 50km fiber (with temperature change of the fiber, so WR actively compensates elongations), so relative clock of one system can be transmitted with about that accuracy to another.
P.S. There is WR workshop this week with some talks being publicly available on CERN's indico website.
Even though you're commenting on While Rabbit post, it took some time to understand "WR" is white rabbit, esp. since describing the pico seconds in brackets.
If both systems have a good clock. Then the synchronization messages only need to contain the time delta to correct the time (phase?) drift to achieve full synchronization.
If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?
> Conventional wisdom is that distributed consensus is not possible at this kind of performance
I'm not sure why you would think that? If you can assume the fiber is the same in both directions you know the round trip time is exactly double the latency of the connection. Then you know to phase shift your start time by that much when you get a start signal and you're in sync.
Obviously it's not trivial in practice, but it's not a fundamentally insurmountable problem.
Haven't dug in on the technicals, but this is coming out of CERN, it looks like - and in that light, the links to "We're hiring" on that page almost feel like a flex...
Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs round trips. At geographic scale that’s tens of milliseconds per decision, and the floor is set by the speed of light.
Tight clock sync collapses this. If clock uncertainty ε is small and bounded, you can timestamp a write, wait ε, and trust the global order without talking to anyone. Spanner’s external consistency works because TrueTime’s ε was a few milliseconds, so commit-wait was tolerable. The latency cost of planet-scale serializability stops depending on how far apart your replicas are and starts depending on how good your clocks are.
That’s the real significance. Time sync converts a coordination problem (bounded by physics) into a local computation (bounded by clock quality). Spanner proved this is possible but required GPS receivers and atomic clocks in every datacenter, which kept the capability inside Google for years. White Rabbit-class sync pushes ε from milliseconds toward sub-nanoseconds over commodity Ethernet hardware, and it’s now in IEEE 1588 as a standard PTP profile. If sub-nanosecond sync becomes baseline network infrastructure, the long-held assumption that strong consistency has to be slow at geographic scale stops holding, and a meaningful chunk of what databases currently work around (HLCs, weak isolation defaults, application-level reconciliation) becomes unnecessary.
Very good explanation and interesting take on the 'humanity scale' or internet scale significance. I work on a phased array system so significance of white rabbit for me was always sample alignment. Assumed CERN had a similar use case of needing to order (sensor data of) physical events happening far apart.
But if we imagine the vast majority of internet and telecom infrastructure is also implemented this way, we can reason about information over time in general. Makes me think of 'earth is a big computer' type of sci fi trope. Neat!
Indeed, time synchronization across detectors is always tricky. Distributed clocks get messy at ATLAS dimensions. WR allows to distribute pretty good time sync over large detector systems.
Sometimes still not good enough though. Time-of-flight detectors try to get to single-digit ps level, and almost by definition, you have to synchronize two detectors that are some distance apart.
I don’t see why sub-nanosecond sync is useful for Spanner-style ordering. Your average server is more than 1 light-ns wide! Your average cable from server to TOR switch is several light-ns long!
If I remember everything is open hardware, CERN should have those repo accessible, last time I used it it was still very much in dev, especially their PCIe cards with custom kernel. This being said, I haven't touched it since ~6 years ago...
If you run "make" in the papers/IBIC2013 directory you'll get this paper: https://cds.cern.ch/record/1743073/files/thbl2.pdf
It's quite interesting - this isn't ethernet as we know it. Instead of each NIC using its own free-running clock, all the physical layers are sync'ed to each other at layer 1. (note that gigabit ethernet, which is what it uses, sends data at all times - when idle it sends the idle symbol)
there are several 10 gbps implementations made in Europe https://gitlab.com/ohwr/project/10G-wr-nic/-/blob/f_ltgt_spe...
Haven't looked into this in depth but sub-nanosecond sync for systems up to 10km apart is interesting since 10km is about 33 light microseconds. There is some trickery going on.
Yes, it uses phased locked loops and measures phase difference between the master clock and the local clock.
Similar to the much smaller scale Wi-Wi setup that was on HN a few days ago (which came after WR): https://news.ycombinator.com/item?id=48209055
It's totally possible to achieve synchronization better than light transmission time. For the purposes of synchronization, the speed of light delay, and any other delay are indistinguishable, and need not be distinguished.
The gravity well time dilation is about 3.5 nanoseconds per meter per year near the surface of the earth. (time changes rate with altitude in a gravity well)
Sub-nanosecond synchronization is getting into the relativity is measurable realm.
That means you get a free clock cycle every 2-3 hours on top of a mountain compared to sea level!
Datacenters in spaaaace!
But they travel at 8 km/s so actually that cancels benefits? EDIT: checked, not enough to cancel them completely.
I wonder I'd that's the math for the ludicrous space data center ideas "floating" around...
Just because you don't understand how something can work doesn't make it ludicrous. People like you are hell bent on destroying what's left of the Earth by turning it into a computer. If we left progress to those without an imagination, we wouldn't even have had a working calculator.
That might be true if the proponents had an argument of any sort in support of their plan, other than "we need idiots to give us money".
You're right.
It's the physics of cooling the beasts and the communication delays that make those plans ludicrous.
To turn your assertion on its head, the fact that the supporters don't seem to be able (or willing) to do the math to fact check these proposals is not an indicator that the plans will work.
As a starting point for comparison, the total power budget of the ISS is under 100kW and a single supercomputer rack dissipates about 4x that. What changes to the ISS can be made to get 100x more power and dissipate 100x more heat?
Oh I have done the math. There are multiple ways to get cooling to work in space:
1. There is no real size limit to radiators in space, especially when in solar orbit.
2. High temperature chip architectures can be used, operating at 600K.
3. Heat pipes can bring the temperature even higher, such as to 1200K.
4. Special 3D radiator geometries can be used to optimize heat escape.
5. Metamaterials can be used to optimize photon emission in the best directions.
Together, these will shrink the required radiator area dramatically. Beyond these standard ideas, other exotic approaches exist at the edge of viability.
The ISS in contrast is restricted considering it has to sit in Earth orbit.
You may see this video (AI researched and generated) to grasp some of the points: https://www.youtube.com/watch?v=s7Mv_OcBXI8
Two-way time transfer measures the round-trip propagation time. As a result, it's not directly relevant to the accuracy.
So then you need to know distance / roundtrip-length within centimeter precision as well (below 29.98 cm for sub-nanosecond precision… to be precise).
Since cm precision is often not possible, is roundtrip-length an estimated average from prior roundtrips?
delay is easy
jitter kills
The roundtrip time is measured and compensated. Even NTP does this. Knowing the distance is not necessary for time synchronization.
gPTP estimates the link delay and the peer clock ratio, see for example this random link I just found for you: https://blog.meinbergglobal.com/2024/03/27/what-is-gptp/
Hmm one would expect heat expansion to change the length of fiber over tens of kilometers. Does it also affect light speed in the fiber? I think consumer fiber is not buried very deep on average, but maybe for these use cases you use something hefty anyway.
It doesn't matter if the length changes provided:
* you measure the round trip time often enough
* the shift affects light in both directions equally
... why would cm precision be often not possible?
Because you're using fibres laid years ago. Often pulse velocity isn't known to better than one part in 1000.
yes, it needs custom built hardware to work.
In our lab tests phase lock jitter between WR client and master is about 10ps (picoseconds) over 50km fiber (with temperature change of the fiber, so WR actively compensates elongations), so relative clock of one system can be transmitted with about that accuracy to another.
P.S. There is WR workshop this week with some talks being publicly available on CERN's indico website.
Even though you're commenting on While Rabbit post, it took some time to understand "WR" is white rabbit, esp. since describing the pico seconds in brackets.
If both systems have a good clock. Then the synchronization messages only need to contain the time delta to correct the time (phase?) drift to achieve full synchronization.
Some may find https://gitlab.com/ohwr/project/white-rabbit/-/wikis/home an easier starting point. Particularly the "Synchronization" page.
In short, it's about giving PTP and SyncE some extra smarts.
Probably even better to start at the collaboration website https://www.white-rabbit.tech/wr-technology/
If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?
> Conventional wisdom is that distributed consensus is not possible at this kind of performance
I'm not sure why you would think that? If you can assume the fiber is the same in both directions you know the round trip time is exactly double the latency of the connection. Then you know to phase shift your start time by that much when you get a start signal and you're in sync.
Obviously it's not trivial in practice, but it's not a fundamentally insurmountable problem.
Twice the path delay + the time it takes to send the return packet. I assume WR does this in hardware to get a predictable time.
Haven't dug in on the technicals, but this is coming out of CERN, it looks like - and in that light, the links to "We're hiring" on that page almost feel like a flex...
What is significance of this?
it is useful e.g. to align the phase of signals being sent from different locations
Distributed systems spend most of their effort on one problem: agreeing on the order of events across machines. Without synchronized physical clocks you have two options. Logical clocks (Lamport, vector) give you causal order but not wall-clock truth, so you can’t answer “did A really happen before B” for events that don’t have a happens-before relationship. Or you run consensus, which gives total order but costs round trips. At geographic scale that’s tens of milliseconds per decision, and the floor is set by the speed of light.
Tight clock sync collapses this. If clock uncertainty ε is small and bounded, you can timestamp a write, wait ε, and trust the global order without talking to anyone. Spanner’s external consistency works because TrueTime’s ε was a few milliseconds, so commit-wait was tolerable. The latency cost of planet-scale serializability stops depending on how far apart your replicas are and starts depending on how good your clocks are.
That’s the real significance. Time sync converts a coordination problem (bounded by physics) into a local computation (bounded by clock quality). Spanner proved this is possible but required GPS receivers and atomic clocks in every datacenter, which kept the capability inside Google for years. White Rabbit-class sync pushes ε from milliseconds toward sub-nanoseconds over commodity Ethernet hardware, and it’s now in IEEE 1588 as a standard PTP profile. If sub-nanosecond sync becomes baseline network infrastructure, the long-held assumption that strong consistency has to be slow at geographic scale stops holding, and a meaningful chunk of what databases currently work around (HLCs, weak isolation defaults, application-level reconciliation) becomes unnecessary.
Very good explanation and interesting take on the 'humanity scale' or internet scale significance. I work on a phased array system so significance of white rabbit for me was always sample alignment. Assumed CERN had a similar use case of needing to order (sensor data of) physical events happening far apart.
But if we imagine the vast majority of internet and telecom infrastructure is also implemented this way, we can reason about information over time in general. Makes me think of 'earth is a big computer' type of sci fi trope. Neat!
Indeed, time synchronization across detectors is always tricky. Distributed clocks get messy at ATLAS dimensions. WR allows to distribute pretty good time sync over large detector systems. Sometimes still not good enough though. Time-of-flight detectors try to get to single-digit ps level, and almost by definition, you have to synchronize two detectors that are some distance apart.
Awesome explainer, thanks for that
I don’t see why sub-nanosecond sync is useful for Spanner-style ordering. Your average server is more than 1 light-ns wide! Your average cable from server to TOR switch is several light-ns long!
Used quite a bit by stock exchanges to ensure consumers and publishers have a reasonably aligned time.
If your system depends on highly accurate time maybe you’re doing it wrong?
Of course no doubt there’s some requirements that cannot avoid that requirement.
But for the most part needing synchronized time is just going to be a forever problem isn’t it?
Not on GitHub?
Its on gitlab but even there I failed to find sources, documentation/presentations are there though
https://www.white-rabbit.tech/wr-technology/
Note that this is also for a large part a hardware-based technology
If I remember everything is open hardware, CERN should have those repo accessible, last time I used it it was still very much in dev, especially their PCIe cards with custom kernel. This being said, I haven't touched it since ~6 years ago...