• Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure.
• To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance.
• With the help of the OCP community, we established the Open Compute Time Appliance Project and open-sourced every aspect of the Open Time Server.

In March 2020, we announced that we were in the process of switching over the servers in our data centers (together with our consumer products) to a new timekeeping service based on the Network Time Protocol (NTP). The new service, built in-house and later open-sourced, was more scalable and improved the accuracy of timekeeping in the Facebook infrastructure from 10 milliseconds to 100 microseconds. More accurate time keeping enables more advanced infrastructure management across our data centers, as well as faster performance of distributed databases.

The new NTP-based time architecture uses a Stratum 1 — an important component that is directly linked to an authoritative source of time, such as a global navigation satellite system (GNSS) or a cesium clock.

Many companies rely on public NTP pools such as time.facebook.com to act as their Stratum 1. However, this approach has its drawbacks. These pools add dependency on internet connectivity and can impact overall security and reliability of the system. For instance, if connectivity is lost or an external service is down, it can result in outages or drift in timing for the dependent system.

To remove these dependencies, we’ve built a new dedicated piece of hardware called Time Appliance, which consists of a GNSS receiver and a miniaturized atomic clock (MAC). Users of time appliances can keep accurate time, even in the event of GNSS connectivity loss. While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance.

Why do we need a new time device?

Off-the-shelf time appliances have their own benefits. They work right out of the box and because many of these devices have been on the market for decades, they are battle-tested and generally stable enough to work without supervision for a long time.

However, these solutions also come with trade-offs:

• In most cases, they are outdated and often vulnerable to software security concerns. Feature requests and security fixes may take months or even years to implement.
• These devices come with closed source software, which makes configuring and monitoring them limited and challenging. While configuration is done manually via a proprietary CLI or Web UI, monitoring often uses SNMP, a protocol that was not designed for this purpose.
• They include proprietary hardware that is not user-serviceable. When a single component breaks, there is no easy way to replace it. You have to either ship it to the vendor for repair or buy an entire new appliance.
• Since off-the-shelf devices are made in low quantities, they come with a higher markup and can become very costly to operate over time. The high cost associated with off-the-shelf devices create limitations for many in the industry. An open source version would open the door to broader applications.

Until now, companies have had to accept these trade-offs and work within the constraints described above. We decided it was time to try something different, so we took a serious look at what it would take to build our new Time Appliance — specifically, one using the x86 architecture.

Prototyping the Time Appliance

Here’s a block diagram of what we envisioned: