VM timekeeping: Using the PTP Hardware Clock on KVM
VM timekeeping: Using the PTP Hardware Clock on KVM
Paul Gear<br>| Thu 25 April 2024
Contents
Background
Performance improvements - chrony
Common prerequisite
Chronyd configuration
Ntpd configuration
Performance improvements - ntpd
Next steps
Thanks
Background
In my last post I described the<br>setup I use to provide time synchronisation to the hosts I maintain in the NTP<br>pool. I only recently learned about the PTP<br>Hardware Clock (PHC) device driver available in KVM, and started testing it in<br>earnest earlier this year.
The main reason to use the PHC is to more closely track the hypervisor host.<br>This is possible because reading from the local PHC is designed to be very<br>efficient and it incurs much less overhead than performing an NTP request and<br>response over the network.
Obviously this requires the host to have a clock worth tracking. My previous<br>post explained the setup I use, but the general guidelines for a VM host would<br>be the same as for any quality NTP setup:
Use at least four clock sources, with a<br>diversity of reference clocks.
Peers should be selected on the basis of reliable network connectivity and<br>reliable timekeeping performance. If you're using a host from the public<br>pool, checking its pool score page is highly recommended. Here's an example:<br>150.101.186.48.
All other things being equal, closer hosts (in terms of network delay) are<br>better than further away hosts, because they allow NTP to constrain its error<br>estimations to a narrower range.
Performance improvements - chrony
So how much difference does it make using the PHC? The first system I enabled<br>(after validation and playbook development on a test VM) was my chronyd pool<br>server. It is using 2 vCPUs and 1 GB RAM on a host with a 6-core AMD Ryzen 5<br>Pro 3600 CPU, with a maximum clock speed of 3.6 GHz.
I was already quite happy with the time sync performance of this VM - it was<br>reporting a system offset within ± 50 µs:
The system frequency error ranged between about -19.44 and -19.64 ppm:
And the root dispersion ranged between 80 and 280 µs:
After enabling the PHC device, the same VM actually reported a larger range of<br>system frequency error, now ranging between -19.1 and -19.8 ppm:
But system offset reduced to ± 4 µs:
And maximum root dispersion less than 2 µs:
Here are the graphs showing a few days on either side of this configuration<br>change.
System offset:
Frequency error:
Root dispersion:
One point to note about this is that by default chrony sees the PHC device as<br>stratum 0, rather than stratum 2 equivalent. More importantly, its figures for<br>root dispersion and root delay are misleadingly low because they only account<br>for the time taken to read the PHC device.
To address this, chrony allows the stratum and root delay of the PHC reference<br>clock to be set manually in the config file. I set the root delay by looking at<br>the host's root delay over a few days prior to the change, and picking a<br>slightly higher value than the minimum (in my case I used 400 µs).<br>Unfortunately, there's no way to obtain the host's root dispersion from the PHC<br>device (hence why the AWS Nitro PHC driver reports clock error bound via a<br>separate sysfs interface), and no mechanism that I'm aware of in chrony to<br>adjust it (although it is influenced by the configured root delay).
Setting the stratum to an accurate value is also potentially problematic,<br>because chrony uses stratum as part of its sync peer selection algorithm. I<br>eventually settled on setting the PHC to stratum 1, so that it doesn't appear to<br>be lower stratum than the stratum 1 servers on my local network, but is still<br>likely to be selected as the sync peer under most circumstances.
Common prerequisite
Before setting up an NTP service to use the PHC device, there's one<br>prerequisite: loading the ptp_kvm driver. It takes no options and should be<br>available in all mainstream Linux kernel builds. Activating it is as simple<br>as:
# modprobe ptp_kvm
On my systems this does not produce any kernel log message, because the core PTP<br>driver is compiled into the default kernel. So the only indication that the<br>module is loaded is the presence of a /dev/ptp0 device file, along with a<br>symlink to it indicating that belongs to KVM:
# ls -la /dev/ptp*<br>crw------- 1 root root 248, 0 Apr 21 12:24 /dev/ptp0<br>lrwxrwxrwx 1 root root 4 Apr 21 12:24 /dev/ptp_kvm -> ptp0
To ensure that this is always loaded on boot, I add the driver name to<br>/etc/modules on my Debian and Ubuntu systems:
ptp_kvm
Other Linux distributions may use a slightly different mechanism for this, e.g.<br>a file in /etc/modules-load.d/ (which also works on Debian-based distros).
Chronyd configuration
Chrony has built-in support for the PHC device as a reference clock. To enable<br>it, I use the following configuration line:
refclock PHC /dev/ptp0 poll 0 delay 0.0004 stratum 1
As mentioned above, the delay and stratum options are to tune these<br>variables so that they're not reported as...