FPGA-based NTP server with Digilent Arty A7-35 and Pmod GPS using Fakernet

Features

• Good time resolution: ≈ 100 ns, typically < 20 ns.
• Packet rate (to be checked, also vs. typical resolution drop): 50000 packets/s (@ 100 Mbps).
• Hard to hack: no OS, only FPGA/VHDL packet responder state machine. (TODO: remove Fakernet UDP controls from NTP build.)
• Cheap: < $ 250 / 250 €.

Page content

• Features
• Equipment
• Setup
• Testing
• With GPSD
• With chrony
• With ntpdig
• Performance tests
  • GPS and FPGA
  • Disturbed by missing PPS and oscillator drift
  • Network
  • Cold start
  • Performance conclusions
• Time tracking (GPS NMEA and PPS → NTP)
• Security
• Download

Equipment

• Digilent Arty A7-35 board.
• Pmod GPS receiever, u-blox NEO-6M/NEO-7M-based boards also tested.
• optional: GPS antenna.

While the GPS receiver is able to work in-doors in some circumstances, the use of an outdoor antenna with a good skyview improves the satellite signals and thus the system stability. Note: antenna requires soldering of an SMA connector. (Digilent: very nice boards, but reduced score for not soldering the SMA connector.)

Setup

openFPGALoader -b arty boards/Arty_A7-35/a35_fnet/a35_fnet.bit

Testing

Basic Fakernet testing is described here.

Monitor GPS UART messages and PPS pulses:

client/fnetctrl 192.168.1.192 --tcp=gps
PPS-1: 4021974195
TRACK-1: 3869485087.000000032 (:000000140) a:8983270 d: -11
UART-1: $GPGGA,165807.000,5742.7691,N,01201.3512,E,1,11,0.82,37.0,M,40.0,M,,*51
UART-1: $GPGSA,A,3,29,11,20,05,07,02,09,06,26,04,30,,1.63,0.82,1.41*0D
UART-1: $GPRMC,165807.000,A,5742.7691,N,01201.3512,E,0.01,188.11,140822,,,A*64
NEXT-1: 3869485088
UART-1: $GPVTG,188.11,T,,M,0.01,N,0.03,K,A*3E
PPS-1: 4121972906
TRACK-1: 3869485088.000000036 (:000000155) a:8983140 d: -26
UART-1: $GPGGA,165808.000,5742.7691,N,01201.3512,E,1,12,0.78,37.0,M,40.0,M,,*58
UART-1: $GPGSA,A,3,29,11,20,05,07,02,09,06,26,04,30,16,1.61,0.78,1.41*0D
UART-1: $GPGSV,4,1,14,07,62,160,37,20,54,283,34,09,52,091,42,11,38,246,32*7E
UART-1: $GPGSV,4,2,14,30,36,196,35,02,35,279,24,16,22,043,16,05,21,305,15*79
UART-1: $GPGSV,4,3,14,06,19,211,16,04,16,085,16,29,11,324,25,44,09,122,*78
UART-1: $GPGSV,4,4,14,26,08,021,26,13,01,258,*7B
UART-1: $GPRMC,165808.000,A,5742.7691,N,01201.3512,E,0.02,188.11,140822,,,A*68
NEXT-1: 3869485089
UART-1: $GPVTG,188.11,T,,M,0.02,N,0.03,K,A*3D
PPS-1: 4221971616
TRACK-1: 3869485089.000000024 (:000000107) a:8983250 d: 22
UART-1: $GPGGA,165809.000,5742.7691,N,01201.3512,E,1,12,0.78,37.0,M,40.0,M,,*59
UART-1: $GPGSA,A,3,29,11,20,05,07,02,09,06,26,04,30,16,1.61,0.78,1.41*0D
UART-1: $GPRMC,165809.000,A,5742.7691,N,01201.3512,E,0.01,188.11,140822,,,A*6A
NEXT-1: 3869485090
UART-1: $GPVTG,188.11,T,,M,0.01,N,0.01,K,A*3C

• PPS: internal 32-bit cycle counter at PPS signal.
• UART: serial NMEA messages from the GPS.
• NEXT: (NTP) seconds at next PPS (decoded from NMEA $GPRMC, +1).
• TRACK: internal NTP timestamp tracking parameters. Current timestamp (a few clock cycles after PPS), fractional addition, NTP fractional second offset at PPS.

With GPSD

Pass UART messages to GPSD for monitoring:

client/fnetctrl 192.168.1.192 --tcp=gps | grep --line-buffered "UART-1" | sed -u -e "s/UART-1: //" | /usr/sbin/gpsd -b -n -N /dev/stdin -p -S 29470

Terminal/text view of GPS status (while above command is running):

cgps localhost:29470

Graphical view of satellites (while first command above is running):

xgps localhost:29470

With chrony

server 192.168.1.192 iburst minpoll 2 maxpoll 2 minsamples 32 prefer
hwtimestamp *

chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.1.192                 1   2   377     3     -6ns[   -9ns] +/-   17us

chronyc sourcestats
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
192.168.1.192              32  15   126     -0.000      0.002     -0ns   151ns

chronyc ntpdata 192.168.1.192
Remote address  : 192.168.1.192 (C0A801C0)
Remote port     : 123
Local address   : 192.168.1.75 (C0A8014B)
Leap status     : Normal
Version         : 4
Mode            : Server
Stratum         : 1
Poll interval   : 4 (16 seconds)
Precision       : -26 (0.000000015 seconds)
Root delay      : 0.000000 seconds
Root dispersion : 0.000000 seconds
Reference ID    : 47505300 (GPS)
Reference time  : Sat Feb 11 09:43:16 2023
Offset          : -0.000000031 seconds
Peer delay      : 0.000032769 seconds
Peer dispersion : 0.000000910 seconds
Response time   : 0.000003520 seconds
Jitter asymmetry: +0.00
NTP tests       : 111 111 1111
Interleaved     : No
Authenticated   : No
TX timestamping : Hardware
RX timestamping : Hardware
Total TX        : 10315
Total RX        : 10238
Total valid RX  : 10238

With ntpdig from ntpsec

ntpdig 192.168.1.192
2023-10-12 19:09:28.809426 (+0000) +0.000085 +/- 0.011816 192.168.1.192 s1 no-leap

Performance tests

GPS and FPGA

The upper (blue) graph shows the offset of the internal timestamp tracking at each PPS signal.

The middle graph shows the actual internal clock counts (100 MHz) between PPS signals (orange), and the same calculated from the internal tracking adjustment parameter (green).

The lower (red/magenta) graph shows the internal (100 MHz) clock count offset of the PPS signals of two additional GPS modules attached to the same board, relative to the first GPS module. (This allows to get a handle on the GPS module performance without introducing any measurement errors by NTP or PC clocks. Only the direct PPS sampling by the FPGA is used here.)

According to schematics, the oscillator is an ASEM1-100.000MHZ-LC-T which is specified to have a frequency stability of ±50 ppm.

Disturbed by missing PPS and oscillator drift

The behaviour when the board 100 MHz oscillator is disturbed by some forced cooling (by blowing air over it), leading to tracking errors of several 100 ns.

The left column is an overview of three disturbances, shown in the next three columns: the second column is for a temporary loss of PPS signal (GPS receiver must have lost satellites). The third column is for periodic (few seconds) cooling. The fourth column is for one longer cooling.

The overall resolution limit seems to be the slow update rate made possible by the 1 Hz discipline from the PPS signal. Luckily (at least this) board oscillator is stable enough to allow good tracking when it (slowly) returns to ambient temperature.

Network

The measurements are performed in rapid bursts, with queries repeated at shorter and shorter intervals, starting with the round-trip time. The total time-span of the burst is a few round-trip times. The circles/solid line (brown) show the first offset measurement in a burst (i.e. not a burst at all). The cross/dashed line (pink) show the result derived from the best combination of transmit and return half of all the measurements in the burst. The plus/dotted line (cyan) has filtered out of each ten successive measurements (i.e. over 10 s) the third lowest round-trip time.

Columns one and two show measurements between two Arty A7-35 boards (100 Mbps) connected by a Netgear FS105 switch. The reference board has an external antenna connected (but located indoors), and so has other board in the pair in first column. The second column shows measurements vs. a board with only the internal antenna of the GPS module.

Columns three and four show measurements between two ALINX AX516 boards (1 Gbps) connected by a Netgear GS105 switch. Both boards have an external antenna connected (but located indoors). For the fourth column, energy-efficient ethernet (EEE) is disabled. When enabled (column three), the effect is surprisingly small in the offset jitter, but leads to approximately 40 μs longer transmission latencies. In burst measurements, i.e. packets sent in rapid succession, the latencies drop for some transmission steps, leading to large shifts.

Rows one and two show measurements as a function of time. The more rapid shifts for an internal antenna (column two) are attributed to more difficulties for the GPS to select satellites with few or no reflections. It should however be noted that a similar amount of maximum deviations are seen for the external antenna case, implying that the even if the short-term variations are smaller, the absolute time offset varies by some 100 ns also in that case.

The third row shows a histogram of the time offsets in row two.

The fourth and fifth rows show the Allen deviation of the measurements.

Cold start

Performance conclusions

• GPS: The first lower (red/magenta) graph suggest a good precision value around 10 ns, but already with simple disturbances, the accuracy (with in-door, no-antenna reception) is not better than 100 ns. The network tests also suggest long-term accuracy drifts around 100 ns.
• FPGA+oscillator: The first upper (blue) graph suggest a good FPGA tracking precision below 10 ns, but again already with simple disturbances, the accuracy loss can easily be a few 100 ns. Here the remedy is to ensure that the board (and especially the oscillator) is not subject to sudden (few seconds) temperature variations.
• Network: The measurement between two independent boards show a precision below 20 ns. The long-term accuracy drifts up to 100 ns are likely attributable to the GPS tracking and varying satellite visibility. However, best measurements with a PC have been at some few 100 ns accuracy.

Time tracking (GPS NMEA and PPS → NTP)

The serial NMEA messages are parsed to decode the $GPRMC sentences to get time (seconds) and valid fix status.

A fractional increment is used to update the NTP timestamp counter each internal clock cycle (here 125 MHz on the Arty board).

On each PPS signal, the difference between the current NTP timestamp and the expected is calculated. From this, determine a correction to the fractional increment such that the difference calculated in 1 s (the next PPS) would be 0, if all frequencies stay constant. Half of the correction is added to the increment, and the other half is applied only until the next PPS signal.

The NTP status (in the leap seconds field) is reported as unsynchronised whenever no PPS pulses (with a preceding time report with valid fix) has been reported for 2 s. When unsynchronised, the precision is also reported as 127.

The precision field is calculated as the 2-logarithm from a decaying average of the absolute value of the NTP timestamp difference at each PPS signal. The running average decays with 3/4 of the old value retained, and 1/4 of the new difference added. (With the used GPS module, this is typically −27 (2⁻²⁷ = 7.5 ns), with fluctuations between −26 and −28.)

Security

• + No OS to hack.
• + Cannot make infinite packet loop.
• − Like with any NTP server, a Fakernet NTP server can be used as an relay to send packets to a third party victim by the aggressor using the victim IP address as the NTP query packet source address.

The table below (work-in-progress!) describes the handling of each octet of all incoming packets, to provide an overview of the (in-)susceptibility to crafted NTP packets.

Unless specified, or a checksum, the outgoing (response) packet uses the incoming value for each octet.

Download / source

Sources are available with the Fakernet sources.

Last modified: Sat Feb 11 09:40:24 CET 2023