how to support ptp

[xiaotailang]

Hello everyone, if I want to add PTP (Precision Time Protocol) synchronization functionality to NuttX system, how can I port ptpd or the PTP sub-module from Linux to NuttX? Are there any reference methods available?

[xiaoxiang781216]

@PetteriAimonen add PTP support recently, you can reference the related patch:
apache/nuttx-apps#2101
#10827

[PetteriAimonen]

Yeah, I'm currently working on PTP support for STM32F4 platform.

The PTP daemon is mostly done, with the few missing features noted in the pull request linked above.

For highest resolution, the platform should have support for timestamping arriving and leaving ethernet packets. Currently there are not yet NuttX drivers for that. But being an RTOS, it will probably achieve better than 0.1 ms accuracy even without hardware timestamping, provided that a high resolution RTC is available.

My plan is to next add support for STM32F4 to use the ethernet peripheral 64-bit timestamp counter for the system time source, and to query timestamps from that through SO_TIMINGS ioctl call.

[xiaotailang]

ok! Thank you very much.

[pkarashchenko]

The ethernet packet timestamping is not something that can be handled by the driver. I mean that of course we can extend the driver to add timestamps, but those will be SW timestamps and will not actually represent time when packets arrive/leave HW. I mean that with having all the DMA stuff in ethernet for reception the actual RX processing may be deferred, so the packet that enters HW at X, but processed at X+N staying for N in memory after DMA operation is finished, but SW didn't start processing, so N will be a small random value.
So packet timestamping is something that should be supported by the HW

[PetteriAimonen]

Yes, for hardware timestamping it is needed for the hardware to have support (STM32F4 hardware does have) and NuttX driver for the hardware to have support.

[acassis]

@PetteriAimonen a simple time synchronization also can be obtained using GPS modules. Even those low cost U-Blox clones with PPS pins are able to deliver good synchronization for something as lower as 10ns. Of course, an external GPS module could involve costs compared to Ethernet IEEE 1588 that is already in the MCU, but it could be something easier to test/implement.

[xiaotailang]

hello everyone ! I encountered a problem while developing Ethernet PTP. After enabling PTP function by setting PTP-related registers, a hardfault occurred. Upon tracing the issue, I found that the address pointed to by rdes2 (the receive descriptor for the Ethernet) changed after enabling PTP. For example, if the initial address set for rdes2 was 0x01, then enabling PTP would cause it to point to some other inaccessible address resulting in a hardfault. I tested disabling the PTP enable bit in the register, and the rdes2 address remained unchanged and did not cause any hardfaults. I have also conducted tests where only the PTP enable bit is set without any additional PTP-related configurations, and this phenomenon still occurs. My board is STM32F767. Have you encountered a similar issue while debugging Ethernet PTP?

[PetteriAimonen]

@xiaotailang For STM32F4x7 it works this way (from reference manual section "Rx DMA configuration":

If IEEE 1588 time stamping is enabled, the DMA writes the time stamp (if available) to
the current descriptor’s RDES2 and RDES3. It then takes the received frame’s status
and writes the status word to the current descriptor’s RDES0, with the OWN bit cleared
and the Last segment bit set.

If software has
enabled time stamping through CSR, when a valid time stamp value is not available for the
frame (for example, because the receive FIFO was full before the time stamp could be
written to it), the DMA writes all ones to RDES2 and RDES3. Otherwise (that is, if time
stamping is not enabled), RDES2 and RDES3 remain unchanged.

The RX DMA descriptor is longer if timestamping is enabled and the code must be changed to account for that.

[xiaotailang]

Thank you very much for the assistance you've provided

[xiaotailang]

Hi@ PetteriAimonen
After adding PTP (Precision Time Protocol), I observed that the timestamp of the slave consistently lags behind the master by approximately 1.5 seconds during clock synchronization. In this situation, PTPd executes a Clock Step, setting the slave's timestamp to match the master's. However, after receiving sync and follow-up messages in the next cycle, the calculated time difference shows the slave lagging behind the master again by about 1.5 seconds. Subsequently, another Clock Step is performed, and this cycle continues.

Upon tracing and debugging, it appears that the slave's time progresses much slower than the master's. I attempted to mitigate this by setting a large value for the addend register in the adj function on the slave, such as 0xfffffff, causing the accumulator to overflow as quickly as possible and updating the system time accordingly. In this scenario, the slave's timestamp does not fall significantly behind, and the overall PTP synchronization process works correctly, gradually bringing the slave's time closer to the master's. However, due to the constant value of 0xffffffff for the addend, the slave's time eventually surpasses the master's by a considerable margin.

I have tested the PTP working clock frequency, which is 50MHz. Have you encountered similar phenomena in your extensive development experience? Specifically, if adjusting the addend value according to the offset calculated by PTPd and updating it proportionally, the slave's timestamp consistently lags significantly behind.

[xiaotailang]

I obtain timestamps from the registers rdes2 and rdes3. After initializing the descriptors, I backup the buffer addresses they point to. In the function responsible for receiving Ethernet packets, I read the values of rdes2 and rdes3 to retrieve the timestamp. Later, I restore the buffer addresses of rdes2 and rdes3 from the backup.

[PetteriAimonen]

No, I haven't seen errors anywhere near that magnitude.

I assume you are using latest ptpd from the nuttx-apps repo?
Can you try enabling CONFIG_NETUTILS_PTPD_DEBUG which should print a message for every sync packet, detailing the calculation results.

[xiaotailang]

Okay, thank you very much for your suggestions.

[xiaotailang]

Hi@ PetteriAimonen, I am currently conducting a test with two STM32 boards where one is set as the master and the other as the slave. Upon re-examining the PTPd-related code in Nuttx 12.4, if I have these two STM32 boards connected directly via an Ethernet cable, how can I configure which board should act as the slave and which as the master? I noticed that when the PTPD_SERVER macro is enabled, it enables the board to send announce messages, indicating its capability to serve as a master clock. However, if this macro is enabled on both boards, how does the system determine which board becomes the master and which one becomes the slave?

[PetteriAimonen]

The best master clock algorithm is used:
https://github.com/apache/nuttx-apps/blob/master/netutils/ptpd/ptpd.c#L238

If the config is identical for boards, it picks based on MAC address of the network interface (lower address wins)

[xiaotailang]

Thank you again for your enthusiastic answers. I noticed that the PTP driver only acquires the timestamp of received packets. According to the PTP protocol, it should acquire both the timestamp of receiving and sending, calculate the time difference between them, and constantly adjust the time of the slave. However, I found that the driver does not get the timestamp from the sending descriptor. What's the design idea of this place?

[PetteriAimonen]

It's just that transmission timestamps are not implemented in NuttX currently. It could be added, but because of the RTOS nature, the packets go out near immediately when sent, so it is not as important to accuracy as the RX timestamp.

I found some difficulty in trying to design how TX timestamps in NuttX would work, because they are possibly known only later when send() has already returned. So I decided not to do it when I achieved the necessary precision (less than 10 us error) without it.

[xiaotailang]

ok! I really appreciate your patient answers.

[xiaotailang]

In the current design, if we want to further improve the precision, such as to achieve a precision of 20-50ns, is there any way to do it?

[PetteriAimonen]

That precision probably requires a specifically designed PTP-aware switch, or a direct cable connection between two devices. It would also require adding the hardware TX timestamp support.

With a generic TP-Link Ethernet switch, I'm seeing packet delays of several microseconds and with 1-2 microsecond variation. The delay measurement and averaging can compensate a bit, but I don't think 50 ns is achievable with generic Ethernet switches.

[xiaotailang]

Thank you very much for your detailed answer. Additionally, I would like to express my gratitude for the assistance you've provided regarding PTP-related issues throughout this period.