Precision Time Protocol in a Parallel Redundant Network
April 29, 2019
- This lab report aimed to address the challenges of seamless high-accuracy time synchronization during redundant network failover.
- We chose to model this scenario because it demonstrates the redundant network with precision time protocol that will face utilities implementing digital substations.
- The results showed that proper implementation of IEEE 1588 PTP and IEC 62439-3 PRP will maintain time synchronization following the loss of one network in a parallel redundant scheme.
For some background, implementations of the substation communications standard IEC 61850 require a robust network designed with zero fail-over time. Two redundancy architectures defined in IEC 62439-3 are Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR). Additionally, the 61850 digital substation requires a high accuracy time source. Substation devices which synchronize to this time source expect the high accuracy time to be available regardless of network architecture.
Our report sought to examine implementations of IEEE 1588 Precision Time Protocol (PTP) Power Profile operating in a Parallel Redundancy Protocol (PRP) network with multiple master clocks. We also intended to illustrate the challenge of providing both the high accuracy time source and network redundancy necessary for a robust digital substation.
For this test setup, we connected two Intelligent Electronic Devices (IEDs) – a feeder protection relay and a line protection relay, two Ethernet switches in transparent clock mode, and two Global Positioning System (GPS) clocks configured as IEEE 1588 master clocks.
Typically, PRP operates by having the IED send identical Ethernet messages from each of its PRP ports. The PRP-enabled IEDs then use the duplicate-discard method to determine which received message to use. We tested this configuration to see how IEDs handle receiving different, rather than identical, PTP messages on each of their PRP ports.
We connected the protection IEDs via fiber optic Ethernet to the two switches. We labeled these small networks as Local Area Network (LAN)-A, and LAN-B. LAN-A incorporated one GPS clock (GPS-A) and one Ethernet switch (Switch-A).
The Ethernet switch functioned as a PTP Peer-2-Peer transparent clock as defined in IEEE 1588-10.3 and IEEE C37.238. LAN-B had the same setup as LAN-A, with a GPS clock (GPS-B) and an Ethernet switch (Switch-B) configured as a transparent clock.
We then connected an engineering workstation to each LAN for clock and switch management, as well as data collection. Figure 1 illustrates a simplified diagram of this test system.
1. Initial Configuration and Observations
Our first step was to properly configure the network and IEDs for PTP operation. We did this for LAN-A only, leaving LAN-B disconnected from the IEDs. We configured IED-1 first:
- IED-1 successfully synchronized with GPS-A.
We were able to verify this by viewing the IEEE 1588 Sync Source from the IED-1 human-machine interface (HMI), which was in slave mode. IEDs which are certified for IEEE 1588 will automatically determine which mode to operate in based on the definition of an Ordinary Clock which use the Best Master Clock algorithm as described later in the report.
- The Grandmaster Clock ID matched the ID of GPS-A.
We observed GPS-A Announce, Sync and Follow-Up messages arriving successfully at IED-1. Switch-A sent Path Delay (PDelay) requests, and the IED responded to the requests in a two-step fashion. That is, the IED would send the PDelay response with a PDelay follow-up message. We also observed the IED responding to PDelay requests from Switch-A, and GPS-A responded to PDelay requests from Switch-A.
2. Application of the Redundant Network
For this next step, we connected the redundant network, LAN-B, to IED-1. This step completed our parallel redundant network architecture:
- With both networks active and both GPS clocks active, we observed the master clocks sending master clock announce sequences.
IED-1 is observed in slave mode and synced with GPS-A, as seen on the IED’s HMI. PDelay requests from Switch-A were being answered by the IED and GPS-A. On LAN-B, PDelay requests from IED-1 were responded to by Switch-B. Likewise, PDelay requests from Switch-B were answered by IED-1. As expected in a PRP architecture, we observed identical frames leaving each IED via their network interface cards.
3. Removal of LAN-A
To simulate a LAN-A failure, we removed the fiber optic cable connecting IED-1 to the LAN-A port on Switch-A:
- After LAN-A was disconnected from IED-1, approximately seven (7) seconds passed before it synchronized with GPS-B.
- Switch-B and IED-1 continued to exchange PDelay messages uninterrupted.
- When LAN-A was re-applied, IED-1 synchronized with GPS-A within a few seconds.
4. Integration of IED-2
Recall that in our test network, IED-2 was connected via fiber optic Ethernet to Switch-A and Switch-B:
- IED-2 synchronized with GPS-A.
We verified this using the front panel HMI, which displayed the Grandmaster ID of GPS-A. Switch-A responded to PDelay requests from IED-2. Switch-B also responded to PDelay requests from IED-2.
5. Removal of LAN-A
To again simulate a LAN-A failure, we removed the fiber optic cable. This time we removed the cable connecting IED-2 to the LAN-A port on Switch-A:
- Even though the HMI on IED-2 indicated it was synchronized on the port for GPS-B, the Grandmaster Clock ID did not indicate GPS-B for a full minute after removal of LAN-A.
- When LAN-A was reconnected, re-synchronization to GPS-A occurred in a few seconds.
6. Best Master Clock
With LAN-A and LAN-B active, the IEDs received PTP announce messages from two sources. The IEDs, which function as Ordinary Clocks, determined which source to synchronize with by using the Best Master Clock (BMC) algorithm, defined in IEEE 1588 9.3.
7. PTP Priority
Step one of the BMC compares the user-configurable ‘priority’ attribute of the clock. A lower value of the priority indicates increased precedence for the algorithm. By setting GPS-B with a lower priority attribute, we caused the IEDs to synchronize to GPS-B. This step forced the IEDs to accept the best clock based on the priority setting. In a real-world application, one clock would normally be designated as the Primary, with a lower priority setting, and the second clock as a back-up.
To demonstrate the IEDs would function in a similar manner regardless of the loss of LAN-A or LAN-B we removed the LAN-B connection. After removal of LAN-B the IEDs synchronized to GPS-A. When LAN-B is restored, the IEDs once again synchronize to GPS-B within a few seconds. This step also demonstrated the function of the BMC based on the clock priority setting.
We were also able to measure the mean path delay for both Switch-A and Switch-B. With LAN-A and LAN-B active, the peer mean path delays were indicated by the switches as illustrated in Figure 2:
Due to the high accuracy of PTP, the mean path delay between each device may have a variance and may fluctuate between each observation. It is unclear why our IED-2 did not respond to PDelay requests from either Switch-A or Switch-B. This will require further investigation.
- The most important step of a properly functioning PTP network in a parallel redundancy architecture is correct configuration of each device in the system. Once the correct configuration is applied, PTP operates reliably under network failure conditions.
- The most reliable way to find the correct configuration is to build a system in a controlled laboratory environment before deployment.
- The digital substation of the future requires a robust network with high accuracy time synchronization.
We found that some information regarding configuration was not included in the manual. For example, it was not clearly stated that the IEDs utilize the BMC algorithm. Also, the location of the Grandmaster ID, an essential piece of information for synchronization verification, on the front HMI of IED-1 is not called out. Configuring the switches and GPS clocks required some trial and error to find the settings which worked best for each IED.
Our testing shows that you can have full seamless network redundancy along with time synchronization redundancy within the same network architecture. In future tests, we’ll examine other IEDs and add additional clocks. Once the PTP implementation of each device in a network is understood, operation in a PRP network can be made reliable, thus obtaining the high accuracy necessary for power systems SCADA, protection and event analysis.