The leap second is coming by Tomonori TAKADA [APRICOT 2015]

Editor's Notes

1. My name is Tomonori Takada and I am a system engineer from QTNet. QTNet is a(エイ) telecom carrier serving individual and corporate customers mainly in the Kyushu region of Japan.
2. The main topic of today’s presentation is how leap seconds are addressed(アドレスド) in NTP.
3. IERS announced the insertion of a leap second for this year.
4. NTP has an identifier called the Leap Indicator to correct the time with leap seconds. For leap second insertions in 2012 and this year, a packet with a(ア) binary value of “01(ゼロワン)” is distributed from the higher NTP server. Once a node receives(レシーブス) the Leap Indicator, it uses(ユーズス) the OS or kernel functionality on its own to adjust the time when a leap second is inserted. The Leap Indicator only makes notification of the(ジ) occurrence(オカランス) of leap second insertions; it is the OS or kernel that actually corrects the time.
5. The last time a leap second was inserted, several problems were reported, particularly in Linux environments. One of the problems was that the kernel would hang and reboot when it received the Leap Indicator. Another problem was that CPU usage skyrocketed(スカイロケテッド) after a leap second was inserted in an environment where specific applications were running. The first problem occurred with older versions(バージョンズ) of the OS or kernel. In fact, various distributors had given warnings beforehand. We were therefore able to prevent this problem by upgrading our(アワ) OS and kernel. The second problem occurred with the newer versions of the OS or kernel. In either case, bugs in the OS or kernel came to the surface with the leap second insertion and the Leap Indicator.
6. Before going over QTNet’s response in 2012, let me talk about the(ジ) operation of NTP at QTNet. QTNet runs two time servers as Stratum 1. One is JJY Server. It acquires(アクワイアス) time information from NICT via(バイア) PSTN. The other is GPS Server. It acquires time information from GPS. These two servers are appliance servers. We run three Linux time servers as Stratum 2. As clock references, these servers use our two Stratum 1 servers as well as the time server provided by NICT via(バイア) the(ジ) Internet(インターネット). They have been configured as peers to one another to equalize(イコライズ) the time among the(ジ) three(スリー) servers(サーバーズ). Over 100 QTNet servers have been configured to use these(ジーズ) Stratum 2 servers as clock references.
7. Now let me talk about how QTNet responded to the leap second insertion in 2012. QTNet had decided not to use the Leap Indicator for leap second adjustment for two reasons: The first reason is that, as I have mentioned before, we had received information beforehand on the hang-up of older versions of the OS or kernel when receiving the Leap Indicator. The second reason is that QTNet’s Stratum 1 time servers had the ability to adjust leap seconds without using the Leap Indicator. This is called “Gradually Adjust mode.” With Gradually Adjust mode, time servers put a clock back gradually about two hours before the leap second insertion and complete a one-second adjustment by such time.
8. Based on this policy, QTNet changed a part of the NTP architecture before the leap second insertion. Specifically, the NICT time server was excluded as a clock reference, because, it distributes the leap indicator. This means that our Stratum 2 servers will use only the time servers that have the “Gradually Adjust mode”, and do not involve the insertion of the Leap Indicator.
9. Now, I would like to show you what happened when a leap second was inserted in 2012. This slide shows the status as of 9:00 a.m. on the day of the leap second insertion. Fortunately, QTNet did not encounter any problems(プロブレムス) related to the Leap Indicator, however another problem occurred. This problem was that the Stratum2 server’s peer status failed. The question is: why did the peer status fail?
10. There can be two reasons: First, the JJY and GPS Servers had two different speeds of putting the clock back. As we checked with our product vendor later, we found out that the JJY Server was designed to start the leap second adjustment 125 minutes before the insertion, while the GPS Server was designed to start 120 minutes before the insertion. This could widen(ワイドン) the difference in clock time between Stratum 2 servers that use the JJY and GPS Servers(サーバーズ) as clock references. Secondly, the polling interval of Stratum 2 for using Stratum 1 as a clock reference differed (among Stratum servers). The interval had been variably(バリアブリィ) configured between 64 seconds and 1(ワンサウザンズ),024(トゥエンティフォー) seconds. This usually causes no problems, but the longer the polling interval is, the longer it takes to correct the time, when the Stratum 1 Server put the clock back gradually. As a result, there could have been a(ア) gap in clock time between Stratum 2 Servers even though they used the same time source.
11. Now, what are we going to do to address the leap second insertion this year? QTNet is currently considering two options: Plan A is to use the Leap Indicator. Plan A is based on the assumption that we will upgrade the(ジ) OS and kernel of all servers to the latest version beforehand. This is what they should always be, but we have some servers with older versions(バージョンズ) of the OS or kernel in reality. Plan B is not to use the Leap Indicator but, instead use the “Gradually Adjust mode” of Stratum 1 servers , just like we did in 2012. We believe that we need to take the following steps to remedy(レメディ) the problem that we experienced. Step 1 is to use only one Stratum 1 server to unify the speed to put the clock back at Stratum 1. Step 2 is to shorten and unify the polling intervals for using Stratum 1 as a clock reference so that the clocks of the Stratum 2 servers are in sync with one another.
12. In this presentation, I quickly looked back on the problems that occurred during the leap second insertion in 2012 and how QTNet responded. I also talked about QTNet’s plans for the leap second insertion scheduled for this year. Whatever option we choose, it is important to check the specifications and constraints(コンストレインツ) of the time servers, ntpd, and the OS and kernel. Whether we like it or not, a leap second will be inserted this year. I would like to wish everyone good luck in getting through the leap second insertion this year.
13. Thank you for listening.