Jul 282017
 

I have two “live” uninterruptible power supplies (UPSs) in my study. One powers my servers and network equipment, the other, my workstation. They are identical models: APC Back-UPS Pro 1500. They were both purchased recently, just a few months ago, replacing units that were more than ten years old, and for which replacement batteries were no longer available from a reliable source at a reasonable cost.

So here is the thing: for the second time in a month, the two UPSs decided to do a self-test at the exact same moment.

Now let me make it clear: apart from being identical models and located in the same room, the two units are not in any way connected. They are not synchronized. There is no data connection between the two. When two UPSs simultaneously go offline and start running on battery power, this usually means a power event; but there was no power event, utility power was steady at 124 volts, with no brownout or interruption.

Looking at various logs, I think I solved the mystery. There was a power event recorded on my server on June 30, a brief brownout with line voltage dropping all the way to 74 V:

19:14:33 122.0 13 27.0 100 2910 OL
19:14:34 74.0 11 26.0 100 2910 OB
19:14:35 74.0 11 26.0 100 2910 OB
19:14:36 74.0 11 26.0 100 2910 OL DISCHRG
19:14:37 74.0 11 26.0 100 2910 OL DISCHRG
19:14:38 121.0 14 26.5 100 2910 OL

And lo and behold, my workstation’s UPS sensed the same event at the same time:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="APC Data Service" /> 
    <EventID Qualifiers="0">0</EventID> 
    <Level>4</Level> 
    <Task>0</Task> 
    <Keywords>0x80000000000000</Keywords> 
    <TimeCreated SystemTime="2017-06-30T23:14:33.228802500Z" /> 
    <EventRecordID>6000</EventRecordID> 
    <Channel>Application</Channel> 
    <Computer>VIKTOR</Computer> 
    <Security /> 
  </System>
  <EventData>
    <Data>PowerEvent handled successfully by the service.</Data> 
  </EventData>
</Event>

When I look at the dates of the two UPS self-test events, the first happened just 44 minutes less than exactly two weeks after this brownout; the second, just 44 minutes less than two weeks after the first.

The UPS manual says that this UPS performs a self-test every two weeks. So what’s with the 44 minutes? It can’t be uncalibrated clocks, since the two UPSs remain in sync to the second. Could they be synchronized to utility power? Perhaps, but that does not explain the 44 minutes. The frequency stability of the East Coast grid is much better than 0.2%. In fact, it is purposefully maintained to have an accurate average frequency that deviates from the standard by less than a few parts per million over the course of a day.

So my best guess is that the UPSs are factory configured to perform a self-test every 13 days, 23 hours and 16 minutes. They do this in sync because they are synchronized to the same time source, namely the utility power frequency.

 Posted by at 9:19 pm

  2 Responses to “The mystery of my synchronized UPSs”

  1. Here is a possibility – a typo in the code. 2 weeks is 1209600 seconds. your interval is 1206900 seconds.

  2. Could be, but the actual interval is closer to 1206971 seconds. Or maybe 1206972:

    voltlog-20170714:18:30:49 123.0 14 27.2 100 2910 OL DISCHRG
    [...]
    voltlog-20170714:18:30:56 123.0 10 25.2 100 3345 OL DISCHRG

    vs.

    voltlog-20170728:17:47:00 124.0 14 27.2 100 2784 OL DISCHRG
    [...]
    voltlog-20170728:17:47:09 124.0 10 25.2 100 3345 OL DISCHRG