Biz & IT —

Coolest jobs in tech (literally): running a South Pole data center

You know it's cold when you have to heat the air used to cool your data center.

Coolest jobs in tech (literally): running a South Pole data center
Photograph by National Science Foundation/F. Descamps

Steve Barnet is hiring, but not for an ordinary IT job. His ideal candidate "will be willing to travel to Polar and high altitude sites."

Barnet, interim Computing Facilities Manager for the Wisconsin IceCube Particle Astrophysics Center (WIPAC) at the University of Wisconsin, is looking to fill what may be the coolest Unix administrator job opening in the world—literally. Plenty of IT jobs exist in extreme and exotic locales, but the WIPAC IT team runs what is indisputably the world’s most remote data center: a high-performance computing cluster sitting atop a two-mile thick glacier at the South Pole.

The data center has over 1,200 computing cores and three petabytes of storage, and it's tethered to the IceCube Observatory, a neutrino detector with strings of optical sensors buried a kilometer deep in the Antarctic ice. IceCube observes bursts of neutrinos from cataclysmic astronomical events, which helps to study both "dark matter" and the physics of neutrinos.

That mission demands a level of reliability that many less remote data centers cannot provide. Raytheon Polar Services held the National Science Foundation’s Antarctic programs support contract until April. As Dennis Gitt, a former director of IT and communications services for the company puts it, a failure anywhere in the Antarctic systems could lose data from events in space that may not be seen again for millennia.

Running that kind of IT operation at one of the most hostile and remote locations in the world creates a whole set of challenges few in IT have ever experienced.

The few, the proud, the cold

Ralf Auer, left, and Steve Barnet, right, with James Roth of the IceCube team, dressed for a walk to the IceCube data center.
Ralf Auer, left, and Steve Barnet, right, with James Roth of the IceCube team, dressed for a walk to the IceCube data center.

A trip to the Amundsen-Scott South Pole Station is as close to visiting another planet as you can get on Earth, with “a palette of whites, blues, greys and blacks,” Gitt says. In summer, it feels like two o’clock in the afternoon 24 hours a day, which “can do interesting things to your diurnal cycle,” says Barnet. In winter, the outside world is lit only by moonlight and the Aurora Australis.

With a maximum population of 150 at the base during the Austral summer, South Pole IT professionals-in-residence are limited to a select few. And they don’t get to stay long—most of the WIPAC IT team only stays for a few months in the summer, during which they have to complete all planned IT infrastructure projects.

The rest of the year, WIPAC's US-based employees remotely walk the wintering members of the IceCube team—usually chosen for their physical science credentials, not for their IT skills—through tasks over satellite calls. Systems are monitored over a multiplexed Iridium satellite connection. “You just try to collect as much information as you can and do what you can remotely,” says Ralf Auer, WIPAC’s South Pole systems administrator.

The wintering-over team “can do a lot of the physical maintenance,” says Barnet, but routine IT tasks can sometimes feel foreign to physical scientists. The result is that Auer, Barnet, and the others back home have to visualize what the over-winter team is seeing in order to walk them through fixes. (Barnet compares it to being an air traffic controller talking down an airliner flown by a flight attendant.) So part of the team’s job is to make these hands-on tasks as simple as possible for the scientists, and to handle as much as possible remotely over whatever bandwidth they can get.

South Pole Station does have other IT support, but it’s not on WIPAC’s payroll. The general IT and communications support team for Amundsen-Scott peaks at 5-8 people during the summer and shrinks to 4-5 during the winter, according to Gitt. More stay at McMurdo Station, the main logistics support base in Antarctica: 35-40 at the peak of the research season in the summer, “depending on the projects,” he says. Smaller stations run by NSF may have only two or three IT and communications people total. (Lockheed-Martin just took over the NSF contract, but the award is currently under protest.)

The IceCube Lab at Amundsen-­Scott South Pole Station—home of the world's southernmost data center.
The IceCube Lab at Amundsen-­Scott South Pole Station—home of the world's southernmost data center.

There’s plenty of work to go around. During the summer, Gitt says, “You may have close to 1,000 scientists rolling into different stations getting their science done.” During the peak of the research season, McMurdo Station can surge to a population of around 1,100 people, some staying for only a few weeks based on their projects and funding. Every one of them wants to make sure they can get the IT they need when they need it—often all at the same time, since they have to reconfigure schedules around what the weather allows.

The low headcount and the urgency of just about every project—where the millions spent on a grant for a particular project may be lost if data can’t be collected or transmitted—demands that the few IT pros on hand in Antarctica have not just depth of knowledge, but skills across other fields as well. “Typically, they’re at an engineer level as opposed to a technician,” Gitt says. “They could fix almost anything.” One technician had to diagnose a problem with the Windows software package needed for a project on a researcher’s laptop—and the program was written and documented in French.

The IT teams in Antarctica don't operate in complete isolation; while Raytheon provided support, the Antarctic equivalent of a help desk was in Centennial, Colorado, for instance. But communications back to the US from the South Pole are akin to communication from space—in fact, South Pole stations use the same satellite network for broadband communications as does the International Space Station.

The most reliable form of communication available is the Iridium satellite network. Individual Iridium connections aren’t exactly blazing—they support a data rate of only about 2,400 bits per second. But according to Gitts, Raytheon did a lot to coax as much bandwidth as possible out of Iridium, including multiplexing Iridium connections, doing compression to shrink the size of e-mails—even adding a wireless server with file-sharing and e-mail services to containerized Iridium ground stations to support some of the smaller field stations.

“We even dropped the package size down further in size and put it on some of the traverse tractors," Gitt says—providing a data lifeline to expeditions crossing the frozen continent.

The higher-bandwidth access is limited to about 10 hours a day of broadband coverage from NASA’s Tracking and Data Relay Satellite System (TDRSS) and the GOES-3 Satellite —a weather satellite launched in 1978 that lost its weather imaging capabilities and now provides 1-megabit per second data transmission for eight hours a day. TDRSS provides the most bandwidth, with transmission speeds of up to 150 megabits per second.

But catching signal from those satellites at 90 degrees south latitude requires some serious work. Two of the IT support team for the South Pole Station are a satellite engineer and a technician, because keeping communications up often requires more than one set of hands. “With TDRSS," says Gitt, “they are actually working four or five contacts a day, where ground station is swinging over the dish to catch the next satellite as it comes over the horizon.”

And they don’t get very high above the horizon; ground station dishes are almost always pointed nearly parallel to the ground.

Channel Ars Technica