Stanford center turns to Sun Blackbox for extra capacity

by James Niccolai

January 30, 2008 —

When the Stanford Linear
Accelerator Center wanted to increase its computing capacity last year it
considered a satellite datacenter or an extension to the existing building,
but in the end it chose a faster and more novel approach: it ordered a datacenter
in a box.

The center, which does high-energy physics research for the U.S.
Department of Energy, was one of the first customers for Sun Microsystems'
Project
Blackbox, which takes standard shipping containers 20 feet by 8 feet by
8 feet (6.1m x 2.4m x 2.4m) and turns them into mini datacenters that can be
delivered and operational in a few weeks, according to Sun.

SLAC's Blackbox was delivered in July last year and was up and running by September.
Aside from a few challenges -- like figuring out how to service the unit when
it's raining -- the center is pleased with its choice and in the process of
installing a second unit.

It is one of four customers that Sun identified Tuesday to illustrate momentum
for Project Blackbox. It also renamed the product the Sun Modular Datacenter,
or Sun MD. The boxes start at US$559,000 without the "payload," or
the computers inside, and Sun will ship them around the world for an extra fee.
Delivery to Amsterdam by air, for example, costs about $15,000.

SLAC turned to Project Blackbox because it needed to expand its compute capacity
fairly quickly. "The datacenter here was at its capacity, especially in
terms of the electrical service to the building and the amount of heat we could
take back out. And the experiments needed their next year's worth of computing,"
said Chuck Boeheim, SLAC's assistant director of computing services.

Modifying its existing datacenter to accommodate another major electrical feed
and "chiller plant" would have taken one to two years and cost several
million dollars. "We also looked at doing a satellite datacenter in a smaller
building, but with the approvals and lead time that was a couple of years out
as well. Blackbox was something we could do very quickly," Boeheim said.

The raw shipping containers are customized by a subcontractor, and Sun typically
installs the payload before delivery. SLAC's Blackbox arrived on a flat-bed
truck last July, fitted with 252 Sun Fire X2200 rack-mount servers, the same
type it uses in its datacenter. Sun had also wired the servers to a Cisco Catalyst
6509 switch that SLAC provided before delivery.

Boeheim described the process in a white
paper on his Web site, along with photos and time-lapse videos that show
the box being hoisted into place by a crane.

Customers can put other vendors' hardware in the unit, but not all equipment
will fit. SLAC bought some Dell
PowerEdge 1950 servers that it wanted to put in its second Blackbox, but
they were too long for the server racks. Instead it will move some existing
Sun servers to the box from its datacenter, and put the Dell machines inside
the building instead.

The computing center already had a 4-megawatt substation serving its area of
the site. "So it was a fairly simple matter to put a 220-kilowatt power
pane on the outside of that and bring up a fairly standard chiller unit and
put that on a concrete pad next to the Blackbox. Both of those things could
just be fork-lifted in and connected up," Boeheim said.

SLAC wanted to do a rigorous installation that would last several years, so
it spent a few weeks getting the unit hooked up. It had to reconnect about half
the power cords which had shaken loose during the truck ride. Aside from that
the installation went smoothly and the machines were turned on for batch work
Sept. 25, about two months after delivery. The 60-ton chiller and the outside
power unit were also delivered in that time.

Servicing the unit has presented some challenges. "We're trying to figure
out how to do it on a day like today when it's pouring with rain," Boeheim
said recently. "The wind doesn't have to be blowing very hard for the rain
to blow in the end, and you pretty much have to open both ends of the box to
service it,"

It's quite cramped in the box, he said, and both ends need to be open so that
a person can maneuver when a server rack is slid out into the narrow central
aisle.

"Our stance right now is to wait for a clear day to open it up,"
he said. Two people are needed to service the box in case one gets trapped inside
or injured, he wrote in the white paper on his Web site. SLAC put a phone in
the box in case someone gets locked in.

The first Blackbox gave the center a roughly 25 percent boost in computing
capacity, and the servers have been running as smoothly as their counterparts
inside the datacenter, Boeheim said. "Once we got them turned on they just
ran. We went a full 30 days not even opening the thing." He put rack servers
in the box rather than storage gear, because storage gear tends to need servicing
more often, he said.

Sun's Blackbox won't be for everyone. Brookhaven
National Laboratory, on Long Island, New York, considered a Blackbox but
decided on a new wing for its datacenter instead, Boeheim said. SLAC's experience
may have been easier than that of an average business because it is a federal
government site located on county land. A business in a city may need more planning
permissions and permits.

A few other companies offer comparable products, such as the Ice
Cube from Rackable Systems, and the Infrastructure
Express from American Power Conversion.

Sun said it is happy with the momentum for Blackbox, although it wouldn't say
how many customers it has. The other three it named are Hansen
Transmissions, a Belgian industrial manufacturer that is using one at a
new plant in India; Mobile
TeleSystems, Russia's largest mobile operator; and the Radboud
University Nijmegen Medical Centre in the Netherlands.

Most of the customers had no room to add capacity in their datacenters, said
Darlene Yaplee, vice president for integrated platforms in Sun's systems marketing
group. "Half the customers are using it while they build extra datacenter
capacity, and then it will remain for disaster recovery," she said.

The next size up in standard shipping containers is 40 feet long. "We
could do 40 feet technically, but most people have been happy with 20 feet,"
Yaplee said. Boeheim said he'd probably choose a wider box rather than a longer
one, which would accommodate his Dell 1950 servers.

IDG News Service