I've worked in the IT business for a long time. One of the hot item issues that has reared its' head in recent years is the issue of providing sufficient electrical power and cooling to data centers. As everyone in the business knows, the ongoing drop in hardware prices coupled with their increases in computational power (Moore's Law) have made it possible for companies to amass tremendous computing power at prices which used to cost a mint. This has led to configurations of as many as 80+ servers in a single server rack and for companies to have racks and racks of servers.
What amuses me is that even though many in the business foresaw that the trend towards commodity, cluster oriented computing was going to happen and were patting themselves on the back for knowing about Moore's Law, what nobody ever seems to have realized is that these developments would eventually put immense pressure on those dreary, unsexy, below the radar issues of data center power and cooling until it actually came along. Then hey wait a minute! Then people realized that this was going to cause a problem with powering and cooling data centers. So much for how much people know about the future. Now the federal government has gotten into the act via the EPA.
I decided to write this to share my company's experience with power and cooling over the past 13 years. Briefly, I work as a sysadmin in a resonably large data center. This data center was designed in 1994 to accomodate an IBM ES9000 mainframe, 20 or so SUN servers, some Compaq servers with Microsoft Windows NT, and a few other sundry servers. We were an early adapter of Linux and started circa 1997-1998 to create relatively small clusters of 128 nodes for seismic data processing. These clusters have gotten bigger and bigger over time.
The first environmental problem we encountered was that in our third generation of cluster builds, we were using Dell 450 workstations stacked 11 or so in a rack. We had 110 volt strips installed on each rack. When we plugged more than 4 workstations into a power strip, that blew the circuit breaker on each power strip. The data center manager then decided to have the entire data center rewired to 208-230 volt power. We also plugged fewer workstations into each power strip and also used 220 volt connections at the rack. This stopped the tripping of the power strips.
The next problem we ran into was about 2 years later. By this time our clusters had grown considerably. We had 500 Dell 1750 1U servers installed at around 38 servers per rack in addition to about 520 of the above mentioned Dell 450 workstations mounted 11 or so to each rack. Up through this time the original power and cooling configuration in our data center, which was composed of 6 PDU's and 7 20 ton air handlers was handling the load all right. Then around January 2005 we installed 500 Dell 1855 blade servers at 50 blades per rack and we deinstalled about 100 of the old Dell 450's. That gave us about 1,400 nodes in our clusters. We had some extra 225 KVA PDU's installed in the data center to provide the extra power needed to handle the new electrical load.
We set up the 1750's and the 1855's in a cold aisle enclosure, with walk in doors on either side, where we had floor fans (we have a raised floor) forcing up air into the enclosure and the air would flow out through the servers. We had the servers facing outwards so that we had the hot aisles on the outsides of the enclosures. Due to the fact that we had given up part of the data center for some corporate meeting rooms, we were also now working with the handicap of having to deal with a smaller area of floor space.
It was at this point that the environmentals in our data center started to break down. We discovered that the data center was all right when there was no work going on, or even when we were employing up to about 500 of the nodes. Anything over that and the data center began to really warm up. When we were employing all 1,000 of the 1750's and 1855', it really got hot! In the early days of this new cluster install, we routinely received hot temperature alarms from our servers.
The data center manager, my supervisor and I tried a number different strategies for cooling the nodes down. We did all the best practices. We sealed off every last crevice in the enclosed area, including using blanking panels on the server racks. We sealed off nearly every open space we could find in floor tile cutouts. We put porous ceiling tiles to allow the hot air to escape more freely. We put fans on top of the racks to help blow the volumes of hot air out of the racks. This all more less stabilized the clusters, but the data center got to be so hot that we had to have fans running if we were working in there when the clusters were fully employed. The ambient temperature around the back wall where our clusters were deployed was 96 degrees Fahrenheit when the clusters were running at full power.
Occasional visitors to the data center inevitably complained about how hot it was in the data center. Weren't we doing something about this? My supervisor was complaining that he wanted to hire an engineering student who knew about thermodynamics and who could tell him how much heat his clusters were generating. He didn't know. He spent some money and sent me to a seminar on cooling data centers that was held here in town. It was a fairly well attended event, but even after hearing these consultants talk for half a day, they didn't tell me anything I didn't know already. They certainly didn't tell me how to figure out how much cooling we were going to need to deal with our data center. What a bunch of worthless bastards consultants.
Meanwhile the data center manager was going around telling people that nothing could be done about the matter. He would make statements saying that "this air handler over there can cool all of those Dell 450's". Great. For a long time, I trusted his judgment, but it was becoming clear that I couldn't do that anymore. It was time to start second guessing his judgment.
Finally the news came that broke the camel's back and forced the matter. We received news that another cluster expansion was coming, this time with another 1,000 nodes. The bid went out and Dell won again with 1855 blades. This time I knew that something had to give. The data center manager was a nice guy, but he was getting up there in age and this new world of computing had put us in unexplored country for a long time. For a long time I had heard his pronouncements, but a question finally dawned on me. How did he "know" that the air handler in the corner could handle "all of those Dell 450's"? it was time for me to start doing some research.
Thank the Heavens for the Internet! At the end of the week one Friday night in the summer of 2005, I sat there for hours and hours looking for information. Finally I found some websites which told me what I was looking for. What I found was that 1 watt of electrical power consumed generates 3.41 BTU's of heat. Suddenly the world became clear to me and I knew what to do.
I went around the data center and added up all of the equipment in there, including the Windows servers which was not under our jurisdiction. I did some quick calculations and my jaws dropped. By my math, we were about 60 tons of air short in the data center! No wonder it was so hot in there. We needed three full 20 ton air handlers in there right now just to stay even. I wrote an email where I detailed my findings to the relevant parties, doing my very best to be diplomatic towards the data center manager. My writings worked. When we did the install of the 1,000 nodes, the data center manager had 157 tons of new cooling added to our current stable of air handlers. Now we have about 295 tons of cooling in our data center.
Here are the yardsticks:
1) 1 watt of electrical power generates = 3.41 BTU's of heat. I simply use 3.5 BTU's of heat, just to make the math come out nice. It only adds 3 percent to the actual number and you can use that as a safety pad in the event something goes wrong.
2) 1 ton of cooling = 12,000 BTU's of heat.
So as an example, Dell's 1855 blade servers generate 300 watts of power at full utilization. When 10 1855's are set in a chassis, then the chassis and the blades consume 3,600 watts. Since we were installing 1,000 1855 blades, the cooling required would be:
100 * 3,600 watts = 360,000 watts.
360,000 watts * 3.5 BTU's / watt = 1,260,000 BTU's of heat.
1,260,000 BTU's / 12,000 BTU's per ton of cooling = 105 tons of cooling.
So we were going to need 105 tons of cooling to cool 1,000 Dell 1855 blades. Of course the number is slightly smaller than that because the actual watt / BTU ratio is 1 / 3.41 and not 1 / 3.5, but you should get the picture.
Again, the 1 watt = 3.41 BTU's of heat generated is far and away the most important thing you will ever need to know about cooling a data center! All other issues, whether you are arguing about whether to use a raised floor or not, blanking panels, sealing off floor tile cutouts, and so on, pale in insignificance to that fact of physics. Of course all other industry best practice cooling issues do matter, but they ultimately matter on the only margin. What you really need to know is that formula above.
Now that I knew this, I knew I was cooking with fire. We installed and powered up our new cluster and guess what? It worked great! We now have about 2,300 servers in our data center, but even when we are running our clusters flat out, the ambient temperature in the hottest parts of the data center near the walls of the cluster get to be only about 83-84 degrees fahrenheit. Now some of you IT types reading this might be thinking that this is bananas, that you would never tolerate operating a data center at this temperature. What you need to know is that the rest of the data center is perfectly fine, with ambient temperatures of 70-72 degrees fahrenheit just yards away from where the hot spots are. People aren't complaining about how hot it is or sweating their butts off anymore when they go in there. I have discovered over the past few years that computer equipment is often a lot more fault tolerant that many people are led to believe or are willing to tolerate (job or career wise) politically. In January 2007, I went to Algeria for my company and literally blew out cups full of desert sand out of servers which had stayed up and operational for 2+ years.
part II is to come shortly.
Ciao for now - Wizard.
Posted by The Mighty Wizard at March 5, 2007 10:17 PM