I suppose I am a little late in writing about the news that Halliburton's CEO, Dave Lesar, will be moving the company headquarters to Dubai in the United Arab Emirates.
Here are a few items to think about:
1) KBR, Halliburton's arm which has been serving America'a armed forces for overseas ventures, makes only 3 percent profit on its contracts. Meanwhile, dozens of KBR contractors and employees have lost their lives in Iraq.
2) The Democrats in Washington want to make hay about Lesar's move and of the "attachment" of the evil oil and gas industry to the Bushies. I would like to remind everyone that KBR stands for Kellogg, Brown and Root. Brown and Root founders George and (particularly older brother) Hermann Brown, were deeply in bed with Lyndon Johnson, to the point where B&R was Johnson's kept company and Johnson was the Brown Brother's kept politician. This was all made clear by Robert Caro in his book Means of Ascent.
3) Corporations owe no more to broader society other than to obey its laws! Those laws might stipulate that evil corporations have to curb their dumping of pollution into the environment, that they have to provide safe working environments, pay minimum wages or follow other work rules, create safe products, don't deceive their customers, pay their taxes, follow political campaign finance limits, not do business with nation states or regimes which their host country deems to be enemies, and so on. As to whatever "morality" that people find in laws which are laid down, well that is up to the broader public.
Some Ralph Nader types might well complain that corporations have far too much influence in the political life of modern day developed nations and that their influence must be curbed. However it seems to me that the very fact that we have piled on all these rules on corporations strongly suggests otherwise. If Mr. Lesar really has something to hide, then the U.S. Congressional Democrats have the power to subpoena Mr. Lesar and squeeze him via rent extraction for campaign contributions crucify him before the television cameras before throwing him in jail for alleged misdeeds.
As always, I really am a nice guy, so have a nice day!
Wizard
And what kind of person did you think The Mighty Wizard is?
You are an alchemist. "Alchemists are those who seek Truth. Judges, scientists, and philosophers (who are a kind of scientist, though they have forgotten it) and even a few clergymen are Alchemists."
Naturally!
But that isn't the end of my tales on power and cooling in data center. Now with our proud new cluster in place, we were in business. Things purred along nicely for about six months. Then in June 2006, I was sitting in the data center around noon one day with some summer college interns. The data center manager was on vacation and everyone else was out to lunch - literally. Suddenly I noticed a flickering in the light fixtures. The flickering ran up and down the entire length of the computer room floor. The flickering then stopped, but it then repeated about 1 minute later. A feeling of dread came over me. I have worked in data centers for many years and have been through many electrical problems and power outages, but I had never seen anything like that before.
Moments later the alarms went off on the UPS's and the batteries started draining. I fired off an emergency email to all IS systems personnel summoning them to the data center right that moment. That was in fact the biggest problem - convincing your coworkers that there is a problem when they aren't there. My servers? They're still up! What the **** are you worried about?
And so it was. I contacted my boss and he told me to start shutting down our stuff. Meanwhile after about 10 minutes I heard a deep sounding collective POOF that echoed across the data center. I looked in horror as I saw that all of the air handlers had lost power simultaneously. I resent my emails to everyone to get them into the data center to start some kind of orderly shutdown, considering that we were now on a extremely short time from to try to attain one.
We eventually were able to get things back together. We had blown some fuses on our air handlers and had lost a phase of electrical power. We turned on the data center after some hours and things ran over the weekend. Then we ran into the same problem over the weekend and again during the next week.
Eventually this led to meetings and pow-wows. The local utility basically said that we were pushing our limits on our allocated line feeds and we in turn queried the electrical contractor who put the permits in to do our electrical work for our cluster expansions. It seems the contractor had rushed the process and had assured us that the necessary power would be there when in fact there were no such assurances. We eventually had to schedule another shutdown as part of an office wide complex shutdown (we rent out of a major office complex which has three buildings in it). Then the power company had to add the needed infrastructure to give us an increase from 1,600 amps to 2,500 amps (at 480 volt , three phase power).
And so it was. What lessons were there to be had in all of this?
1) As your power and cooling demands grow, the potential scope for **** ups will also grow. At first it was only blown power strips at the rack level. Eventually it was overheating the entire data center and overloading the electrical line feeds into the building. You need to do your homework in advance to avoid these problems.
2) Study these issues. Become at least mildly familiar with electrical power and cooling terms, concepts and issues.
Enough for now.
I've worked in the IT business for a long time. One of the hot item issues that has reared its' head in recent years is the issue of providing sufficient electrical power and cooling to data centers. As everyone in the business knows, the ongoing drop in hardware prices coupled with their increases in computational power (Moore's Law) have made it possible for companies to amass tremendous computing power at prices which used to cost a mint. This has led to configurations of as many as 80+ servers in a single server rack and for companies to have racks and racks of servers.
What amuses me is that even though many in the business foresaw that the trend towards commodity, cluster oriented computing was going to happen and were patting themselves on the back for knowing about Moore's Law, what nobody ever seems to have realized is that these developments would eventually put immense pressure on those dreary, unsexy, below the radar issues of data center power and cooling until it actually came along. Then hey wait a minute! Then people realized that this was going to cause a problem with powering and cooling data centers. So much for how much people know about the future. Now the federal government has gotten into the act via the EPA.
I decided to write this to share my company's experience with power and cooling over the past 13 years. Briefly, I work as a sysadmin in a resonably large data center. This data center was designed in 1994 to accomodate an IBM ES9000 mainframe, 20 or so SUN servers, some Compaq servers with Microsoft Windows NT, and a few other sundry servers. We were an early adapter of Linux and started circa 1997-1998 to create relatively small clusters of 128 nodes for seismic data processing. These clusters have gotten bigger and bigger over time.
The first environmental problem we encountered was that in our third generation of cluster builds, we were using Dell 450 workstations stacked 11 or so in a rack. We had 110 volt strips installed on each rack. When we plugged more than 4 workstations into a power strip, that blew the circuit breaker on each power strip. The data center manager then decided to have the entire data center rewired to 208-230 volt power. We also plugged fewer workstations into each power strip and also used 220 volt connections at the rack. This stopped the tripping of the power strips.
The next problem we ran into was about 2 years later. By this time our clusters had grown considerably. We had 500 Dell 1750 1U servers installed at around 38 servers per rack in addition to about 520 of the above mentioned Dell 450 workstations mounted 11 or so to each rack. Up through this time the original power and cooling configuration in our data center, which was composed of 6 PDU's and 7 20 ton air handlers was handling the load all right. Then around January 2005 we installed 500 Dell 1855 blade servers at 50 blades per rack and we deinstalled about 100 of the old Dell 450's. That gave us about 1,400 nodes in our clusters. We had some extra 225 KVA PDU's installed in the data center to provide the extra power needed to handle the new electrical load.
We set up the 1750's and the 1855's in a cold aisle enclosure, with walk in doors on either side, where we had floor fans (we have a raised floor) forcing up air into the enclosure and the air would flow out through the servers. We had the servers facing outwards so that we had the hot aisles on the outsides of the enclosures. Due to the fact that we had given up part of the data center for some corporate meeting rooms, we were also now working with the handicap of having to deal with a smaller area of floor space.
It was at this point that the environmentals in our data center started to break down. We discovered that the data center was all right when there was no work going on, or even when we were employing up to about 500 of the nodes. Anything over that and the data center began to really warm up. When we were employing all 1,000 of the 1750's and 1855', it really got hot! In the early days of this new cluster install, we routinely received hot temperature alarms from our servers.
The data center manager, my supervisor and I tried a number different strategies for cooling the nodes down. We did all the best practices. We sealed off every last crevice in the enclosed area, including using blanking panels on the server racks. We sealed off nearly every open space we could find in floor tile cutouts. We put porous ceiling tiles to allow the hot air to escape more freely. We put fans on top of the racks to help blow the volumes of hot air out of the racks. This all more less stabilized the clusters, but the data center got to be so hot that we had to have fans running if we were working in there when the clusters were fully employed. The ambient temperature around the back wall where our clusters were deployed was 96 degrees Fahrenheit when the clusters were running at full power.
Occasional visitors to the data center inevitably complained about how hot it was in the data center. Weren't we doing something about this? My supervisor was complaining that he wanted to hire an engineering student who knew about thermodynamics and who could tell him how much heat his clusters were generating. He didn't know. He spent some money and sent me to a seminar on cooling data centers that was held here in town. It was a fairly well attended event, but even after hearing these consultants talk for half a day, they didn't tell me anything I didn't know already. They certainly didn't tell me how to figure out how much cooling we were going to need to deal with our data center. What a bunch of worthless bastards consultants.
Meanwhile the data center manager was going around telling people that nothing could be done about the matter. He would make statements saying that "this air handler over there can cool all of those Dell 450's". Great. For a long time, I trusted his judgment, but it was becoming clear that I couldn't do that anymore. It was time to start second guessing his judgment.
Finally the news came that broke the camel's back and forced the matter. We received news that another cluster expansion was coming, this time with another 1,000 nodes. The bid went out and Dell won again with 1855 blades. This time I knew that something had to give. The data center manager was a nice guy, but he was getting up there in age and this new world of computing had put us in unexplored country for a long time. For a long time I had heard his pronouncements, but a question finally dawned on me. How did he "know" that the air handler in the corner could handle "all of those Dell 450's"? it was time for me to start doing some research.
Thank the Heavens for the Internet! At the end of the week one Friday night in the summer of 2005, I sat there for hours and hours looking for information. Finally I found some websites which told me what I was looking for. What I found was that 1 watt of electrical power consumed generates 3.41 BTU's of heat. Suddenly the world became clear to me and I knew what to do.
I went around the data center and added up all of the equipment in there, including the Windows servers which was not under our jurisdiction. I did some quick calculations and my jaws dropped. By my math, we were about 60 tons of air short in the data center! No wonder it was so hot in there. We needed three full 20 ton air handlers in there right now just to stay even. I wrote an email where I detailed my findings to the relevant parties, doing my very best to be diplomatic towards the data center manager. My writings worked. When we did the install of the 1,000 nodes, the data center manager had 157 tons of new cooling added to our current stable of air handlers. Now we have about 295 tons of cooling in our data center.
Here are the yardsticks:
1) 1 watt of electrical power generates = 3.41 BTU's of heat. I simply use 3.5 BTU's of heat, just to make the math come out nice. It only adds 3 percent to the actual number and you can use that as a safety pad in the event something goes wrong.
2) 1 ton of cooling = 12,000 BTU's of heat.
So as an example, Dell's 1855 blade servers generate 300 watts of power at full utilization. When 10 1855's are set in a chassis, then the chassis and the blades consume 3,600 watts. Since we were installing 1,000 1855 blades, the cooling required would be:
100 * 3,600 watts = 360,000 watts.
360,000 watts * 3.5 BTU's / watt = 1,260,000 BTU's of heat.
1,260,000 BTU's / 12,000 BTU's per ton of cooling = 105 tons of cooling.
So we were going to need 105 tons of cooling to cool 1,000 Dell 1855 blades. Of course the number is slightly smaller than that because the actual watt / BTU ratio is 1 / 3.41 and not 1 / 3.5, but you should get the picture.
Again, the 1 watt = 3.41 BTU's of heat generated is far and away the most important thing you will ever need to know about cooling a data center! All other issues, whether you are arguing about whether to use a raised floor or not, blanking panels, sealing off floor tile cutouts, and so on, pale in insignificance to that fact of physics. Of course all other industry best practice cooling issues do matter, but they ultimately matter on the only margin. What you really need to know is that formula above.
Now that I knew this, I knew I was cooking with fire. We installed and powered up our new cluster and guess what? It worked great! We now have about 2,300 servers in our data center, but even when we are running our clusters flat out, the ambient temperature in the hottest parts of the data center near the walls of the cluster get to be only about 83-84 degrees fahrenheit. Now some of you IT types reading this might be thinking that this is bananas, that you would never tolerate operating a data center at this temperature. What you need to know is that the rest of the data center is perfectly fine, with ambient temperatures of 70-72 degrees fahrenheit just yards away from where the hot spots are. People aren't complaining about how hot it is or sweating their butts off anymore when they go in there. I have discovered over the past few years that computer equipment is often a lot more fault tolerant that many people are led to believe or are willing to tolerate (job or career wise) politically. In January 2007, I went to Algeria for my company and literally blew out cups full of desert sand out of servers which had stayed up and operational for 2+ years.
part II is to come shortly.
Ciao for now - Wizard.
Some months ago, I put in for a TXPIA request to Metro for their ridership numbers from 2003 - 2006. Just recently I put in for a second ridership request backdating the new request to 1997. I am in the process of putting together a spreadsheet which will have all ridership data for all bus routes, along with Metro rail (including boardings at each station). The dataset as it is will have about 20,000 pieces of data in it, but I am optimisitic I can have an incomplete version posted within 2 weeks and a completed on done within 2 months.
The TXPIA request comes back in the form of hardcopy printouts, ergo I tried to scan them in and post the numbers online. After getting a few unsatisfactory results, it dawned on me that it would be much more interesting to simply go through the effort of creating a spreadsheet and then lots of things could be more readily observed, such as correlating whether people do ride transit more often when gas prices are high.
I am also in the process of gathering together old information on Metro system maps from the past. The effects of route changes can then be noticed.
Enough for now. This will keep me busy for a while.
Kevin at BlogHouston writes that 10 condo developments are going up in the Montrose area in a one half mile square area. The story was carried by KHOU TV.
I hope those developers don't end up overbuilding. Maybe $60+ a barrel oil fever is hitting some of them! I can remember as a teenager around 1981 or so listening to some well respected voices in Houston at that time saying that oil was going to go to $100 per barrel by 1990 and that downtown Houston was going to encompass the entire 610 Inner Loop.
I don't know how many townhouse or condo units are being developed, but it is clear that we are seeing the revelation of public preferences here. Those who want to live in the Inner Loop want to live in Montrose and not downtown! There are a number of obvious reasons for this. One is that you probably don't such a problem of dealing with the homeless and the derelict. Another is that downtown is relatively close by, ergo you don't necessarily have to live downtown to have access to it. What I would find amusing (if my tax dollars hadn't been wasted along the way) is that the political classes have spent a horde of taxpayer money trying to lure people to live downtown when the real estate developers are putting their money where their mouth is by betting that people might like what Inner Loop life has to offer without wanting to live there.
Come to think about it, if we had a strong zoning ordainance, it is conceivable that some of those projects might have been blocked politically by people who already live in the area and would not want to see anymore development.
Traffic is not the only issue here. There are also the issues of sewers and drainage infrastructure. Also, it is density but density over a much wider area and much higher scale that would be needed over the long run to justify the building of rail. As it is, we will see at least one vehicle on the road for every two newcomers in Montrose and this still is a relatively small area. Furthermore, the Census and FTA data shows little correlation between density and how many miles people drive.
Midrises are something of an economic compromise when it comes to development. The problem with highrises is that developers might underestimate their costs because the marginal cost of building way up goes up the further you build upwards. The same principle applies to parking lots. You do not see to many multi-story parking garages because they are expensive to develop vis-a-vis simply acquiring some land and paving it over for parking space. Usually you only see parking garages in major real estate developments, primarily commercial in nature.
The problems associated with rising costs from building highrises sometimes results in early buyers getting into lawsuits with the developers over the fact that buyers might have been lured into buying a condo at a price, only to have the developer see cost overruns and then try to either get out of the deal or renegotiate the sale. This is what happened to a former owner of the property I now own.
It will be really interesting to see what the 2010 Census numbers turn out like. Despite the building boom, I still am of the opinion that there is more development happening outside of the City limits then inside of them.
Mayor White and Council do have a problem on their hands. Street layouts will need to be rethought and maybe even widened - yikes! Nonetheless, this is a yet another sign that we are living in good times here in Houston. I would much rather be trying to deal with these problems rather than presiding over a city which is struggling economically.
Addendum - May 20, 2007: I saw an ad in the Sunday May 13, 2007 edition of the Houston Chronicle. There were two downtown lofts being advertised. I cannot remember the details of one of the lofts, but I do remember that the other was being pitched for $165,000. And what, pray tell, do you get for your $165,000? How about 704 square feet of living space. Isn't paying $230 per square foot for the equivalent of a one bedroom apartment a bargain?
Also, there was a story carried by Channel 11 recently whereby people who now live in Montrose are starting to complain not only about the construction, but also about the added crowding that the new density is probably going to bring. Stay tuned.