It’s a hot Monday in Nanchang, and not just due to the heated competition of the ASC18 Student Cluster Competition. It’s just damned hot. This is a tropical climate where the temperature might not be all that high, but there’s lots of humidity. Judging by today, I think the largest cash crop in Nanyang must be humidity.
Take the natural heat of early summer Nanchang and couple it with a smallish room containing 103 dual processor compute nodes containing 2,884 CPU cores, add 160 NVIDIA V100 GPUs, and around 175 people milling around. We’re not talking about a data center room either. We’re talking a standard university function center with air conditioning that definitely isn’t designed for this kind of heat load. It was hot, damned hot.
How hot was it? It was so hot that I drank water for the first time in years. It was sweat through your shirt in five minutes hot. I tried to measure the heat with a widget on my phone, but it just displayed a message saying “please call the fire department.”
The heat really affected the competitors in the early stages of the competition. They are arranged in three aisles that take the shape of a squished figure eight. In the upper and lower aisles, there are five teams and the heat is bearable, since the portable air conditioners are nearby. But where the two circles meet, there are ten teams arranged on a single aisle – and the heat is intense.
In fact, it was so hot that there was concern that during the LINPACK and HPCG runs, when all of the systems are at full crank, we might see some shutdowns due to the overheating.
So how did the competition committee deal with the heat? They used a combination of modern and old school technology. On the modern side, there were three or four large portable air conditioners that were spitting out cold air at a furious rate. They had equally large heat exchangers pushing heat out into some of the hallways – including one hallway that you had to traverse in order to use the restrooms. It was like walking through a gigantic hot air hand dryer.
However, the air conditioners were overmatched by the outdoor conditions and the raw compute power in the room. Unfortunately, there weren’t any more large scale air conditioners available in the city.
This is where old school tech comes into play. The ASC18 organizers thought fast, made some phone calls, and had a huge number of ice blocks delivered to the competition floor. They were placed into buckets and delivered to the hottest places in the room.
I didn’t think this would have much effect. But I was wrong. Over time, as the ice melted; you could feel a real difference in the conditions. There was also another beneficial side effect: instant ice chest for drinks. I’m proud to say that I was the first to pioneer this usage model, and others soon joined me in storing their drinks in the ice buckets. Oh, and LINPACK and HPCG went off without a hitch. ASC18 organizers used their ingenuity to beat the heat. Great job!