In 1993, Donald Becker and Thomas Sterling began designing x86-based computer clusters, intended to be low-cost alternatives to the large supercomputers of the day. The 'Beowulf' systems caught on quickly and spread throughout government and academic research communities. The era of commodity clusters was born.

More than a decade later, Microsoft decided that the HPC market was now ready for a commodity cluster management solution. And today, Microsoft Corporation announced the release to manufacturing of Windows Compute Cluster Server 2003 (CCS), its first offering into the HPC market. The CCS product is designed to manage 64-bit x86 computational clusters for high performance workloads, challenging Linux dominance of this market.

The release to manufacturing represents the company's formal product announcement. It means that the Microsoft development team has signed off on the completion of the CCS product. Although evaluation versions will be available very soon from the Microsoft website for downloading, the product won't be generally available for purchasing until August. Other partnering OEMs will be announcing support over the next few months.

Following the spread of HPC from government and academia into the enterprise, the CCS product is designed to give more users access to the power of high performance computing. With this product, Microsoft is going after the volume cluster market, what IDC classifies as workgroup and departmental cluster systems, although CCS is designed to scale to much larger systems. Kyril Faenov, Microsoft's director for HPC, said that this offering will accelerate the proliferation of commodity clusters into mainstream applications.

“This is a fundamental part of our strategy — to bring HPC to the broadest number of users possible,” said Faenov. “What is particularly exciting about this time is that the last few years have seen tremendous growth in HPC clustering. IDC data for 2005 has shown 23 percent growth in the HPC market, overall, and the cluster portion specifically, growing 70 percent by revenue. So there's an explosion of opportunity happening.”

The CCS product has been under development at Microsoft for the last two years. It is built on the foundation of the Windows Server 2003 64-bit platform, which provide mission-critical capabilities for the enterprise. This enables the company to leverage its investment in enterprise server software and lets Microsoft customers avoid requalifying a different server stack.

Faenov observed that, today, many people essentially go out and purchase a rack of servers and with the idea of turning them into functional clusters. In this scenario, the user has to find HPC specialists to essentially piece these clusters together from stand-alone operating system and network components. Usually this entails selection of an open-source cluster-capable OS. Then they have to figure out the parallel programming stacks that will support their applications, make sure that the underlying interconnect support is available and integrate these things with the OS that they're using. An additional complication is figuring out how to maintain security for the cluster resource with the rest of their infrastructure. Integrated cluster solutions, typically Linux-based, using commodity processors and interconnects tend to hide most of this complexity; but in this case the customer is limited to the particular vendor's hardware offerings.

What Microsoft wanted to do was provide an integrated software solution for clusters that was hardware neutral — at least in the 64-bit x86 sense — and could be easily integrated into existing enterprise infrastructure. For existing Windows shops, it gives organizations the ability to leverage skill sets, especially Windows network administrations skills. In this environment, CCS should fit in rather easily.

Its integration with Active Directory, which is installed in 80 percent of large enterprises, allows CCS to provide secure cluster access and job scheduling. A cluster system becomes just another Active Directory object, like a network printer. So from a workstation, you can discover clusters, monitor your jobs, submit new jobs and modify them. In this sense, CCS just extends the Active Directory model to recognize a new kind of resource. Faenov said that Microsoft is also working with ISVs to provide cluster access from within vendor applications, just as you would access a printer from any Windows program. This suggests that enterprises that are highly invested in Windows-based systems today will have a pretty clear path to cluster scale out.

“On the other hand, we still think there are advantages for shops that might have deployed Linux or proprietary Unix-based HPC systems — specifically in providing better integration for their users,” said Faenov. “There are really no purely Linux shops or purely Windows shops, for that matter, especially when it comes to HPC. In many cases, what we see in engineering shops is that the people have Windows workstations. When it comes time to use the computational resource, they have to drop to the command line and use SSH or Telnet into the remote cluster, move their data over there, [do the computation,] then move it back. What that does is disconnect the user from an important part of their workflow — analysis of their models and the results.”

Another aspect of CCS is that it provides a standardized platform for HPC applications. The MS-MPI message passing interface, derived from Argonne National Lab's Open Source MPI2 implementation, is used to support message passing for parallel applications. According to Microsoft documentation, MS-MPI is compatible with the MPICH2 Reference Implementation and other MPI implementations and supports an API of more than 160 function calls. In addition, the MS-MPI stack supports multiple interconnects including Ethernet, InfiniBand and Myrinet. According to Faenov, Microsoft worked closely with the interconnect vendor community, especially InfiniBand providers, to make sure that those products would be well-supported on CCS.

“We adopted the Beowulf-style paradigm of the head node and the compute nodes,” said Faenov. “Again, that's a model that will be very familiar to HPC users that have deployed Linux clusters before. At the same time we reduced the total number of concepts to the key elements — such as MPI, the job scheduler, the remote management capability — for the Windows administrators.”

According to Faenov, Microsoft focused on reaching out to the key application vendors that HPC customers depend on to make sure those solutions will be deployed on CCS. Software vendor partners announced include The Mathworks, Fluent, MSC.Software, ANSYS, The Portland Group (PGI), Livermore Software Technology Corp (LSTC), among others. Those vendors are expected to be bringing out their solutions for the CCS platform throughout the remainder of this year.

“Ultimately, the folks that we're talking to are focused on getting their work done,” said Faenov. “They just want to be able to run a core set of applications to get their results. They really don't particularly care what the underlying system is. They've been forced to learn Linux clusters, in many cases, by necessity, because a complete solution hasn't been available from Microsoft. We're consistently hearing from folks that if you give us the same environment that we're used to in Windows, we'll be glad to adopt that.”