Microsoft is readying its Dryad/DryadLINQ distributed computing framework for a commercial debut. The software is currently available for evaluation purposes, but according to a ZDNet article published last week, Microsoft is transferring the technology from its research division to its technical computing group. A “Community Technology Preview” of the software stack is scheduled for release in November 2010, with a product launch to follow sometime in 2011.
Dryad is a MapReduce-like distributed computing runtime that Microsoft Research has developed for coarse-grained parallel applications. It is designed as a general-purpose engine for parallelizing applications across a cluster or even an entire datacenter. The runtime handles job creation and management, resource management, job monitoring and visualization, fault tolerance, re-execution, scheduling, and accounting. DryadLINQ is layered on top of Dryad to generate computations via LINQ (Language-Integrated Query).
The stack also includes front-ends for machine learning, data mining, eScience, as well as Azure-flavored cluster services and data management layers on the backend. The product being released next year is designed to run atop Windows HPC Server.
The Azure-HPC linkage reflect’s the company’s aim to generalize deployment of data-parallel applications from small clusters all the way to ultra-scale cloud infrastructure. It also reflects Microsoft’s propensity to invent its own software frameworks, with the aim of attracting application software to its existing HPC platform and its nascent cloud business.