The Universe might be expanding, but at least it's getting easier to see. On Monday, at the annual Microsoft Research Faculty Summit, the software maker unveiled the largest and clearest image of the night sky ever assembled. This so-called "TeraPixel" sky map was generated with the help of some of Microsoft's latest HPC and parallel software assets.

The TeraPixel project from the folks at Microsoft Research was essentially a recomputation of the image data collected by Digitized Sky Survey over the last 50 years. The input data was made up of 1,791 pairs of red-light and blue-light plates produced by two ground high-powered telescopes: the Palomar telescope in California (US) and the UK Schmidt telescope in New South Wales (Australia). Between them, the two installations covered the night sky of the Northern and Southern hemispheres.

As one might suspect of photographs collected over a long period of time with different equipment and under different conditions, the quality of the images varied considerably. Different color saturation, contrast, noise, and brightness, as well as the presence of vignetting (darkening toward the image corners) meant that the data would require a lot of post-processing to produce what the researchers were going for: a seamless photograph of the entire sky.

Compared to the old sky image, the TeraPixel version is much more refined. With all the artifacts, seams and inconsistencies processed away, it looks like a true unified image of the sky above. It's like going from Super Mario Brothers on 1985-era Nintendo consoles to Halo 2 on Xbox 360s.

According to Dan Fay, the director of Earth, Energy and Environment at Microsoft Research, to get this level of refinement, all the images had to go through a four-stage processing to correct for the irregularities. The first stage attacked the vignetting artifact to brighten up the dark corners. The next step was more complex. Since each plate had a red and blue version for each area, these had to be processed separately and then realigned into one image. They even had to account for multiple overlapping plates. In some cases, Fay says, they chose the best pixels on the various plates to come up with the highest quality image. The third step involved stitching the individual images together and smoothing out the seams. Lastly, the multi-resolution images were generated so that users could zoom in for greater detail. The final result was a spherical panorama of the night sky in 24-bit RGB format.

Much of the software relied on Microsoft software as well as Microsoft programmers. The project used the global image optimization program developed by Hugues Hoppe and Dinoj Surendran of Microsoft Research and Michael Kazhdan of Johns Hopkins. The DryadLINQ and the .NET parallel extensions framework was employed to construct and manage the applications. DryadLINQ is a programming environment for running parallel applications across a cluster, using LINQ (Language Integrated Query) as a query engine on top of the Dryad runtime. The latter takes the queries and distribute them across the nodes. Windows HPC Server was used to schedule the more tightly-coupled jobs and the Project Trident Workbench was employed to manage the entire workflow.

By HPC standards, the hardware platform was relatively modest. A 16-node Intel Xeon cluster was used to process the TeraPixel image, but the final runs were done on a 64-node system. The image was built iteratively since the algorithms were continuously tweaked to get better refinement. A full run on the 16-node machine took three days, while on the larger machine, it took just over half a day.

One of the costliest operations, time-wise, was shuffling the images around the cluster. "Some of the biggest issues were data movement," notes Fay. "When you start getting to that many nodes and parallel jobs, moving the data ends up taking a lot of the time." Just transferring the final 1,025 files (802 GB total) off the cluster took 2.5 hours using a 1 Gbps link.

The TeraPixel image can be viewed by researchers and the general public on Microsoft Research's WorldWide Telescope Web site. It also can be accessed via Bing Maps, via a plug-in, where you can see a street-wise view of the sky above. Because of the high resolution of the imagery, viewers are able to zoom into any area of the sky and see greater detail of specific star systems.

The sky image they've produced has been verified by astronomers, who made sure that nothing is rotated incorrectly or is otherwise erroneous. According to Fay, the feedback from the community has been gratifying. "No one has ever seen an image of the sky like this," he says.