Microsoft Releases New Software Tools for Researchers

By Michael Feldman

July 13, 2009

As scientists increasingly rely on big data to drive their research, a new set of software tools is emerging. Two of these new tools, developed by Microsoft’s External Research division, were launched on Monday at the Microsoft Research Faculty Summit in Redmond, Wash. They include the Project Trident workbench and the Dryad/DryadLINQ programming environment.

Project Trident was originally aimed at oceanographic applications (hence the name). The work began as a collaboration between Microsoft External Research, the University of Washington and the Monterey Bay Aquarium to provide a high-level workflow tool for oceanographers. Oceanography, like many scientific domains today, is being inundated with a deluge of data that researchers are struggling to manage.

Once the proof-of-concept stage for Project Trident was completed, Microsoft realized it could be used as a general-purpose platform for other areas, such as astronomy, environmental science, medicine or essentially any type of research that is dominated by workflow issues. The data is coming from a growing number of inexpensive sensors that collect information in real time as well as an ever-expanding collection of scientific databases being stored on the Internet or in private repositories. In many cases, both data rates and data volumes are growing beyond the capabilities of traditional software environments.

Unlike the commercial world, the science community tends to freely pass its data around. But turning the raw information into useful knowledge often requires weeks, months or even years of software development involving customized scripts and applications. The whole idea behind Trident is to enable workflow applications to be developed by scientists, rather than programmers, by structuring the process into modular steps.

“Why lock your knowledge up into scripts or programs when you could actually write it in a tool that other people stand a chance of reusing,” asks Roger Barga, who is leading Microsoft’s development of Project Trident. According to him, researchers are recognizing that the model of customized workflow development is not sustainable. Even if software maintenance were less expensive, scientists are looking for the kind of speed and flexibility that a code rewrite does not allow.

The Trident workbench is being used today by oceanographers at the University of Washington for seafloor-based research that uses thousands of ocean sensors and by researchers at the Monterey Bay Aquarium Research Institute to study Typhoon intensification.  The workbench is also being employed by astronomers at Johns Hopkins University to support the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) project, which is looking for objects in the solar system that could pose a threat to Earth. In this case the data being ingested comes from an array of 1.4 gigapixels digital cameras that capture images of the night sky.

Ecogenomic sensors

In a nutshell, the Trident workbench tool provides a visual framework for managing and developing workflows. At startup, the user sees a library of existing workflows and activities (or workflow steps). In the GUI, one can add or delete steps from the pipeline by simply dragging and dropping. The idea is that domain experts with no programming knowledge can go in and mix and match existing workflow components to author new experiments and run them on the fly.

A typical workflow would start with reading in the raw data — data files and/or sensor devices. The next step would be to convert the various data sources into a common format. An analysis pipeline — filtering and conditioning algorithms — would come next. Typically the last step is to produce a visual representation of the result.

It’s not all just shuffling objects around a GUI, however. The individual activities, such as reading the raw data, analyzing it, and creating visualizations have to be developed in the first place, as you would any other piece of software. But once developed, the activities can be bound to any user-generated workflows. According to Barga, their experience has been that once you get more than a dozen or so workflows constructed, the users find they’re no longer writing much new code.

One of the important strengths of Trident is that it can utilize HPC clusters. Scientific analysis at scale often requires a high performance computing platform for reasonable performance. By default Trident assumes a single node execution, but users can schedule a job across multiple cluster nodes by creating a workflow application that communicates with the HPC job scheduler.

As one might have guessed, the assumed clustering environment here is Microsoft’s Windows HPC Server, but Trident does allow you to plug in your own scheduler too. This enables researchers to run on a Linux cluster, which remains a much more common platform today for high performance computing. Barga says plugging into a non-Windows scheduler is just one of the different ways Trident has been designed for extensibility, noting that even the tool’s GUI can be replaced should users wish to have a customized look and feel. One dependency that cannot be jettisoned, however, is the Windows .NET framework. The .NET environment contains the Windows Workflow, which is the foundation of the Trident workbench.

The other tools Microsoft released on Monday — Dryad and DryadLINQ — are aimed at developers rather than end users. Dryad itself is a general-purpose data parallel programming runtime designed to run distributed applications on Windows clusters. The runtime is responsible for scheduling resources, handling hardware and software failures, and distributing data and code across the cluster as needed. DryadLINQ is an abstraction layer that runs LINQ (Language Integrated Query) operations on top of Dryad, the idea being to be able to execute data queries that automatically get parallelized via the Dryad runtime.

Unlike MPI, Dryad is not for latency sensitive computation. It is aimed at applications that can increase data throughput via loosely-coupled parallelization. Microsoft Research itself uses Dryad internally for search engine and machine learning research. Barga says they have scaled such applications up to 3,000 nodes on a Windows HPC Server cluster, noting that some of these jobs run for dozens of hours. “The beauty of the Dryad runtime is that if an individual node drops out or there’s a failure in one of the jobs, Dryad automatically recovers, moving the computation off the failed node and reproducing inputs that node was responsible for,” says Barga.

Microsoft is really offering Trident and Dryad/DryadLINQ as two separate solutions, but with interoperability. Trident includes a pre-defined custom activity that invokes Dryad/DryadLINQ, allowing the programmer to pass it LINQ queries. But the real intention seems to be to encourage users to develop their own Dryad/DryadLINQ components to hook into the Trident workbench or use them in standalone applications.

Trident and Dryad/DryadLINQ will be released under the MSR-LA license (Microsoft Research License Agreement) and, as such, is for non-commercial academic use only. Barga says Microsoft is considering some sort of license arrangement for commercial users, but without any requirement for royalty paybacks. The bottom line here is that Microsoft is not looking to generate revenue directly from these tools, but rather to expand the Windows ecosystem for researchers and encourage use of the Windows HPC Server platform.

Barga couldn’t talk about any future interoperability between these tools and Microsoft’s Azure cloud computing platform, but it’s reasonable to assume that all these technologies are heading toward convergence. “Science is moving to the cloud and we want to make sure that all of the tools that we offer, including things like Dryad and Trident … will work on the cloud for scientists who want to do really big data challenges,” says Barga.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Data Vortex Users Contemplate the Future of Supercomputing

October 19, 2017

Last month (Sept. 11-12), HPC networking company Data Vortex held its inaugural users group at Pacific Northwest National Laboratory (PNNL) bringing together about 30 participants from industry, government and academia t Read more…

By Tiffany Trader

AI Self-Training Goes Forward at Google DeepMind

October 19, 2017

DeepMind, Google’s AI research organization, announced today in a blog that AlphaGo Zero, the latest evolution of AlphaGo (the first computer program to defeat a Go world champion) trained itself within three days to play Go at a superhuman level (i.e., better than any human) – and to beat the old version of AlphaGo – without leveraging human expertise, data or training. Read more…

By Doug Black

Researchers Scale COSMO Climate Code to 4888 GPUs on Piz Daint

October 17, 2017

Effective global climate simulation, sorely needed to anticipate and cope with global warming, has long been computationally challenging. Two of the major obstacles are the needed resolution and prolonged time to compute Read more…

By John Russell

HPE Extreme Performance Solutions

Transforming Genomic Analytics with HPC-Accelerated Insights

Advancements in the field of genomics are revolutionizing our understanding of human biology, rapidly accelerating the discovery and treatment of genetic diseases, and dramatically improving human health. Read more…

Student Cluster Competition Coverage New Home

October 16, 2017

Hello computer sports fans! This is the first of many (many!) articles covering the world-wide phenomenon of Student Cluster Competitions. Finally, the Student Cluster Competition coverage has come to its natural home: H Read more…

By Dan Olds

Data Vortex Users Contemplate the Future of Supercomputing

October 19, 2017

Last month (Sept. 11-12), HPC networking company Data Vortex held its inaugural users group at Pacific Northwest National Laboratory (PNNL) bringing together ab Read more…

By Tiffany Trader

AI Self-Training Goes Forward at Google DeepMind

October 19, 2017

DeepMind, Google’s AI research organization, announced today in a blog that AlphaGo Zero, the latest evolution of AlphaGo (the first computer program to defeat a Go world champion) trained itself within three days to play Go at a superhuman level (i.e., better than any human) – and to beat the old version of AlphaGo – without leveraging human expertise, data or training. Read more…

By Doug Black

Student Cluster Competition Coverage New Home

October 16, 2017

Hello computer sports fans! This is the first of many (many!) articles covering the world-wide phenomenon of Student Cluster Competitions. Finally, the Student Read more…

By Dan Olds

Intel Delivers 17-Qubit Quantum Chip to European Research Partner

October 10, 2017

On Tuesday, Intel delivered a 17-qubit superconducting test chip to research partner QuTech, the quantum research institute of Delft University of Technology (TU Delft) in the Netherlands. The announcement marks a major milestone in the 10-year, $50-million collaborative relationship with TU Delft and TNO, the Dutch Organization for Applied Research, to accelerate advancements in quantum computing. Read more…

By Tiffany Trader

Fujitsu Tapped to Build 37-Petaflops ABCI System for AIST

October 10, 2017

Fujitsu announced today it will build the long-planned AI Bridging Cloud Infrastructure (ABCI) which is set to become the fastest supercomputer system in Japan Read more…

By John Russell

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Intel Debuts Programmable Acceleration Card

October 5, 2017

With a view toward supporting complex, data-intensive applications, such as AI inference, video streaming analytics, database acceleration and genomics, Intel i Read more…

By Doug Black

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Leading Solution Providers

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

  • arrow
  • Click Here for More Headlines
  • arrow
Share This