HPCwire: Stork Data Scheduler Now Available

Since 1986 - Covering the Fastest Computers
in the World and the People Who Run Them

Translation Disclaimer

Subscribe | Sign In

Visit additional Tabor Communication Publications

Leading HPC
Solution Providers

December 03, 2008

Stork Data Scheduler Now Available

Dec. 3 -- Louisiana State University Stork team announced today that first full release of Stork Data Scheduler (Stork 1.0) is now available on the Stork Project Web page.

Stork is a batch scheduler specialized in data placement and data movement, which is based on the concept of making data placement a first class entity in a distributed computing environment. Stork understands the semantics and characteristics of data placement tasks and implements techniques specific to queuing, scheduling, and optimization of these type of tasks.

One key benefit of distributed resources is that it allows institutions and organizations to gain access to resources needed for large-scale applications that they would not otherwise have. But in order to facilitate the sharing of compute, storage, and network resources between collaborating parties, middleware is needed for planning, scheduling, and management of the tasks as well as the resources. The majority of existing research has been on the management of compute tasks and resources, as they are widely considered to be the most expensive. As scientific applications become more data intensive, however, the management of storage resources and data movement between the storage and compute resources is becoming the main bottleneck. Many jobs executing in distributed environments are failed or are inhibited by overloaded storage servers. These failures prevent scientists from making progress in their research.

Accessing and transferring widely distributed data can be extremely inefficient and can introduce unreliability. For instance, an application may suffer from insufficient storage space when staging-in the input data, generating the output, and staging-out the generated data to a remote storage. This can lead to trashing of the storage server and subsequent timeout due to too many concurrent read data transfers, ultimately causing server crashes due to an overload of write data transfers. Other third party data transfers may stall indefinitely due to loss of acknowledgment. And even if transfer is performed efficiently, faulty hardware involved in staging and hosting can cause data corruption. Furthermore, remote access will suffer from unforeseeable contingencies such as performance degradation due to unplanned data transfers, and intermittent network outages.

Traditional distributed computing systems closely couple data handling and computation. They consider data resources as second class entities, and access to data as a side effect of computation. Data placement (i.e. access, retrieval, and/or movement of data) is either embedded in the computation and causes the computation to delay, or performed as simple scripts which do not have the privileges of a job. The insufficiency of the traditional systems and existing CPU-oriented schedulers in dealing with the complex data handling problem has yielded a new scheduler specializing in data placement: the Stork Data Scheduler.

Using Stork, the users can transfer very large data sets via single command. The checkpointing, error recovery and retry mechanisms ensure the completion of the tasks even in case of unexpected failures. Multi-protocol support makes Stork one of the most powerful data transfer tools available. This feature does not only allow Stork to access and manage different data storage systems, but can also be used a s a fall-back mechanism when one of the protocols fails in transferring the desired data. Optimizations such as request ordering, task aggregation, and connection caching provide enhanced performance compared to other data transfer tools.

The Stork Data Scheduler can also interact with higher level planners and workflow managers for the coordination of compute and data tasks. This allows the users to schedule both CPU resources and storage resources asynchronously as two parallel universes, overlapping computation and I/O. Currently, some of the widely used workflow tools such as Condor DAGMan and Pegasus already come with Stork support.

The Stork Data Scheduler is currently being used in several NSF, DOE, and ONR funded projects such as PetaShare, UCoMS, and SCOOP.

For more information on the Stork Project, contact the project lead Dr. Tevfik Kosar at kosar@cct.lsu.edu.

Stork Project Web page: http://www.storkproject.org.

-----

Source: Louisiana State University

Share Options

Subscribe

Subscribe to HPCwire

Discussion

There are 0 discussion items posted.

Join the Discussion

Become a Registered User Today!

Registered Users Log in join the Discussion

RSS Feeds

Feeds by Topic

Feeds by Industry

Feeds by Content Type

Subscribe to All Content

All

May 23, 2012

May 22, 2012

May 21, 2012

May 18, 2012

May 17, 2012

May 16, 2012

May 15, 2012

May 14, 2012

Feature Articles

NVIDIA Works On CPU Co-Dependency Issues with Kepler GPU

NVIDIA is telling everyone that the GK110, its new Kepler GPU aimed at supercomputing, is all about improving performance per watt. But the other driving theme behind the new architecture is reducing the GPU's reliance on its CPU host. How well it accomplishes both these goals areas could determine the success of the new chip in high performance computing.
Read more...

OpenACC Starts to Gather Developer Mindshare

PGI, Cray, and CAPS enterprise are moving quickly to get their new OpenACC-supported compilers into the hands of GPGPU developers. At NVIDIA's GPU Technology Conference this week, there was plenty of discussion around the new HPC accelerator framework, and all three OpenACC compiler makers, as well as NVIDIA, were talking up the technology.
Read more...

NVIDIA Launches Kepler Into HPC

NVIDIA has introduced its first Kepler-generation GPU product for high performance computing, and revealed some of the inner working of the new architecture. The announcement took place at the kickoff of the company's GPU Technology Conference taking place this week in San Jose, California.
Read more...