Pharmaceutical pills and Martian soils. What in the world — worlds? — do these things have in common? For one thing, they consist of granular material; and for another, they are subjects of great interest to Youssef Hashash, associate professor in the University of Illinois' Civil and Environmental Engineering Department.

Hashash has been developing a simulation code to represent the flow of granular material (such as soil, gravel, rocks, grains, raw materials, and crushed materials) for the past four years. He is trying to understand the flow of granular material and what kind of forces it imparts on equipment (e.g., silos, load buckets, Martian surface exploratory devices) that manipulates this material, which, for example, may have an impact on the design of the equipment itself.

The codes involved in this research are computationally demanding. Hashash found that running the code for one second, in real time, using 40,000 particles, took up to 12 hours on a fairly recent PC. At that rate, in order to conduct meaningful simulations, Hashash says that he needs not hours, but days. Besides being slow, 40,000 particles do not properly represent the particles but instead represent a cluster of particles. “In a real situation we are talking about hundreds of thousands of particles, if not millions, that are needed in a simulation,” he says.

Hashash spent the first few years developing the underpinning of the code. Last year he started running larger simulations on NCSA's SGI Altix (Cobalt) and quickly started hitting a wall: some of these simulations became difficult to handle. He encountered long simulation times and large datasets.

Looking for ways to speed up development, Hashash received some informal help from the Performance Engineering and Computational Methods (PECM) group. As a result, he was able to achieve a minor, but insufficient, speed up of approximately four times.

“Clearly, we realized that we needed to do something more formal,” says Hashash. “We needed a relationship with somebody with expertise at NCSA who could really help us.” He submitted a proposal, “Large-Scale Numerical Simulations via Parallel Computing: An Application Using Discrete Element Modeling of Granular Material,” to the Strategic Applications Program (SAP). It was accepted, and he began working with Greg Bauer, a research programmer in PECM.

Hashash says that working with PECM's programmer has been an eye-opening experience because Bauer identified inefficiencies within the code. Bauer has been conducting rigorous testing of the code in terms of its performance and how well it runs large particles.

In addition to the help from the SAP, Hashash learned that NCSA had recently opened its Faculty Fellow program to previous fellowship winners (He had received a fellowship in 1999-2000.) Hashash applied for and received an NCSA fellowship for 2005-2006. Having been awarded this fellowship, Hashash now has additional NCSA help at his fingertips.

The fellowship aspect of the project uses reconfigurable hardware and computing systems to explore the placement of specific parts of the code onto field programmable gate arrays (FPGA). David Raila, senior research programmer in Computer Science, is aiding in this endeavor. The idea is to embed part of the code on hardware, which has the potential for a speed up of ten to a hundred times.

Collaborating with Bauer on parallelism and Raila on reconfigurable computing, Hashash has high hopes.

“The objective through this close collaboration with NCSA, is to combine their expertise together with our expertise, is to be able to run these extremely large simulations. Because if we're able to speed up this code a lot — ten times, a hundred times, a thousand times — then you open up a class of simulations that you would not have been able to do otherwise.”

On this project, funded by NSF and Caterpillar, Hashash has teamed up with his colleague Jamshid Ghaboussi, professor emeritus in the Civil and Environmental Engineering Department, and two graduate students.

This article originally appeared in the December 2005 issue of datalink newsletter and has been provided courtesy of NCSA.