The increasing number of CPU cores makes me worry about the relatively low utilization of the recent manycore CPUs, along with the potential increase in wasted energy, and the low degree of parallelization of many applications submitted to many-core based servers. As I knew about eXludus multicore optimizer MCOPt development, I decided to ask CEO Dale Geldart a couple of questions.

We are seeing rapid increases in processor core counts, and a great deal of discussion is taking place in the industry concerning the best way to make use of large core count servers. What is eXludus’ view on this topic?

Geldart: We have certainly taken note of the discussion, as our main focus at eXludus is developing solutions that allow for more efficient use of multicore processing power. We are looking at the problem a bit differently than most, though. The majority of discussion and activity on this subject relates to parallelizing applications so that an application may natively tap into the parallel processing capabilities of multicore. While we certainly concur that parallelizing applications can provide benefit, we believe that improving system resource allocations can yield equally large, and possibly larger, benefits. So, it’s the resource allocation side of the problem that we’re addressing, which is a much less discussed side of the problem.

While that does present a different view from the mainstream, why do you feel that resource allocation improvements can provide such benefits?

Geldart: There’s at least two parts to the answer. First, we have to look at how the industry as a whole can improve the level of application parallelism, and the timing of such improvements. Then, we have to look at how well a parallel application can utilize available system level resources. Regarding the first point, the industry has some major hurdles given that most applications in use today are serial or only marginally parallel, so there are major re-writes that need to occur across a very wide spectrum of applications. And, there is an extreme shortage of developers that have experience in developing parallel applications. So the parallelism effort is going to be a grassroots effort; new school curriculums focusing on parallel development, new tools, new libraries, and so forth. This will clearly require a long-term industry effort, three to five years at minimum. But, then, in resource consumption terms, we have to look at how well these new parallel application will make use of many-core architectures. The challenge is that most problems are not inherently massively parallel. You have to look hard to find examples of even 32-way parallel applications. And most problems are not more than 75 to 90 percent parallel, meaning that most parallel applications are going to consume somewhere between 4 and 10 cores.

We do tend to forget about Amdahl’s Law of Serialization! If your analysis is correct, given that 32, 64, 128 cores, and more, will soon be appearing in mainstream server architectures, most parallel applications will still not be able to make use of a large percentage of available cores.

Geldart: That’s right. Today’s serial application running in a 4 or 8 core server consumes an equal or greater percentage of core resources to tomorrow’s 8-way parallel application running in a 64 core server; even less when average servers have 128-plus cores! That’s not to say we shouldn’t parallelize applications – we should, where possible, as it makes sense. But this activity will only solve part of the problem.

So, how does eXludus ‘improved resource allocation’ solution fit into the equation?

Geldart: Our software, MCOPt, has been designed to automatically optimize the use of available core and memory resources. The short answer is that MCOPt learns about each applications’ resource needs, whether the application is serial or parallel, and allocates resources in a manner that improves system throughput by allowing resources to be put to their fullest possible use while avoiding shared resource contentions.

Today, core and memory resources are allocated by the operating system. Are you stating then that MCOPt can do a better job than the OS?

Geldart: Yes, to date this function has been the exclusive domain of the OS, and the OS is very efficient at resource allocation to a certain point. But, as the number of cores increases, and users try to run more concurrent tasks as a means to tap into the available processing power, the time-share nature of the operating system gets cumbersome. The net result of running many concurrent tasks is that average job runtimes tend to increase due to round-robin access to some shared resources. And, the OS is too accommodating.

How so?

Geldart: A good example would be in regard to memory allocations. If the running jobs request more memory than physically available, the OS will start paging memory contents out to disk because it wants to accommodate everyone. This has a detrimental impact on all running jobs as disk access is so relatively slow as compared to memory access. This behavior is actually rather unfair. There could be several tasks that have been running in a system for an extended period of time, progressing well to their completion. Then a new task gets dispatched to the system, and this new task may make memory requests that lead to this detrimental paging activity. Not only does this new task realize relatively poor performance, ALL of the previously well running jobs get hit too! Very unfair when you think about it.

And MCOPt has some means of better controlling or preventing this sort of behavior?

Geldart: Right. MCOPt uses underlying queuing theory principles to determine the best possible throughput given a) the resource demands of executing tasks and b) the state of system resources at that instance in time. If an application needs access to a core, but providing that access will slow down other applications, MCOPt will not service the request, and the application will be temporarily held in the system until the resource becomes available. Or, if a memory request is made that will put the system into a heavy paging state the request will not be fulfilled until the request can be safely satisfied. Basically, in this instance MCOPt will prevent system thrashing, so users will never experience the situation where performance falls off a cliff, so to speak.

Then do the applications need to be modified in some manner to be made aware of MCOPt?

Geldart: No, MCOPt functionality is transparent to the application. It takes about a minute to install the software and, once installed, MCOPt starts performing its resource allocation function.

How does MCOPt interface with a Linux scheduler? Does MCOPt replace the scheduler?

Geldart: No, MCOPt was designed to work with the scheduler, so the OS does not need to be patched or modified. MCOPt determines what the best scheduling outcome is, and feeds requests into the scheduler. Just like MCOPt is transparent to applications, MCOPt is transparent to the OS.

While this functionality sounds theoretically interesting as a non-intrusive means to providing better system throughput, what’s been the real-world practical experience?

Geldart: In summary terms, environments using workload managers to dispatch jobs to processing nodes have reported average throughput gains of 25 to 30 percent with MCOPt. But, it’s important to note that the benefit will always be a function of the workload that has been dispatched to an individual system at some instance in time, and the requirements of that mix of jobs.

Can you provide some concrete examples?

Geldart: Sure. A large US DOE lab recently measured throughput gains of 8 to 58 percent across a range of applications, due to MCOPt; a large chip design company measured 25 to 33 percent gains running some of their chip simulation codes; two separate companies running mechanical CAD codes measured gains of 20 percent to as high as 100 percent when running as few as 20 concurrent users; a Tier-1 server vendor measured gains that ranged from 17 percent to over 100 percent when running a mix of small and large memory foot-print jobs. So, as you can see the gains with MCOPt can be quite significant.

Most of the above examples involved 8-core servers, dual-socket/quad-core configurations. As the average core counts increase, the contention for shared resources will increase, so we expect that MCOPt’s benefits will become even more pronounced over time.

The hot topic of the year is cloud computing. Now that Amazon has announced its EC2 Cloud Compute Instances each consisting of a pair of quad-core Intel Nehalem processors, could eXludus MCOPt help to optimize utilization of resources in the Cloud to which the user doesn’t have direct access?

Geldart: In fact, we run our own QA tests on EC2, so MCOPt works well within the virtual machine. Basically, MCOPt manages the resources within the virtual machine just as it would on a physical machine, with similar benefits.

Finally, what can we expect next from eXludus?

Geldart: MCOPt learns a great deal about each application’s resource needs. Today, we use this information only for short term scheduling decisions. In the near future we will release an application profiling functionality that will store this learned information in a database. We can then use this information to further refine and improve scheduling decisions within each system, and we’re also working with some of the workload manager companies to enable them to use this information to refine their macro scheduling decisions across the cluster, grid, or cloud. Users will also be able to tap into this repository to learn more about the applications they are running.