There’s a buzz in the HPC community about Bright Computing, a young company whose integrated cluster management software is attracting deals with hardware majors, investors, and customers. HPCwire asked Bright Computing CEO Matthijs van Leeuwen to explain his strategy and give us an update on what’s going on at the company.

HPCwire: Why did you found Bright Computing? What did you think was missing from existing cluster management software?

Van Leeuwen: Before we founded Bright, like everyone else we were using a toolkit approach where you take a Linux distribution, add some provisioning and monitoring tools, and script everything together.

However, we realized that this approach can never lead to a solution that is truly easy-to-use, scalable, flexible and maintainable. That is when we started developing from scratch what we called our next-generation cluster management software.

When that software was ready for prime time, we got such great feedback that we realized that we were onto something bigger. So we decided to create a dedicated company to further develop it and take it to market.
 
HPCwire: How are you different today from your competitors?

Van Leeuwen: Pretty much all our competitors, whether open source or commercially available products, use the toolkit approach. However, the problem with that approach is that the tools were never designed for use on HPC clusters and they were also not designed to work together. So you end up spending a lot of time trying to make them work together and make them suitable for HPC.

Another problem is that each tool has its own user interface, which means multiple learning curves. Also, each tool tends to have its own daemon and database, which makes things inefficient.

For Bright, we started completely from scratch. We took a much more fundamental approach, with a single lightweight daemon for all cluster management functionality, a single relational database for all configuration and monitoring data, and a single command line and graphical user interface.

Bright Cluster Manager also offers a number of unique features, such as native head node failover, multi-cluster management, workload manager integration and health checking. Speaking of health checking, we are the only solution that effectively addresses “the black-hole node syndrome.”

HPCwire: What’s “the black hole node syndrome?”

Van Leeuwen: It can be an annoying problem that can kill productivity in a cluster. Most cluster management software does not detect when nodes are “sick,” rather than “dead.” These sick nodes will crash jobs, either consistently or, worse, on a seemingly random basis. This syndrome can only be solved if the workload manager and the cluster manager work closely together, because workload managers do not have enough information about the nodes to solve this problem on their own. Workload managers are smart enough to know if a node is up or down, but they are rarely smart enough to detect subtle health issues, such as a mount point missing, a full disk, or an unloaded InfiniBand driver.

Bright Cluster Manager’s health checking framework allows any type of hardware or software health check to be configured and will warn the workload manager so that the job can be dynamically rescheduled to a set of healthy nodes. The user won’t even notice.

HPCwire: One thing we hear from HPC buyers is that your software is easy to learn, even for beginners, yet this ease-of-use doesn’t impose any serious performance penalty? Can you comment on that?

Van Leeuwen: Right from the start, we worked to take the complexity out of cluster management. Our goal was to provide a complete management solution that was immediately intuitive to use, even for people without Linux experience. View, drill down, point, click, and move on, even for multiple clusters at once. We built our solution around an intuitive, straightforward GUI and provided a single, straightforward management shell for automating tasks or intervention.

As for no impact on performance, that single, lightweight daemon I mentioned previously is the key. We developed it specifically for use on large-scale HPC clusters. By contrast, most cluster management toolkits require multiple daemons for different pieces of functionality and were designed for general datacenter use, with little thought about resource consumption and performance. By the way, our development team spent a lot of time on our daemon, especially in the beginning.

HPCwire: Does the HPC market pay enough attention to cluster management software as a category?

Van Leeuwen: Much of the HPC market has come to accept that cluster management is a painful necessity, and they suffer in silence. Or more to the point, the task is passed on to systems administrators who take on the job of writing complex scripts to make it work, and who must be constantly vigilant to keep the systems running. In the meantime, they have many other important priorities that often take a back seat to keeping the clusters performing. Sys admins are highly skilled and expensive. Writing scripts is a costly waste of their talents and time. And when they move on to another role, someone else has to step in and figure out how the scripts work.

This complexity creates a needless barrier to expanding the use of HPC in research and industry. The HPC team at IDC recently cited the shortage of skilled systems administrators as a growing, critical issue in HPC. I believe that by taking the complexity out of cluster management, Bright alleviates much of this constraint. This also removes a significant hurdle for HPC usage at the Workgroup end of the market, the segment tagged by IDC with the fastest growth. HPC users should be able to focus on their real work, not on how to keep their cluster running.

HPCwire: Do HPC sites budget enough for cluster management software?

Cluster owners and administrators often grossly underestimate the positive impact well-integrated cluster management software has on the TCO of an HPC cluster. The cost benefits come from three aspects: performance improvements, increased throughput and reduced man-power needs.

Let’s start with performance. Cluster management software should have a very small CPU and memory footprint on compute nodes. Every wasted CPU cycle and megabyte can be directly translated to TCO dollars. An efficient cluster management solution maximizes the available compute power for the cluster, which in turn delivers the greatest return on investment.

As for increased throughput, most toolkits are fine for identifying dead nodes and enabling the sys admin to sideline them, but usually after a job crashes. The same for the black-hole node syndrome I mentioned previously, that can crash all the jobs in the queue – a huge productivity loss. It’s difficult to consider the value of jobs completing the first time in a budgeting decision, but it can produce savings in terms of fewer nodes required for computing over the long term.

And lastly, there are many hidden man-power costs that well-integrated cluster management software alleviates. From installation to customization to software upgrades to ongoing management, this significantly reduces the need for human intervention, freeing people to focus on other tasks. Furthermore, the ability to manage multiple clusters at once enables HPC labs to better manage sys admins’ workloads.

HPCwire: What process should HPC sites use to select management software?

Van Leeuwen: An ideal process is to combine TCO analysis with a detailed understanding of the overall HPC context. This approach is exactly what one of our customers, a global manufacturing giant, recently used. Although they purchase their cluster hardware from various suppliers, they choose not to take whatever management software their suppliers pre-installed. Instead, they realized the benefits of standardizing on one cluster management solution that best meets the diverse HPC requirements throughout the organization.

HPCwire: What questions should HPC sites ask themselves and the software vendors?

Van Leeuwen: There’s a long list, so I’ll break the questions out by category.

For the installation, will the vendor deliver the cluster pre-tested and certified, or will your HPC team take ownership of burn-in testing? How much time will you allocate to the initial deployment?

For compute requirements. will users’ needs be met with CPUs alone, or GPUs as well? Be sure the cluster management software can handle GPUs without requiring sys admins to waste time researching user forums, reading multiple sets of documentation and writing complex scripts.

How large do you need to grow your system? Do you have special requirements, like provisioning over InfiniBand or to RAM disk?

Does your user community have homogeneous or diverse needs? Will the sys admins need to frequently reconfigure the clusters? Will large-scale and flexible SMP be needed on occasion, or for every application? Would a user portal add value? Is the community large enough to require dedicated login systems?

What about the available skillsets? Are your sys admins already working around the clock, or do you have a surplus of highly skilled people to tap into? How much maintenance can you afford? Could you spread part of your workload to lower-skilled people? Do you already have a mixture of people who require different access levels?

Will you have a single cluster or the need to manage multiple clusters? Will your cluster be fully utilized, or can you afford frequent job losses? Do you need head node failover capability?

How important is security? When the cluster needs to be replaced, will you need an easy path to transfer configuration and user information?

There’s a lot to consider, but a well-thought-through process will yield immediate and long-term benefits.

HPCwire: How has management software evolved in the past 5-10 years? What have been the biggest developments?

Van Leeuwen: Actually, the lack of innovation in cluster management software over the past 10-15 years is what inspired us to create Bright Cluster Manager. There has been impressive innovation in the space of processors, interconnects and accelerators, but most clusters and cluster toolkits still use the same old tools that were never designed for HPC clusters. Take for example Nagios, Cfengine, SystemImager and Puppet. None of these tools were originally developed for HPC clusters, but they are painfully ubiquitous on HPC clusters.

HPCwire: What are the largest challenges in the future?

Van Leeuwen: The largest challenges come from the increasing size, heterogeneity and general complexity of HPC clusters, as well as the new cloud computing paradigm. As we move from petascale to exascale, cluster sizes grow from thousands to tens of thousands of nodes. This requires novel ways of provisioning, monitoring and visualizing. Increased complexity and heterogeneity come from the use of accelerators, parallel file systems, multi-core systems and cloud computing. Obviously, we have planned our roadmap to best address these challenges over time.

HPCwire: Public cloud computing has been ramping up more slowly in HPC than in enterprise computing. What role can management software play in accelerating HPC cloud adoption?

Van Leeuwen: We see a growing interest from HPC users to use public cloud resources, such as Amazon EC2, and we see many of our customers and prospects starting to experiment with running HPC jobs in public clouds. The easiest and most efficient way to do this is by bursting into public clouds directly from your existing cluster, or by deploying complete clusters inside public clouds from your workstation. This will soon be possible with Bright Cluster Manager.

HPCwire: Data-intensive computing, using software frameworks such as MapReduce and Hadoop, is becoming more important. What does this mean for Bright Computing?

Van Leeuwen: Hadoop and other MapReduce frameworks often run on clusters. Such clusters have very similar management and monitoring requirements to HPC clusters. We already have some customers using Bright to manage their Hadoop clusters. Bright is also used by a major IT company to manage all their distributed database clusters for a business intelligence application.

HPCwire: What has the company been up to lately?

Van Leeuwen: Two key things: extending our product reach and extending our go-to-market capabilities.

To extend our product reach, we recently closed a substantial round of investment with a venture capital partner, which allows us to pursue an ambitious development roadmap for Bright Cluster Manager, including cloud, multi-cluster and parallel file system features. We have also sold Bright Cluster Manager outside of the traditional HPC space, and are exploring ways to expand on this beachhead.

As for our go-to-market capabilities, we have invested substantial time in expanding and enabling our reseller channel and strategic alliances. We now have over 25 resellers in the USA, South America, Europe and Asia. We are on the Dell price list and have partnerships with Cray, IBM and HP , as well as several key technology partners such as NVIDIA, Intel, NextIO, ScaleMP, Altair and Adaptive.

One OEM relationship that we are particularly proud of is with the world’s best known supercomputer company, who is now using Bright Cluster Manager to manage storage for large clusters, and to provide additional services not normally associated with cluster management. A press announcement about this partnership is imminent.