During my university days, and as my thesis project, my friend and professor Jorge Castellanos asked me to develop a tool that automatically detects some characteristics of the nodes in an HPC cluster, such as memory, processor type and processor quantity. The idea is to be able to compile a set of benchmarks that clearly characterize cluster performance and let users run these tests and see the results via a GUI on their desktop PC.

To effectively measure the performance of a cluster, there are many factors to considerer, as well as many tools for measuring each of these factors, so it is necessary to know all the components involved in the performance of the system under study.

Not all people using HPC clusters are familiar with the hardware, operating system and low-level system libraries of which the cluster depends. Thus there is a need to develop a tool that allows the measurement process to be practical for cluster administrators or even to non-specialized users (i.e., physicists, chemists, engineers, etc.). The tool would cover the main elements influencing cluster performance, group them under a single graphical interface, and provide the user with a default configuration for the benchmarking process. It also must be capable of showing results and making suggestions that identify factors that could affect system performance.

In order to accomplish that, I selected an initial set of benchmarks capable of providing good coverage of the major subsystems (CPU, memory, network) associated with the performance of an HPC cluster. These benchmarks are:

• The HPC Challenge: This benchmark is composed of seven individual tests that can be run as a single suite or each one individually (which is a customization I made).

o PTRANS (network): Allows you to measure the total network transfer capacity of the cluster using large blocks of data, performing a matrix transpose.

o High Performance Linpack (HPL): The classical Linpack measures floating point operations per second (FLOPS) across the whole cluster.

o DGEMM (CPU): Similar to Linpack, but just matrix-matrix multiply on separate nodes without communications between them.

o STREAM (memory): Measures bandwidth between CPU and memory using vector-scalar multiply-add operations.

o RandomAccess (memory): Measures the update rate of integers in random memory locations.

o FFTE (CPU): Measures CPU execution rate in FLOPS for Discrete Fast Fourier Transforms.

o b_eff (network): Measures latency and bandwidth of the cluster’s network using small data sets and MPI routines.

• Netperf (network): Used for peer-to-peer network testing.
• IOzone (disk): Measures various I/O operations.

Having defined this, lets talk about the tool, which I call ClusterNumbers.

ClusterNumbers was coded in Python and is broken into three main modules: the Server Module, the Communications Module and the Graphical Module.

The Server Module (SM) runs as a daemon on the administration node of the cluster and is in charge of analyzing the entire cluster by walking each node and gathering information about processor type and quantity, OS architecture and amount of memory of each node. With this information, the tool adjusts parameters in the MPI daemon (MPICH2) config files and in some of the libraries and benchmarks for compiling and running them in an optimal way.

The interaction with the SM can be made via a command line client (local on the cluster’s admin node) or using the event-driven GUI client on a remote desktop PC. In both cases a Unix socket (pipe) is used to pass messages to the running daemon.

The Communications Module (CM) consists of two parts, the first one is a standard SSH server running on the admin node of the cluster. The second one is a set of libraries and routines on the client side called by the Graphical Module. These routines are coded using the Twisted-conch networking engine for Python. The CM works by opening a channel between the client and the admin node, using the SSH protocol, and passes commands and data through it. It also handles authentication.

The Graphical Module (GM) runs from a desktop PC and handles the interaction with the user. The first time you connect to a cluster using the GM you’ll be presented with this log-in window.

Main window

The first step is authentication. A user must provide the IP name or address of the admin node, and enter their user name and password for that system.

Upon successful log-in, the Configuration menu gets activated. Here you set up some system parameters, like the network communication protocol between nodes (rsh and ssh so far), the path on the admin node to the directory where the daemon (SM) resides, whether the cluster has a dedicated admin node or not, and the nodes of the cluster. As soon as you finish this setup, the GM sends the configuration to the SM, incorporates the information about the cluster, and then activates the next interface.

System configuration window

At this point, the Compile Benchmarks sub-menu appears. As soon as you click on it, the GM sends a command to the SM and starts compiling each benchmark. This process could take awhile.

If you select the Get Configuration sub-menu, you can force a status update of the compiling process and the “Execution Parameters” sub-menu gets activated.

If you select the the Execution Parameters sub-menu, it will show you a recommended initial configuration for some of the benchmarks. You can adjust them as you see fit.

HPCC/HPL runtime parameters

When the first benchmark gets ready to run, the Benchmarks menu appears displaying the sub-menu corresponding to the compiled benchmark. I recommend you not run any benchmark until the whole compilation process completes, since it will interfere with the results.

When you’re ready to run a benchmark, you can select it from four categories: Global Benchmarks that makes use, directly or indirectly, of more than one component of the cluster, CPU Benchmarks, Memory Benchmarks and Communication Benchmarks.

As soon as any of these benchmarks finish, the Results sub-menu appears. If you click it, a window will appear showing a list results, a few analysis and suggestions. You can save them on a text file by clicking the Save button.

Keep in mind that this is just an overview of the ClusterNumbers. A complete functional description of the tool is out of the scope of this article.

At this time, ClusterNumbers is completely functional, but only works on Linux x86 and x86_64 clusters using the shared NFS filesystem and an Ethernet interconnect. This was the only type of cluster I had access to during development, which is the main reason I’m opening my tool for other developers.

I believe that allowing other users to contribute to this project is the best way to expand support for other cluster architectures. In addition, with community feedback, me or any contributor can add new benchmarks and functionality. So far, I received great response and interest from HPC users around the world.

The next step for ClusterNumbers? As an open source project is to create a roadmap using a few ideas I have, as well as feedback from the HPC community. With that we can give the project direction and set specific goals and timelines. Also, I’m in the process of enhancing the current documentation to make it easier for new contributors to understand the code.

The project will be hosted at Source Forge and will be open for contribution by the end of this month waiting for your help and ideas.

Hope to see you on the contributors list!

—–

About the Author

Raul Gomez, aka NachoGomez, is a “Licenciado” in Computer Science from Valencia, Venezuela. He has a strong background in the areas of high performance computing, computer benchmarking, programming and virtualization. Currently, he is working for RadiumTec, his own consulting company. He can be reached at [email protected].