Create Mixed HPC/Big Data Clusters Today Says Bright Computing

By John Russell

February 3, 2016

The latest version of convergence – blending traditional HPC and big data computing into a ‘single’ environment – dominates much of the conversation in the HPC today, including within HPCwire. Elegant unified solutions, certainly at the very high end, are still in the making. That said, Bright Computing is in the thick of efforts to transform clusters and clouds into ‘unified’ platforms comprised of hybrid resources that support both traditional HPC and big data today.

In mid January Bright Computing released version 7.2 of its cluster and cloud management software suite. Last week Bright was awarded a 1.5 million Euros grant from the European Commission Horizon 2020 program Grant around integrating HPC-big data-OpenStack functionalities into its product suite. Sometime in Q1 2016 the company expects to announce more OEM partners, said company founder and CEO Matthijs van Leeuwen. If timing is everything, then the timing seems right for Bright Computing.

The infrastructure complexity being created by efforts to combine HPC and big data infrastructures represents a huge challenge to the HPC community and a rapidly expanding opportunity for Bright Computing. Well established in HPC and growing quickly in the enterprise, the Bright Computing platform is a resource allocation and management platform. To the extent the company’s products – Bright Cluster Manager for HPC, Bright Cluster Manager for Big Data, and Bright Open Stack – deliver as promised, the platform is an important tool for creating single use (HPC or big data) and mixed use (HPC and big data) clusters and (private) clouds, along with the ability to burst to public clouds as needed.

Bright Computing founder and CEO Matthijs van Leeuwen compared the current situation to the early days of Boewulf clusters.

Matthijs van Leeuwen
Matthijs van Leeuwen

“Researchers began putting clusters together themselves, ignoring traditional supercomputer vendors, and there was no commercial management software available. I think we are going to see the same situation now as we go into a new era where everyone wants to combine HPC and big data technologies. Those who are still doing a lot of this stuff in-house, with their own scripts, sys admin, and dev-ops are going to find it is too cumbersome, time-consuming, and risky. They will want to have that automated.”

Consider just a couple of the difficulties in merging the two environments. A Hadoop cluster typically has a bunch of storage on every compute node. HPC clusters may have diskless nodes with no storage whatsoever. Hadoop uses Hadoop Distributed File System; HPC systems tend to rely on GPFS (now Spectrum Scale), Lustre, or NFS – although van Leeuwen is quick to add, “it is possible with Bright software to easily set up a common files system, Lustre in particular, that will work both with the HPC and big data for Hadoop and Spark.”

The point is that combining HPC and big data in a single environment isn’t a trivial task, and that cobbled together hybrid resources are immensely challenging to manage. There are many hardware, software, and administration issues to deal with (for example, see HPCwire article, Toward a Converged Exascale-Big Data Software Stack). Moreover mixed-use environment performance typically doesn’t match pure-play HPC.

There is also a more general trend for consolidation inside the datacenter, according to van Leeuwen, “It’s a move away from datacenters with dedicated HPC clusters, dedicated Hadoop clusters, and other types of dedicated clusters or server farms. More and more HPC will move to the public cloud, but I don’t think it is going to be a dramatic shift. It’s been gradual over the years and I don’t see anything coming that would cause a dramatic shift.”

The Bright Computing platform performs provisioning, monitoring, and management down to bare metal. Job schedulers, Slurm for example, are integrated on top of Bright. Start with servers, the physical infrastructure, all running Linux. “We install Bright across that with agents running inside each [server]. I call this the foundation [layer],” said van Leeuwen.

Bright Computaing image 2As shown in the diagram (left), it’s possible to create and manage multiple different clusters (HPC or Hadoop) defined by Bright and implemented through the foundation layer over the physical infrastructure.

From its start Bright has focused on HPC, the natural result of van Leeuwen’s background. His Ph.D. in Stochastic Subsurface Modeling (Imperial College, UK) required HPC resources including running simulations on Cray and Fujitsu supercomputers as well as commodity HPC clusters. In recent years the company has also aggressively incorporated capabilities for handling ‘big data’ oriented infrastructure (Hadoop and Spark) and embraced OpenStack for private clouds.

“Many people don’t realize that OpenStack does not install on bare metal. You need to do a lot of preparation before you can start installing OpenStack. The Bright foundation provides all of that that. Neither does Hadoop install directly on bare metal,” said van Leeuwen. It’s true that Hadoop suppliers such as Hortonworks and Cloudera have begin introducing bare metal deployment mechanisms, he agreed, “but they only work if you have a dedicated Hadoop cluster.”

Positioned as a complete cluster resource allocation/management solution – provisioning, monitoring, security, and updating – Bright works with most job schedulers. “We actually install and configure them and take care of failover of the workload management. That’s a real critical element. We sample all the metrics from the workload management and when a customer grows and shrinks its cluster, we make sure that the workload manager knows so the sys admin will tell Bright I’m adding 10 nodes and bright will automatically in the background reallocate work,” said van Leeuwen.

Users can choose between graphical or command line UIs. van Leeuwen emphasized ease-of-use as a differentiator. The company issues product updates twice yearly, and now tried to closely track OpenStack releases to incorporate its latest features. Here are a few version 7.2 highlights:

  • Bright Cluster Manager includes enhanced support for running containers on nodes using Docker. Administrators and other users can easily set up a web workload orchestration system or to run other containerized workloads with Kubernetes. Administrators can configure the system, add new users, monitor the system, and easily update it. Other updates include improved integration with Puppet to simplify fine-grained control over individual nodes, ZFS support to provide a robust alternative to BtrFS for software image revision control, and support for Intel Omni-Path.
  • Bright Cluster Manager for Big Data has enhanced support for Apache Spark, as well as the latest releases from Apache, Cloudera, and Hortonworks. It now also offers built in integration for such components as Apache Accumulo, Apache Kafka, Apache Pig, and Apache Storm. New tools like Apache Drill, Apache Flink, Apache Zeppelin, and the Tachyon open source memory-centric distributed storage system are now also supported.
  • Bright OpenStack has been updated to OpenStack Liberty, which offers users finer-grained management controls, performance enhancements for large deployments, and more powerful tools for managing new technologies such as containers in production environments. Deployment has been simplified by grouping OpenStack nodes into logical units using configuring overlays, streamlining the cluster manager graphic user interface (CMGUI) OpenStack deployment wizard, and adding a new basic CMGUI wizard.

“The [new] Docker and Kubernetes integration is one of those features that basically leads a whole new track on our roadmap. For some features it’s a one-off; you do it and it is done. Some features are basically keeping up with updates from third party packages etc. Then there’s new functionality that really opens a new track on the road map and container integration is definitely one of those with recurring improvements,” said van Leeuwen.

He noted that container (‘lightweight’) and virtual machine (‘heavyweight’ but more power) advocates continue jockeying for sway. It’s not an either or proposition, van Leeuwen insists. Within HPC world dominated by a knowledgeable user-focused community, containers are often sufficient. Enterprise requirements are often quite varied and require more sys admin coordination.

Bright Computing 3“[In a big company] there’s going to be a department that quickly needs a web server or five servers for SAP and containers are going to be no use to them; they are going to need virtual machines. Or somebody that quickly needs to set up an Oracle database; again containers are going to be no use, they are going to need virtual machines. Containers are more important from a user perspective, from the application perspective, but virtual machines are more important from the systems administrator’s perspective. I think in HPC the users more often need a container than a virtual machine.”

OpenStack, van Leeuwen noted, is adept at managing virtual machines, “It can stand one up, move it around, shut it down, but OpenStack doesn’t really have much control over what’s going on inside the virtual machine. It looks at them as a black box. All these virtual machines run an operating system, mostly Linux, some may run Windows or something else.”

Bright solves this challenge by deploying its agents inside VM’s operating systems, gaining full control of the environment inside the virtual machine. With the agents in place, it’s possible to then put yet another Bright foundation layer on top. This ability to layer is important on mixed-use environments, he said: “We can start building the same things, HPC on top, big data on top, other, and even again open stack, in theory you continue building on top.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

At SC18: AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

Dell EMC’s HPC Chief on Strategy and Emerging Processor Diversity

November 16, 2018

Last January Thierry Pellegrino, a long-time Dell/Dell EMC veteran, became vice president of HPC. His tenure comes at a time when the very definition of HPC is blurring with AI writ large (data analytics, machine learnin Read more…

By John Russell

IBM’s AI-HPC Combine for ‘Intelligent Simulation’: Eliminating the Unnecessary 

November 16, 2018

A powerhouse concept in attaining new knowledge is the notion of the “emergent property,” the combination of formerly stovepiped scientific disciplines and exploratory methods to form cross-disciplinary intelligence Read more…

By Doug Black

HPE Extreme Performance Solutions

AI Can Be Scary. But Choosing the Wrong Partners Can Be Mortifying!

As you continue to dive deeper into AI, you will discover it is more than just deep learning. AI is an extremely complex set of machine learning, deep learning, reinforcement, and analytics algorithms with varying compute, storage, memory, and communications needs. Read more…

IBM Accelerated Insights

From Deep Blue to Summit – 30 Years of Supercomputing Innovation

This week, in honor of the 30th anniversary of the SC conference, we are highlighting some of the most significant IBM contributions to supercomputing over the past 30 years. Read more…

How the United States Invests in Supercomputing

November 14, 2018

The CORAL supercomputers Summit and Sierra are now the world's fastest computers and are already contributing to science with early applications. Ahead of SC18, Maciej Chojnowski with ICM at the University of Warsaw discussed the details of the CORAL project with Dr. Dimitri Kusnezov from the U.S. Department of Energy. Read more…

By Maciej Chojnowski

At SC18: AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

Dell EMC’s HPC Chief on Strategy and Emerging Processor Diversity

November 16, 2018

Last January Thierry Pellegrino, a long-time Dell/Dell EMC veteran, became vice president of HPC. His tenure comes at a time when the very definition of HPC is Read more…

By John Russell

IBM’s AI-HPC Combine for ‘Intelligent Simulation’: Eliminating the Unnecessary 

November 16, 2018

A powerhouse concept in attaining new knowledge is the notion of the “emergent property,” the combination of formerly stovepiped scientific disciplines and Read more…

By Doug Black

How the United States Invests in Supercomputing

November 14, 2018

The CORAL supercomputers Summit and Sierra are now the world's fastest computers and are already contributing to science with early applications. Ahead of SC18, Maciej Chojnowski with ICM at the University of Warsaw discussed the details of the CORAL project with Dr. Dimitri Kusnezov from the U.S. Department of Energy. Read more…

By Maciej Chojnowski

At SC18: Humanitarianism Amid Boom Times for HPC

November 14, 2018

At SC18 in Dallas, the feeling on the ground is one of forward-looking buoyancy. Like boom times that cycle through the Texas oil fields, the HPC industry is en Read more…

By Doug Black

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

New Panasas High Performance Storage Straddles Commercial-Traditional HPC

November 13, 2018

High performance storage vendor Panasas has launched a new version of its ActiveStor product line this morning featuring what the company said is the industry’s first plug-and-play, portable parallel file system that delivers up to 75 Gb/s per rack on industry standard hardware combined with “enterprise-grade reliability and manageability.” Read more…

By Doug Black

SC18 Student Cluster Competition – Revealing the Field

November 13, 2018

It’s November again and we’re almost ready for the kick-off of one of the greatest computer sports events in the world – the SC Student Cluster Competitio Read more…

By Dan Olds

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

TACC Wins Next NSF-funded Major Supercomputer

July 30, 2018

The Texas Advanced Computing Center (TACC) has won the next NSF-funded big supercomputer beating out rivals including the National Center for Supercomputing Ap Read more…

By John Russell

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

Requiem for a Phi: Knights Landing Discontinued

July 25, 2018

On Monday, Intel made public its end of life strategy for the Knights Landing "KNL" Phi product set. The announcement makes official what has already been wide Read more…

By Tiffany Trader

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

At SC18: AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This