HPE, EPFL Launch Blue Brain 5 Supercomputer

By John Russell

July 10, 2018

HPE and the Ecole Polytechnique Federale de Lausannne (EPFL) Blue Brain Project yesterday introduced Blue Brain 5, a new supercomputer built by HPE, which displaces a long line of IBM Blue Gene systems that previously supported the 13-year-old Blue Brain Project whose ambitious goal is to “digitally reconstruct and simulate” the human brain. The Blue Brain project, which has occasionally stirred debate among European brain researchers, came into being in June 2005 when IBM and EPFL signed an agreement to launch the project and install a Blue Gene at EPFL – hence the name Blue Brain for the machine.

The most recent system, Blue Brain 4, was a Blue Gene/Q machine installed in 2014 at the Swiss National Computing Centre (CSCS). When IBM stopped development of Blue Gene around 2015, EPFL was forced to broaden its search for next generation supercomputing technology, said Felix Shürmann, EPFL, co-director EPFL Blue Brain Project during a press briefing yesterday. (Project timeline posted on the project website.)

Many vendors (presumably IBM among them) responded to its RFP and in the end, “[HPE] not only fulfilled the technical specs and scored well on the price, but I think what is important about HPE is that they are a technology company with their own research and forward-looking perspective to computing and the upcoming challenges, and this notion that this is a technology partner that allows us to navigate this space of available technologies and make the right choices, that was an important part in the selection process,” said Shürmann.

HPE SGI 8600 System

This is clearly a win for HPE although IBM may not be losing any sleep given its giant success on the most recent Top500 list with Summit (122PF, Rmax) on top and Sierra (71.6PF, Rmax) at number three (See HPCwire article, GPUs Power Five of World’s Top Seven Supercomputers).

Bragging rights aside, the new HPE machine, say both HPE and EPFL, was carefully designed to meet the massive data handling and simulation requirements of modern brain research. Repeating numbers you have likely already heard, there are on the order of 100 billion neurons and 100-to-1,000 trillion synapses in the human brain. Moreover there is a bewildering number of ways in which neurons and synapses interact helped along by a rather long list of other players (proteins, glial cells, etc.) all operating in a multi-scale system that spans the tiny (molecules) to the comparatively large (regions of the brain).

Henry Markram, EPFL Brain Project founder and co-director, said, “To gather all this data and to organize it and to make sense of how all this data fits together is really one of the biggest big data challenges that exists today.”

Schürmann added, “The Blue Brain Project’s scientific mission is critically dependent on our supercomputing capabilities. Modeling an individual neuron at Blue Brain today leads to around 20,000 ordinary differential equations – when modeling entire brain regions, this quickly raises to 100 billion equations that have to be solved concurrently. HPE helps us to navigate the challenging technology landscape in supercomputing.”

The Blue Brain 5 core system is an HPE SGI 8600 system comprised of 372 compute nodes. HPE says it delivers 1.06 petaflops of peak performance and can be easily scaled up. The system has 94 terabytes of memory and runs Intel Xeon Gold 6140 and Intel Xeon Phi 7230 processors as well as Nvidia Tesla V100 graphic processors. Blue Brain 5 uses single and dual-rail Mellanox InfiniBand high-performance networks and has 4 petabytes of high-performance storage from DataDirect Networks (DDN), delivering more than 50 GB/s aggregated bandwidth, associated with an innovative 80 GB/s Infinite Memory Engine (IME) flash-based burst buffer. Blue Brain 5 was installed at CSCS this spring and is now running at full production.

The secret sauce – and HPE didn’t reveal much technical detail yesterday – is how HPE has apparently blended four systems into one. Its flexible architecture, says HPE, can host different sub systems that are specifically geared for tasks like visualization or deep learning, while being operated as one single system.

“Our first focus was on the workflow of the Blue Brain project,” said Eng Lim Goh, VP, CTO, HPC and AI, HPE. “We came up with a supercomputer that is made up of four subsystems, tightly bound together to work as one. The four subsystems include one that is strong for extracting data from storage, another subsystem was strong and designed to do well at extracting data fast from memory, the third one was for visualization, and a fourth one was a general purpose one spans across the other three. on top of that we bound the four subsystems tightly together with high bandwidth network, and in some areas doubling up on the bandwidth.”

“As you can see it was a detailed design process done by studying the workflow of the project and designing a supercomputer that matches the workflow. For example, when the workflow starts and it is trying to discover all the possible touch points from the neuron onward, the high IO bandwidth subsystem, the one that can extract data fast from storage, is first employed. After which, when simulation is actually done in detail, and at high resolution, we switch to the second subsystem where we can extract data very fast from memory. Finally when you visualize, the visualization subsystem is used. This gives you one example of how we looked at the workflow carefully and designed a system to match it,” explained Goh.

Broadly, HPE characterizes the subsystems as:

  • Subsystem 1: Intel KNL, 16GB of HBM, 96GB DRAM
  • Subsystem 2: Dual Intel Xeon, 768 GB memory + 4 Nvidia V100 GPUs
  • Subsystem 3: Dual Intel Xeon, 384 GB memory
  • Subsystem 4: Dual Intel Xeon, 384 GB memory + 2 NVME

Brain research is expensive and over the years there has been vigorous discussion over how best to spend scarce research dollars. The EPFL Blue Brain project, led by Markram, sometimes drew criticism for its approach. In recent years, the Swiss-funded Blue Brain project has become part of the broader European-wide Human Brain Project (HBP) whose compass is broader. The HBP is exploring everything from new neuromorphic processor technology, to bioinformatics tools development, and specific disease research.

Said Markram, “What is very different and unique about the Blue Brain Project is that we are very much focused on the biology. Our goal is to be able to capture [brain biology] with as high fidelity as possible – as much as the computing allows us and as the biological data informs us – to be able to build models that are as accurate and as the actual biological specimen. So in a way it is a digital reconstruction.”

Henry Markram, EPFL Blue Brain Project

The insights can later be applied to other cognitive and biomedical research. One of the Blue Brain Project’s discoveries, said Markram, is how ‘micro circuits’ in the brain are structured and behave.

“We deliberately chose the micro circuit. It’s the minimal systems of neurons. Neurons need other neurons. There’s a minimum number of them [to form a micro circuit]. In the mammalian cortex that turned out to be 30,000, which is a number we discovered through the simulation; it was not a number that was known in experiments or through theory. That’s the minimum ecosystem. We wanted to identify that first because that provides the sort of unit of operation in the mammalian neocortex,” said Markram.

“This gave us a new insight into what the brain is trying to process but what we don’t know today is how that changes when you go from a micro circuit to a brain region such as the touch region or the vision region or the hearing region. These are larger regions [and] can be composed of several hundreds of these micro circuits. It’s a question of how all these micro circuits interact that we are now going to be able to explore with the kind of computing that we have obtained with HPE. That will allow us to begin understanding how entire sensory modalities are forming in response to when the sensory input comes in, which neurons are active, which synapses are active, and how these micro circuits are interacting to sculpture an electrical landscape which is effectively coding information for what is happening in the outside world.”

HPE has long been active in life sciences research including NIH’s Living Heart Project and the work with the Center for Neurodegenerative Diseases and the University of Bonn targeting, among other things Alzheimer’s Disease.

During yesterday’s press briefing, a question about needing exascale compute power came up. It will indeed be helpful agreed EPFL and HPE. HPE, of course, has been an active participant in the Path Forward program in the U.S. although it isn’t currently the prime contractor on any U.S.-funded pre-exascale machines. Markram was quick to note that simulating a full human brain was probably not realistic any time soon.

“It depends on the resolution of the simulation,” he said. “If you take it to the extreme level, as I’ve said, you have a billion organic proteins in a single neuron and you got 100 billion of them and their interactions are happening at sort of microsecond scale. If you had to do the calculation, if you wanted to simulate every molecule in the human brain you would probably need a computer that’s about 1030 flops so that’s way beyond the yotta scale (1024 ). This is not what we are talking about when we say we are going to simulate the brain.

“What we are doing is building – with the constraints of what the computing can allow us – technologies [that] we call multiscale simulation. That means that a lot of the big parts of the brain would be simulated at low resolution. That neurons may not be as elaborate, they may be even point neurons. Not all the molecules in all of the neurons would be simulated. But when you identify an area of interest, you would be able to zoom in effectively and as you zoom in you’d be able to see the region, the activity there at much higher resolution. We are building models that allow you to go down to not just the neurons but the supporting cell which are of course ether glial cells and even down to the blood vessels to see how neurons are being supported by the blood supply.”

Big Science takes time and, increasingly, lots of computational power.

Blue Brain Project video:

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Graphcore Introduces Next-Gen Intelligence Processing Unit for AI Workloads

July 15, 2020

British hardware designer Graphcore, which emerged from stealth in 2016 to launch its first-generation Intelligence Processing Unit (IPU), has announced its next-generation IPU platform: the IPU-Machine M2000. With the n Read more…

By Oliver Peckham

heFFTe: Scaling FFT for Exascale

July 15, 2020

Exascale computing aspires to provide breakthrough solutions addressing today’s most critical challenges in scientific discovery, energy assurance, economic competitiveness, and national security. This has been the mai Read more…

By Jack Dongarra and Stanimire Tomov

There’s No Storage Like ATGC: Breakthrough Helps to Store ‘The Wizard of Oz’ in DNA

July 15, 2020

Even as storage density reaches new heights, many researchers have their eyes set on a paradigm shift in high-density information storage: storing data in the four nucleotides (A, T, G and C) that constitute DNA, a metho Read more…

By Oliver Peckham

Get a Grip: Intel Neuromorphic Chip Used to Give Robotics Arm a Sense of Touch

July 15, 2020

Moving neuromorphic technology from the laboratory into practice has proven slow-going. This week, National University of Singapore researchers moved the needle forward demonstrating an event-driven, visual-tactile perce Read more…

By John Russell

What’s New in HPC Research: Volcanoes, Mobile Games, Proteins & More

July 14, 2020

In this bimonthly feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

By Oliver Peckham

AWS Solution Channel

INEOS TEAM UK Accelerates Boat Design for America’s Cup Using HPC on AWS

The America’s Cup Dream

The 36th America’s Cup race will be decided in Auckland, New Zealand in 2021. Like all the teams, INEOS TEAM UK will compete in a boat whose design will have followed guidelines set by race organizers to ensure the crew’s sailing skills are fully tested. Read more…

Intel® HPC + AI Pavilion

Supercomputing the Pandemic: Scientific Community Tackles COVID-19 from Multiple Perspectives

Since their inception, supercomputers have taken on the biggest, most complex, and most data-intensive computing challenges—from confirming Einstein’s theories about gravitational waves to predicting the impacts of climate change. Read more…

Joliot-Curie Supercomputer Used to Build First Full, High-Fidelity Aircraft Engine Simulation

July 14, 2020

When industrial designers plan the design of a new element of a vehicle’s propulsion or exterior, they typically use fluid dynamics to optimize airflow and increase the vehicle’s speed and efficiency. These fluid dyn Read more…

By Oliver Peckham

Graphcore Introduces Next-Gen Intelligence Processing Unit for AI Workloads

July 15, 2020

British hardware designer Graphcore, which emerged from stealth in 2016 to launch its first-generation Intelligence Processing Unit (IPU), has announced its nex Read more…

By Oliver Peckham

heFFTe: Scaling FFT for Exascale

July 15, 2020

Exascale computing aspires to provide breakthrough solutions addressing today’s most critical challenges in scientific discovery, energy assurance, economic c Read more…

By Jack Dongarra and Stanimire Tomov

Get a Grip: Intel Neuromorphic Chip Used to Give Robotics Arm a Sense of Touch

July 15, 2020

Moving neuromorphic technology from the laboratory into practice has proven slow-going. This week, National University of Singapore researchers moved the needle Read more…

By John Russell

Max Planck Society Begins Installation of Liquid-Cooled Supercomputer from Lenovo

July 9, 2020

Lenovo announced today that it is supplying a new high performance computer to the Max Planck Society, one of Germany's premier research organizations. Comprise Read more…

By Tiffany Trader

President’s Council Targets AI, Quantum, STEM; Recommends Spending Growth

July 9, 2020

Last week the President Council of Advisors on Science and Technology (PCAST) met (webinar) to review policy recommendations around three sub-committee reports: Read more…

By John Russell

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

Q&A: HLRS’s Bastian Koller Tackles HPC and Industry in Germany and Europe

July 6, 2020

In this exclusive interview for HPCwire – sadly not face to face – Steve Conway, senior advisor for Hyperion Research, talks with Dr.-Ing Bastian Koller about the state of HPC and its collaboration with Industry in Europe. Koller is a familiar figure in HPC. He is the managing director at High Performance Computing Center Stuttgart (HLRS) and also serves... Read more…

By Steve Conway, Hyperion

OpenPOWER Reboot – New Director, New Silicon Partners, Leveraging Linux Foundation Connections

July 2, 2020

Earlier this week the OpenPOWER Foundation announced the contribution of IBM’s A21 Power processor core design to the open source community. Roughly this time Read more…

By John Russell

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

[email protected] Turns Its Massive Crowdsourced Computer Network Against COVID-19

March 16, 2020

For gamers, fighting against a global crisis is usually pure fantasy – but now, it’s looking more like a reality. As supercomputers around the world spin up Read more…

By Oliver Peckham

[email protected] Rallies a Legion of Computers Against the Coronavirus

March 24, 2020

Last week, we highlighted [email protected], a massive, crowdsourced computer network that has turned its resources against the coronavirus pandemic sweeping the globe – but [email protected] isn’t the only game in town. The internet is buzzing with crowdsourced computing... Read more…

By Oliver Peckham

Supercomputer Simulations Reveal the Fate of the Neanderthals

May 25, 2020

For hundreds of thousands of years, neanderthals roamed the planet, eventually (almost 50,000 years ago) giving way to homo sapiens, which quickly became the do Read more…

By Oliver Peckham

DoE Expands on Role of COVID-19 Supercomputing Consortium

March 25, 2020

After announcing the launch of the COVID-19 High Performance Computing Consortium on Sunday, the Department of Energy yesterday provided more details on its sco Read more…

By John Russell

Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

Honeywell’s Big Bet on Trapped Ion Quantum Computing

April 7, 2020

Honeywell doesn’t spring to mind when thinking of quantum computing pioneers, but a decade ago the high-tech conglomerate better known for its control systems waded deliberately into the then calmer quantum computing (QC) waters. Fast forward to March when Honeywell announced plans to introduce an ion trap-based quantum computer whose ‘performance’ would... Read more…

By John Russell

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Leading Solution Providers

Contributors

Nvidia’s Ampere A100 GPU: Up to 2.5X the HPC, 20X the AI

May 14, 2020

Nvidia's first Ampere-based graphics card, the A100 GPU, packs a whopping 54 billion transistors on 826mm2 of silicon, making it the world's largest seven-nanom Read more…

By Tiffany Trader

‘Billion Molecules Against COVID-19’ Challenge to Launch with Massive Supercomputing Support

April 22, 2020

Around the world, supercomputing centers have spun up and opened their doors for COVID-19 research in what may be the most unified supercomputing effort in hist Read more…

By Oliver Peckham

Australian Researchers Break All-Time Internet Speed Record

May 26, 2020

If you’ve been stuck at home for the last few months, you’ve probably become more attuned to the quality (or lack thereof) of your internet connection. Even Read more…

By Oliver Peckham

15 Slides on Programming Aurora and Exascale Systems

May 7, 2020

Sometime in 2021, Aurora, the first planned U.S. exascale system, is scheduled to be fired up at Argonne National Laboratory. Cray (now HPE) and Intel are the k Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

TACC Supercomputers Run Simulations Illuminating COVID-19, DNA Replication

March 19, 2020

As supercomputers around the world spin up to combat the coronavirus, the Texas Advanced Computing Center (TACC) is announcing results that may help to illumina Read more…

By Staff report

$100B Plan Submitted for Massive Remake and Expansion of NSF

May 27, 2020

Legislation to reshape, expand - and rename - the National Science Foundation has been submitted in both the U.S. House and Senate. The proposal, which seems to Read more…

By John Russell

John Martinis Reportedly Leaves Google Quantum Effort

April 21, 2020

John Martinis, who led Google’s quantum computing effort since establishing its quantum hardware group in 2014, has left Google after being moved into an advi Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This