BSC Researchers Shrink Floating Point Formats to Accelerate Deep Neural Network Training

By Ken Strandberg

April 15, 2019

Sometimes calculating solutions as precisely as a computer can wastes more CPU resources than is necessary. A case in point is with deep learning. In early stages of training a Deep Neural Network (DNN), a lot of guesswork goes on. The algorithm assigns random values to the weights and computes the error. But the error is enormous in the beginning, and the values of the weights are a long way from the ones selected at the end. Representing weights as a 32-bit floating-point number is costly in terms of processing, yet most of the bits of the mantissa are not needed in early training. As training progresses and it hones the value of the weights, then greater precision becomes important in order to optimize the solution.

Using reduced precision floating point number formats offers benefits in memory footprint and bandwidth and in processing time, which can translate to power savings. These savings can possibly be significant if the benefits can be scaled out to accommodate training of massive DNNs. But will less precision affect overall accuracy of the training?

A lot of research in reduced precision for AI training and inferencing has gone on over the last year. Across Europe and the U.S., industry, academia, and research institutions are looking at this aspect of AI, including U.S. National Labs, Google, and Microsoft. Thus far, the work has resulted in papers, proposals, and some code. Google’s experiments with DNNs have shown that reducing the mantissa in 32-bit floating point numbers for certain calculations of DNNs is okay, “as long as you can represent tiny values closer to zero as part of the summation of small differences during training” (https://en.wikichip.org/wiki/brain_floating-point_format).

Google integrated the bfloat16 format, which provides the same size exponent as the IEEE standard 32-bit FP (float32) but with a smaller mantissa, into some of its products. Bfloat16 is being implemented in a range of future Intel processors for AI deep learning applications.

Intel has integrated a reduced representation format into the Vector Neural Network Instruction (VNNI), a part of Intel Deep Learning Boost (DL Boost), added to the Intel Advanced Vector Extensions 512 instruction set in 2nd generation Xeon Scalable processors.

But the jury is still out on which numbering format or code is best to use at different stages of training and for inferencing. What are the benefits to be gained, in terms of processing performance and power, for the different formats used? And what conditions tell a developer the best format or code to use and when? These are all areas of great interest to Marc Casas, Senior Researcher, at Barcelona Supercomputing Center (BSC).

“We believe dynamic numerical precision approaches offer the best benefit to training and inferencing,” stated Casas. “We are evaluating the applications of many formats and codes, including Intel DL Boost (such as VNNI and others), 32-bit and 64-bit floating point, Flexpoint, and integer formats, at various phases of training neural networks and inferencing.” Flexpoint is a format proposed by Intel for tensors and will be integrated in its Nervana Neural Network processors.

Casas and his team, including John Haiber Osorio Rios and Marc Ortiz of BSC, expect to identify at what phase of training it is best to apply different numerical presentations and how they benefit the network evolution without loss of accuracy. They will also study their impact on processor performance and power consumption on Intel hardware. But, understanding when to use an appropriate format and the impact on the hardware is only one aspect.

“We propose to not only develop innovative ways to exploit the potential of DL Boost and these numerical representations, but to dynamically adjust the Flexpoint/Bfloat16 formats to determine which DL Boost instructions to apply at different phases of training,” add Casas. “We will develop an algorithm to drive these dynamic adjustments based on different proxies describing the network evolution. These adaptive and dynamic schemes used for learning or inferencing phases of DNNs will make it possible to switch across different precisions on runtime.”

Casas says their baseline models are built on Alexnet and Resnet using the Imagenet data set. The project will use software emulations and eventually be applied and evaluated on Intel hardware designed to implement the numerical formats as the next-generation Intel silicon becomes available.

In 2017, BSC installed MareNostrum4, a large supercomputing cluster from Lenovo built on Intel Xeon Scalable processors and Intel Omni-Path Architecture fabric. Casas and his team will use MareNostrum4 to help them answer these questions.

“Understanding the use of dynamic numerical formats and developing schemes to apply them with will change the way industry is training networks,” concluded Casas. “Our work will shed light on allowing a more flexible training mechanism. We will look for ways to apply it to DNN Frameworks, like Intel’s version of Caffe and TensorFlow, so everyone can use it.”

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Cerebras Doubles AI Performance with Second-Gen 7nm Wafer Scale Engine

April 20, 2021

Nearly two years since its massive 1.2 trillion transistor Wafer Scale Engine chip debuted at Hot Chips, Cerebras Systems is announcing its second-generation technology (WSE-2), which its says packs twice the performance Read more…

The New Scalability

April 20, 2021

HPC is all about scalability. The most powerful systems. The biggest data sets. The most cores, the most bytes, the most flops, the most bandwidth. HPC scales! Notwithstanding a few recurring arguments over the last twenty years about scaling up versus scaling out, the definition of scalability... Read more…

Supercomputer-Powered Climate Model Makes Startling Sea Level Rise Prediction

April 19, 2021

The climate science community is tasked with striking a difficult balance: inspiring precisely the amount of alarm commensurate to the climate crisis. Make estimates that are too conservative, and the public might not re Read more…

San Diego Supercomputer Center Opens ‘Expanse’ to Industry Users

April 15, 2021

When San Diego Supercomputer Center (SDSC) at the University of California San Diego was getting ready to deploy its flagship Expanse supercomputer for the large research community it supports, it also sought to optimize Read more…

GTC21: Dell Building Cloud Native Supercomputers at U Cambridge and Durham

April 14, 2021

In conjunction with GTC21, Dell Technologies today announced new supercomputers at universities across DiRAC (Distributed Research utilizing Advanced Computing) in the UK with plans to explore use of Nvidia BlueField DPU technology. The University of Cambridge will expand... Read more…

AWS Solution Channel

Research computing with RONIN on AWS

To allow more visibility into and management of Amazon Web Services (AWS) resources and expenses and minimize the cloud skills training required to operate these resources, AWS Partner RONIN created the RONIN research computing platform. Read more…

The Role and Potential of CPUs in Deep Learning

April 14, 2021

Deep learning (DL) applications have unique architectural characteristics and efficiency requirements. Hence, the choice of computing system has a profound impact on how large a piece of the DL pie a user can finally enj Read more…

Cerebras Doubles AI Performance with Second-Gen 7nm Wafer Scale Engine

April 20, 2021

Nearly two years since its massive 1.2 trillion transistor Wafer Scale Engine chip debuted at Hot Chips, Cerebras Systems is announcing its second-generation te Read more…

The New Scalability

April 20, 2021

HPC is all about scalability. The most powerful systems. The biggest data sets. The most cores, the most bytes, the most flops, the most bandwidth. HPC scales! Notwithstanding a few recurring arguments over the last twenty years about scaling up versus scaling out, the definition of scalability... Read more…

San Diego Supercomputer Center Opens ‘Expanse’ to Industry Users

April 15, 2021

When San Diego Supercomputer Center (SDSC) at the University of California San Diego was getting ready to deploy its flagship Expanse supercomputer for the larg Read more…

GTC21: Dell Building Cloud Native Supercomputers at U Cambridge and Durham

April 14, 2021

In conjunction with GTC21, Dell Technologies today announced new supercomputers at universities across DiRAC (Distributed Research utilizing Advanced Computing) in the UK with plans to explore use of Nvidia BlueField DPU technology. The University of Cambridge will expand... Read more…

The Role and Potential of CPUs in Deep Learning

April 14, 2021

Deep learning (DL) applications have unique architectural characteristics and efficiency requirements. Hence, the choice of computing system has a profound impa Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

Nvidia Aims Clara Healthcare at Drug Discovery, Imaging via DGX

April 12, 2021

Nvidia Corp. continues to expand its Clara healthcare platform with the addition of computational drug discovery and medical imaging tools based on its DGX A100 platform, related InfiniBand networking and its AGX developer kit. The Clara partnerships announced during... Read more…

Nvidia Serves Up Its First Arm Datacenter CPU ‘Grace’ During Kitchen Keynote

April 12, 2021

Today at Nvidia’s annual spring GPU Technology Conference (GTC), held virtually once more due to the pandemic, the company unveiled its first ever Arm-based CPU, called Grace in honor of the famous American programmer Grace Hopper. The announcement of the new... Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers

Contributors

Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire