IBM Pitches Quantum Volume as Benchmarking Tool for Gate-based Quantum Computers

By John Russell

March 6, 2019

IBM this week announced it had achieved its highest Quantum Volume number to date at the American Physical Society (APS) March meeting being held in Boston. What’s Quantum Volume, you ask? Broadly, it’s a ‘holistic measure’ introduced by IBM in a paper last November that’s intended to characterize gate-based quantum computers, regardless of their underlying technology (semiconductor, ion trap, etc.), with a single number. IBM is urging wide adoption of QV by the quantum computing community.

The idea is interesting. The highest QV score so far is 16 which was attained by IBM’s fourth generation 20-qubit IBM Q System One; that’s double the QV of IBM’s 20-qubit IBM Q Network devices. You can see qubit count isn’t the determinant (but it is a factor). Many system-wide facets – gate error rates, decoherence times, qubit connectivity, operating software efficiency, and more – are effectively baked into the measure. In the paper, IBM likens QV to LINPACK for its ability to compare diverse systems.

IBM has laid out a roadmap in which it believes it can roughly double QVs every year. This rate of progress, argues IBM, will produce quantum advantage – which IBM defines as “a quantum computation is either hundreds or thousands of times faster than a classical computation, or needs a smaller fraction of the memory required by a classical computer, or makes something possible that simply isn’t possible now with a classical computer” – in the 2020s.

Addison Snell, CEO, Intersect360 Research noted, “Quantum volume is an interesting metric for tracking progress toward the ability to leverage quantum computing in ways that would be impractical for conventional supercomputers. With the different approaches to quantum computing, it is difficult to compare this achievement across the industry, but it is nevertheless a compelling statistic.”

There’s a lot to unpack here and it’s best done by reading the IBM paper, which isn’t overly long. Bob Sutor, VP, IBM Q Strategy and Ecosystem, and Sarah Sheldon, research staff at IBM T.J. Watson Research Center, briefed HPCwire on QV’s components, use, and relevance to the pursuit of quantum advantage. Before jumping into how Quantum Value is determined, Sutor’s comments on timing and what the magic QV number might be to achieve quantum advantage are interesting.

“We’re not going to go on record saying this or that particular QV number [will produce quantum advantage]. We have now educated hunches based on the different paths that people are taking, that people are taking for chemistry, for AI explorations, for some of the Monte Carlo simulations, and frankly the QV number may be different and probably will be different for each of those. We are certainly on record as saying in the 2020s and we hope in 3-to-5 years,” said Sutor.

The APS meeting served as a broad launchpad for QV with IBM making several presentations on various quantum topics while also seeking to stimulate conversation and urge adoption of QV within the gate-based quantum computing crowd. IBM issued a press release, a more technical blog with data points, and continued promoting the original paper (Validating quantum computers using randomized model circuits) which is freely downloadable. Rigetti has reportedly implemented QV. Noteworthy, QV is not meant for use with adiabatic annealing quantum systems such as D-Wave’s.

A central challenge in quantum computing is the variety of error and system influences that degrade system control and performance. Lacking practical and powerful enough error correction technology, the community has opted for labelling the modern class of quantum computers as noisy intermediate-scale quantum (NISQ) systems. Recognizing this is a situation likely to persist for some time, the IBM paper’s authors[I] do a nice job describing the problem and their approach to measuring performance. Excerpt:

“In these noisy intermediate-scale quantum (NISQ) systems, performance of isolated gates may not predict the behavior of the system. Methods such as randomized benchmarking, state and process tomography, and gateset tomography are valued for measuring the performance of operations on a few qubits, yet they fail to account for errors arising from interactions with spectator qubits. Given a system such as this, whose individual gate operations have been independently calibrated and verified, how do we measure the degree to which the system performs as a general purpose quantum computer? We address this question by introducing a single-number metric, the quantum volume, together with a concrete protocol for measuring it on near-term systems. Similar to how LINPACK is used for comparing diverse classical computers, this metric is not tailored to any particular system, requiring only the ability to implement a universal set of quantum gates.

“The quantum volume protocol we present is strongly linked to gate error rates, and is influenced by underlying qubit connectivity and gate parallelism. It can thus be improved by moving toward the limit in which large numbers of well-controlled, highly coherent, connected, and generically programmable qubits are manipulated within a state-of-the-art circuit rewriting toolchain. High-fidelity state preparation and readout are also necessary. In this work, we evaluate the quantum volume of current IBM Q devices, and corroborate the results with simulations of the same circuits under a depolarizing error model.”

In practice, explained Sheldon, “We generate model circuits which have a specific form where they are sequences of different layers of random entangling gates. The first step is entangling gates between different pairs of qubits on the device. Then we permute the pairing of qubits, into another layer of entangling gates. Each of these layers we call the depth. So if we have three layers, it’s depth3. What we are looking at are circuits we call square circuits with the same number of qubits as the depth in the circuit. Since we are still talking about small enough numbers of qubits that we can simulate these circuits [on classical systems].

“We run an ideal simulation of the circuit and from get a probability distribution of all the possible outcomes. At the end of applying the circuit, the system should be in some state and if we were to measure it we would get a bunch of bit streams, outcomes, with some probabilities. Then we can compare the probabilities from the ideal case to what we actually measured. Based on how close we are to the ideal situation, we say whether or not we were successful. There are details in the paper about how we actually define the success and how we compare the experimental circuits to the ideal circuits. The main point is by doing these model circuits we’re sort of representing a generic quantum algorithm – [we realize] a quantum algorithm doesn’t use random circuits but this is kind of a proxy for that,” she said.

Shown below are some data characterizing IBM systems – IBM Q System One, IBM Q Network systems “Tokyo” and “Poughkeepsie,” and the publicly-available IBM Q Experience system “Tenerife.” As noted in IBM’s blog the performance of a particular quantum computer can be characterized on two levels: metrics associated with the underlying qubits in the chip—what we call the “quantum device”—and overall full-system performance.

“IBM Q System One’s performance is reflected in some of the best/lowest error rates we have ever measured. The average two qubit gate error is less than two percent, and the best gate has less than one percent error rate. Our devices are close to being fundamentally limited by coherence times, which for IBM Q System One averages 73μs,” write Jay Gambetta (IBM Fellow) and Sheldon in the blog. “The mean two-qubit error rate is within a factor of two (x1.68) of the coherence limit, the theoretical limit set by the qubit T1 and T2 (74μs and 69μs on average for IBM Q System One). This indicates that the errors induced by our controls are quite small, and we are achieving close to the best possible qubit fidelities on this device.”

It will be interesting to see how the quantum computing community responds to the CV metric. Back in May when Hyperion Research launched its quantum practice, analyst Bob Sorensen said, “One of the things I’m hoping we can at least play a role in is the idea of thinking about quantum computing benchmarks. Right now, if you read the popular press, and I say ‘IBM’ and the first thing you think of is, yes they have a 50-qubit system. That doesn’t mean much to anybody other than it’s one more qubit than a 49-qubit system. What I am thinking about is asking these people how can we start to characterize across a number of different abstractions and implementations to gain a sense of how we can measure progress.”

IBM has high hopes for Quantum Volume.

Link to release:

Link to blog:

Link to paper:

Feature image; IBM Q System One

[i]Validating quantum computers using randomized model circuits, Andrew W. Cross, Lev S. Bishop, Sarah Sheldon, Paul D. Nation, and Jay M. Gambetta IBM T. J. Watson Research Center,



Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Finland’s CSC Chronicles the COVID Research Performed on Its ‘Puhti’ Supercomputer

May 11, 2021

CSC, Finland’s IT Center for Science, is home to a variety of computing resources, including the 1.7 petaflops Puhti supercomputer. The 682-node, Intel Cascade Lake-powered system, which places about halfway down the T Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x speedup in simulating molecules. Qiskit is IBM’s quantum soft Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base clock of 2.0GHz – implemented in HPE's single-socket ProLian Read more…

Supercomputer Research Tracks the Loss of the World’s Glaciers

May 7, 2021

British Columbia – which is over twice the size of California – contains around 17,000 glaciers that cover three percent of its landmass. These glaciers are crucial for the Canadian province, which relies on its many Read more…

Meet Dell’s Pete Manca, an HPCwire Person to Watch in 2021

May 7, 2021

Pete Manca heads up Dell's newly formed HPC and AI leadership group. As senior vice president of the integrated solutions engineering team, he is focused on custom design, technology alliances, high-performance computing Read more…

AWS Solution Channel

FLYING WHALES runs CFD workloads 15 times faster on AWS

FLYING WHALES is a French startup that is developing a 60-ton payload cargo airship for the heavy lift and outsize cargo market. The project was born out of France’s ambition to provide efficient, environmentally friendly transportation for collecting wood in remote areas. Read more…

Fast Pass Through (Some of) the Quantum Landscape with ORNL’s Raphael Pooser

May 7, 2021

In a rather remarkable way, and despite the frequent hype, the behind-the-scenes work of developing quantum computing has dramatically accelerated in the past few years. DOE, NSF, academia and industry are all in the rac Read more…

IBM Debuts Qiskit Runtime for Quantum Computing; Reports Dramatic Speed-up

May 11, 2021

In conjunction with its virtual Think event, IBM today introduced an enhanced Qiskit Runtime Software for quantum computing, which it says demonstrated 120x spe Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Fast Pass Through (Some of) the Quantum Landscape with ORNL’s Raphael Pooser

May 7, 2021

In a rather remarkable way, and despite the frequent hype, the behind-the-scenes work of developing quantum computing has dramatically accelerated in the past f Read more…

IBM Research Debuts 2nm Test Chip with 50 Billion Transistors

May 6, 2021

IBM Research today announced the successful prototyping of the world's first 2 nanometer chip, fabricated with silicon nanosheet technology on a standard 300mm Read more…

LRZ Announces New Phase of SuperMUC-NG Supercomputer with Intel’s ‘Ponte Vecchio’ GPU

May 5, 2021

At the Leibniz Supercomputing Centre (LRZ) in München, Germany – one of the constituent centers of the Gauss Centre for Supercomputing (GCS) – the SuperMUC Read more…

Crystal Ball Gazing at Nvidia: R&D Chief Bill Dally Talks Targets and Approach

May 4, 2021

There’s no quibbling with Nvidia’s success. Entrenched atop the GPU market, Nvidia has ridden its own inventiveness and growing demand for accelerated computing to meet the needs of HPC and AI. Recently it embarked on an ambitious expansion by acquiring Mellanox (interconnect)... Read more…

Intel Invests $3.5 Billion in New Mexico Fab to Focus on Foveros Packaging Technology

May 3, 2021

Intel announced it is investing $3.5 billion in its Rio Rancho, New Mexico, facility to support its advanced 3D manufacturing and packaging technology, Foveros. Read more…

Supercomputer Research Shows Standard Model May Withstand Muon Discrepancy

May 3, 2021

Big news recently struck the physics world: researchers at the Fermi National Accelerator Laboratory (FNAL), in the midst of their Muon g-2 experiment, publishe Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from I Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

Quantum Computer Start-up IonQ Plans IPO via SPAC

March 8, 2021

IonQ, a Maryland-based quantum computing start-up working with ion trap technology, plans to go public via a Special Purpose Acquisition Company (SPAC) merger a Read more…

Leading Solution Providers


Can Deep Learning Replace Numerical Weather Prediction?

March 3, 2021

Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate modeling, and Read more…

AMD Launches Epyc ‘Milan’ with 19 SKUs for HPC, Enterprise and Hyperscale

March 15, 2021

At a virtual launch event held today (Monday), AMD revealed its third-generation Epyc “Milan” CPU lineup: a set of 19 SKUs -- including the flagship 64-core, 280-watt 7763 part --  aimed at HPC, enterprise and cloud workloads. Notably, the third-gen Epyc Milan chips achieve 19 percent... Read more…

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

The History of Supercomputing vs. COVID-19

March 9, 2021

The COVID-19 pandemic poses a greater challenge to the high-performance computing community than any before. HPCwire's coverage of the supercomputing response t Read more…

HPE Names Justin Hotard New HPC Chief as Pete Ungaro Departs

March 2, 2021

HPE CEO Antonio Neri announced today (March 2, 2021) the appointment of Justin Hotard as general manager of HPC, mission critical solutions and labs, effective Read more…

  • arrow
  • Click Here for More Headlines
  • arrow