IBM Pitches Quantum Volume as Benchmarking Tool for Gate-based Quantum Computers

By John Russell

March 6, 2019

IBM this week announced it had achieved its highest Quantum Volume number to date at the American Physical Society (APS) March meeting being held in Boston. What’s Quantum Volume, you ask? Broadly, it’s a ‘holistic measure’ introduced by IBM in a paper last November that’s intended to characterize gate-based quantum computers, regardless of their underlying technology (semiconductor, ion trap, etc.), with a single number. IBM is urging wide adoption of QV by the quantum computing community.

The idea is interesting. The highest QV score so far is 16 which was attained by IBM’s fourth generation 20-qubit IBM Q System One; that’s double the QV of IBM’s 20-qubit IBM Q Network devices. You can see qubit count isn’t the determinant (but it is a factor). Many system-wide facets – gate error rates, decoherence times, qubit connectivity, operating software efficiency, and more – are effectively baked into the measure. In the paper, IBM likens QV to LINPACK for its ability to compare diverse systems.

IBM has laid out a roadmap in which it believes it can roughly double QVs every year. This rate of progress, argues IBM, will produce quantum advantage – which IBM defines as “a quantum computation is either hundreds or thousands of times faster than a classical computation, or needs a smaller fraction of the memory required by a classical computer, or makes something possible that simply isn’t possible now with a classical computer” – in the 2020s.

Addison Snell, CEO, Intersect360 Research noted, “Quantum volume is an interesting metric for tracking progress toward the ability to leverage quantum computing in ways that would be impractical for conventional supercomputers. With the different approaches to quantum computing, it is difficult to compare this achievement across the industry, but it is nevertheless a compelling statistic.”

There’s a lot to unpack here and it’s best done by reading the IBM paper, which isn’t overly long. Bob Sutor, VP, IBM Q Strategy and Ecosystem, and Sarah Sheldon, research staff at IBM T.J. Watson Research Center, briefed HPCwire on QV’s components, use, and relevance to the pursuit of quantum advantage. Before jumping into how Quantum Value is determined, Sutor’s comments on timing and what the magic QV number might be to achieve quantum advantage are interesting.

“We’re not going to go on record saying this or that particular QV number [will produce quantum advantage]. We have now educated hunches based on the different paths that people are taking, that people are taking for chemistry, for AI explorations, for some of the Monte Carlo simulations, and frankly the QV number may be different and probably will be different for each of those. We are certainly on record as saying in the 2020s and we hope in 3-to-5 years,” said Sutor.

The APS meeting served as a broad launchpad for QV with IBM making several presentations on various quantum topics while also seeking to stimulate conversation and urge adoption of QV within the gate-based quantum computing crowd. IBM issued a press release, a more technical blog with data points, and continued promoting the original paper (Validating quantum computers using randomized model circuits) which is freely downloadable. Rigetti has reportedly implemented QV. Noteworthy, QV is not meant for use with adiabatic annealing quantum systems such as D-Wave’s.

A central challenge in quantum computing is the variety of error and system influences that degrade system control and performance. Lacking practical and powerful enough error correction technology, the community has opted for labelling the modern class of quantum computers as noisy intermediate-scale quantum (NISQ) systems. Recognizing this is a situation likely to persist for some time, the IBM paper’s authors[I] do a nice job describing the problem and their approach to measuring performance. Excerpt:

“In these noisy intermediate-scale quantum (NISQ) systems, performance of isolated gates may not predict the behavior of the system. Methods such as randomized benchmarking, state and process tomography, and gateset tomography are valued for measuring the performance of operations on a few qubits, yet they fail to account for errors arising from interactions with spectator qubits. Given a system such as this, whose individual gate operations have been independently calibrated and verified, how do we measure the degree to which the system performs as a general purpose quantum computer? We address this question by introducing a single-number metric, the quantum volume, together with a concrete protocol for measuring it on near-term systems. Similar to how LINPACK is used for comparing diverse classical computers, this metric is not tailored to any particular system, requiring only the ability to implement a universal set of quantum gates.

“The quantum volume protocol we present is strongly linked to gate error rates, and is influenced by underlying qubit connectivity and gate parallelism. It can thus be improved by moving toward the limit in which large numbers of well-controlled, highly coherent, connected, and generically programmable qubits are manipulated within a state-of-the-art circuit rewriting toolchain. High-fidelity state preparation and readout are also necessary. In this work, we evaluate the quantum volume of current IBM Q devices, and corroborate the results with simulations of the same circuits under a depolarizing error model.”

In practice, explained Sheldon, “We generate model circuits which have a specific form where they are sequences of different layers of random entangling gates. The first step is entangling gates between different pairs of qubits on the device. Then we permute the pairing of qubits, into another layer of entangling gates. Each of these layers we call the depth. So if we have three layers, it’s depth3. What we are looking at are circuits we call square circuits with the same number of qubits as the depth in the circuit. Since we are still talking about small enough numbers of qubits that we can simulate these circuits [on classical systems].

“We run an ideal simulation of the circuit and from get a probability distribution of all the possible outcomes. At the end of applying the circuit, the system should be in some state and if we were to measure it we would get a bunch of bit streams, outcomes, with some probabilities. Then we can compare the probabilities from the ideal case to what we actually measured. Based on how close we are to the ideal situation, we say whether or not we were successful. There are details in the paper about how we actually define the success and how we compare the experimental circuits to the ideal circuits. The main point is by doing these model circuits we’re sort of representing a generic quantum algorithm – [we realize] a quantum algorithm doesn’t use random circuits but this is kind of a proxy for that,” she said.

Shown below are some data characterizing IBM systems – IBM Q System One, IBM Q Network systems “Tokyo” and “Poughkeepsie,” and the publicly-available IBM Q Experience system “Tenerife.” As noted in IBM’s blog the performance of a particular quantum computer can be characterized on two levels: metrics associated with the underlying qubits in the chip—what we call the “quantum device”—and overall full-system performance.

“IBM Q System One’s performance is reflected in some of the best/lowest error rates we have ever measured. The average two qubit gate error is less than two percent, and the best gate has less than one percent error rate. Our devices are close to being fundamentally limited by coherence times, which for IBM Q System One averages 73μs,” write Jay Gambetta (IBM Fellow) and Sheldon in the blog. “The mean two-qubit error rate is within a factor of two (x1.68) of the coherence limit, the theoretical limit set by the qubit T1 and T2 (74μs and 69μs on average for IBM Q System One). This indicates that the errors induced by our controls are quite small, and we are achieving close to the best possible qubit fidelities on this device.”

It will be interesting to see how the quantum computing community responds to the CV metric. Back in May when Hyperion Research launched its quantum practice, analyst Bob Sorensen said, “One of the things I’m hoping we can at least play a role in is the idea of thinking about quantum computing benchmarks. Right now, if you read the popular press, and I say ‘IBM’ and the first thing you think of is, yes they have a 50-qubit system. That doesn’t mean much to anybody other than it’s one more qubit than a 49-qubit system. What I am thinking about is asking these people how can we start to characterize across a number of different abstractions and implementations to gain a sense of how we can measure progress.”

IBM has high hopes for Quantum Volume.

Link to release: https://newsroom.ibm.com/2019-03-04-IBM-Achieves-Highest-Quantum-Volume-to-Date-Establishes-Roadmap-for-Reaching-Quantum-Advantage

Link to blog: https://www.ibm.com/blogs/research/2019/03/power-quantum-device/

Link to paper: https://arxiv.org/pdf/1811.12926.pdf

Feature image; IBM Q System One

[i]Validating quantum computers using randomized model circuits, Andrew W. Cross, Lev S. Bishop, Sarah Sheldon, Paul D. Nation, and Jay M. Gambetta IBM T. J. Watson Research Center, https://arxiv.org/pdf/1811.12926.pdf

 

 

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

UT Dallas Grows HPC Storage Footprint for Animation and Game Development

October 28, 2020

Computer-generated animation and video game development are extraordinarily computationally intensive fields, with studios often requiring large server farms with hundreds of terabytes – or even petabytes – of storag Read more…

By Staff report

Frame by Frame, Supercomputing Reveals the Forms of the Coronavirus

October 27, 2020

From the start of the pandemic, supercomputing research has been targeting one particular protein of the coronavirus: the notorious “S” or “spike” protein, which allows the virus to pry its way into human cells a Read more…

By Oliver Peckham

AMD Reports Record Revenue and $35B Deal to Buy Xilinx

October 27, 2020

AMD this morning reported record quarterly revenue of $2.8 billion and a finalized deal to buy FPGA-maker Xilinx for $35 billion in an all-stock transaction. The acquisition helps AMD keep pace during a time of consolida Read more…

By John Russell

Nvidia-Arm Deal a Boon for RISC-V?

October 26, 2020

The $40 billion blockbuster acquisition deal that will bring chip maker Arm into the Nvidia corporate family could provide a boost for the competing RISC-V architecture. As regulators in the U.S., China and the Europe Read more…

By George Leopold

OpenHPC Progress Report – v2.0, More Recipes, Cloud and Arm Support, Says Schulz

October 26, 2020

Launched in late 2015 and transitioned to a Linux Foundation Project in 2016, OpenHPC has marched quietly but steadily forward. Its goal “to provide a reference collection of open-source HPC software components and bes Read more…

By John Russell

AWS Solution Channel

Rapid Chip Design in the Cloud

Time-to-market and engineering efficiency are the most critical and expensive metrics for a chip design company. With this in mind, the team at Annapurna Labs selected Altair AcceleratorRead more…

Intel® HPC + AI Pavilion

Berlin Institute of Health: Putting HPC to Work for the World

Researchers from the Center for Digital Health at the Berlin Institute of Health (BIH) are using science to understand the pathophysiology of COVID-19, which can help to inform the development of targeted treatments. Read more…

NASA Uses Supercomputing to Measure Carbon in the World’s Trees

October 22, 2020

Trees constitute one of the world’s most important carbon sinks, pulling enormous amounts of carbon dioxide from the atmosphere and storing the carbon in their trunks and the surrounding soil. Measuring this carbon sto Read more…

By Oliver Peckham

AMD Reports Record Revenue and $35B Deal to Buy Xilinx

October 27, 2020

AMD this morning reported record quarterly revenue of $2.8 billion and a finalized deal to buy FPGA-maker Xilinx for $35 billion in an all-stock transaction. Th Read more…

By John Russell

OpenHPC Progress Report – v2.0, More Recipes, Cloud and Arm Support, Says Schulz

October 26, 2020

Launched in late 2015 and transitioned to a Linux Foundation Project in 2016, OpenHPC has marched quietly but steadily forward. Its goal “to provide a referen Read more…

By John Russell

Nvidia Dominates (Again) Latest MLPerf Inference Results

October 22, 2020

The two-year-old AI benchmarking group MLPerf.org released its second set of inferencing results yesterday and again, as in the most recent MLPerf training resu Read more…

By John Russell

HPE, AMD and EuroHPC Partner for Pre-Exascale LUMI Supercomputer

October 21, 2020

Not even a week after Nvidia announced that it would be providing hardware for the first four of the eight planned EuroHPC systems, HPE and AMD are announcing a Read more…

By Oliver Peckham

HPE to Build Australia’s Most Powerful Supercomputer for Pawsey

October 20, 2020

The Pawsey Supercomputing Centre in Perth, Western Australia, has had a busy year. Pawsey typically spends much of its time looking to the stars, working with a Read more…

By Oliver Peckham

DDN-Tintri Showcases Technology Integration with Two New Products

October 20, 2020

DDN, a long-time leader in HPC storage, announced two new products today and provided more detail around its strategy for integrating DDN HPC technologies with Read more…

By John Russell

Is the Nvidia A100 GPU Performance Worth a Hardware Upgrade?

October 16, 2020

Over the last decade, accelerators have seen an increasing rate of adoption in high-performance computing (HPC) platforms, and in the June 2020 Top500 list, eig Read more…

By Hartwig Anzt, Ahmad Abdelfattah and Jack Dongarra

Nvidia and EuroHPC Team for Four Supercomputers, Including Massive ‘Leonardo’ System

October 15, 2020

The EuroHPC Joint Undertaking (JU) serves as Europe’s concerted supercomputing play, currently comprising 32 member states and billions of euros in funding. I Read more…

By Oliver Peckham

Supercomputer-Powered Research Uncovers Signs of ‘Bradykinin Storm’ That May Explain COVID-19 Symptoms

July 28, 2020

Doctors and medical researchers have struggled to pinpoint – let alone explain – the deluge of symptoms induced by COVID-19 infections in patients, and what Read more…

By Oliver Peckham

Nvidia Said to Be Close on Arm Deal

August 3, 2020

GPU leader Nvidia Corp. is in talks to buy U.K. chip designer Arm from parent company Softbank, according to several reports over the weekend. If consummated Read more…

By George Leopold

Intel’s 7nm Slip Raises Questions About Ponte Vecchio GPU, Aurora Supercomputer

July 30, 2020

During its second-quarter earnings call, Intel announced a one-year delay of its 7nm process technology, which it says it will create an approximate six-month shift for its CPU product timing relative to prior expectations. The primary issue is a defect mode in the 7nm process that resulted in yield degradation... Read more…

By Tiffany Trader

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

HPE Keeps Cray Brand Promise, Reveals HPE Cray Supercomputing Line

August 4, 2020

The HPC community, ever-affectionate toward Cray and its eponymous founder, can breathe a (virtual) sigh of relief. The Cray brand will live on, encompassing th Read more…

By Tiffany Trader

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Aurora’s Troubles Move Frontier into Pole Exascale Position

October 1, 2020

Intel’s 7nm node delay has raised questions about the status of the Aurora supercomputer that was scheduled to be stood up at Argonne National Laboratory next year. Aurora was in the running to be the United States’ first exascale supercomputer although it was on a contemporaneous timeline with... Read more…

By Tiffany Trader

Is the Nvidia A100 GPU Performance Worth a Hardware Upgrade?

October 16, 2020

Over the last decade, accelerators have seen an increasing rate of adoption in high-performance computing (HPC) platforms, and in the June 2020 Top500 list, eig Read more…

By Hartwig Anzt, Ahmad Abdelfattah and Jack Dongarra

Leading Solution Providers

Contributors

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Nvidia and EuroHPC Team for Four Supercomputers, Including Massive ‘Leonardo’ System

October 15, 2020

The EuroHPC Joint Undertaking (JU) serves as Europe’s concerted supercomputing play, currently comprising 32 member states and billions of euros in funding. I Read more…

By Oliver Peckham

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

Microsoft Azure Adds A100 GPU Instances for ‘Supercomputer-Class AI’ in the Cloud

August 19, 2020

Microsoft Azure continues to infuse its cloud platform with HPC- and AI-directed technologies. Today the cloud services purveyor announced a new virtual machine Read more…

By Tiffany Trader

Oracle Cloud Infrastructure Powers Fugaku’s Storage, Scores IO500 Win

August 28, 2020

In June, RIKEN shook the supercomputing world with its Arm-based, Fujitsu-built juggernaut: Fugaku. The system, which weighs in at 415.5 Linpack petaflops, topp Read more…

By Oliver Peckham

HPE, AMD and EuroHPC Partner for Pre-Exascale LUMI Supercomputer

October 21, 2020

Not even a week after Nvidia announced that it would be providing hardware for the first four of the eight planned EuroHPC systems, HPE and AMD are announcing a Read more…

By Oliver Peckham

DOD Orders Two AI-Focused Supercomputers from Liqid

August 24, 2020

The U.S. Department of Defense is making a big investment in data analytics and AI computing with the procurement of two HPC systems that will provide the High Read more…

By Tiffany Trader

Oracle Cloud Deepens HPC Embrace with Launch of A100 Instances, Plans for Arm, More 

September 22, 2020

Oracle Cloud Infrastructure (OCI) continued its steady ramp-up of HPC capabilities today with a flurry of announcements. Topping the list is general availabilit Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This