Benchmarking is an important topic in quantum computing. There’s consensus it’s needed but opinions vary widely on how to go about it. Last week, IBM introduced a new tool — Benchpress — intended to help evaluate various quantum computer SDKs and stacks. IBM issued a paper (Benchmarking the performance of quantum computing software) describing the new tool.

At first glance Benchpress sounds interesting. Here’s the paper’s abstract.

“We present Benchpress, a benchmarking suite for evaluating the performance and range of functionality of multiple quantum computing software development kits. This suite consists of a collection of over 1000 tests measuring key performance metrics for a wide variety of operations on quantum circuits comprised of up to 930 qubits and O(10^6) two-qubit gates, as well as an execution framework for running the tests over multiple quantum software packages in a unified manner.

“We give a detailed overview of the benchmark suite, its methodology, and generate representative results over seven different quantum software packages. The flexibility of the Benchpress framework allows for benchmarking that not only keeps pace with quantum hardware improvements but can preemptively gauge the quantum circuit processing costs of future device architectures. Being open-source, Benchpress ensures the transparency and verification of performance claims.”

A key issue in quantum computing benchmarking is picking what to benchmark. Some early benchmarks, such as Quantum Volume developed by IBM, are reasonable measures for internal system quality (gate fidelity, speed, coherence) but not especially helpful for giving guidance on how an application/algorithm will run. The Quantum Economic Development Consortium (QED-C) has an effort to develop benchmark tools focused on application/algorithm performance on different types of quantum computers. Stack and SDK tools assessment is another area ripe for development.

IBM researcher note, “There is a variety of quantum software development kits (SDKs) that perform all or some fraction of quantum circuit pre-execution processing, such as Braket, BQSKit, Cirq, Qiskit, Qiskit Transpiler Service (QTS) extension, Staq, and Tket, amongst others. Here we define an SDK to be a collection of software development tools within a single installable package. The need to evaluate the performance of these software stacks has led to the creation of several collections of assessment circuits and Hamiltonians, as well as some examples of how to execute collections of circuits in a benchmarking framework.”

“Benchpress stands out for its common framework that allows testing across three key areas: quantum circuit construction, manipulation, and optimization. Although the intersection of functionality varies dramatically across the quantum computing software landscape, Benchpress utilizes notional collections of tests called “workouts” that allow the full test-suite to be executed across any quantum SDK with tests defaulting to being skipped if they are not explicitly overwritten with an SDK specific implementation. Benchpress is thus able to quantify not only relative performance metrics amongst SDKs but also the breadth of functionality in a given software package.”

Perhaps not surprisingly, Qiskit developed by IBM but since open sourced, performs well when tested using Benchpress. See table and figure below. IBM has also blogged about the Qiskit showing.

IBM contends Benchpress allows for accommodating a wide variety of quantum computing frameworks, “leveraging the interoperability of Qiskit to provide a uniform testing environment, irrespective of SDK functionality.” The researcher note that “Benchpress also allows for measuring the memory consumption of SDKs for each test case using Memray. However, this analysis comes with a substantial overhead.”

The paper is a good read and the devil, as always, will be in the details. It will be interesting to see how widely Benchpress is adopted. QV has a fairly substantial following in the industry and is used by competitors.

Link to paper, https://arxiv.org/abs/2409.08844

Link to IBM announcement, https://www.ibm.com/quantum/blog/qiskit-performance