[Paper Review] Quality, Speed, and Scale: three key attributes to measure the performance of near-term quantum computers

Andrew Wack, Hanhee Paik+5 more|arXiv (Cornell University)|Oct 26, 2021

Quantum Computing Algorithms and Architecture55 citations

TL;DR

The paper defines three performance attributes for near-term quantum computers—scale, quality, and speed—and introduces a CLOPS benchmark to measure speed, alongside quantum volume for quality and qubit count for scale, with measurements on IBM systems.

ABSTRACT

Defining the right metrics to properly represent the performance of a quantum computer is critical to both users and developers of a computing system. In this white paper, we identify three key attributes for quantum computing performance: quality, speed, and scale. Quality and scale are measured by quantum volume and number of qubits, respectively. We propose a speed benchmark, using an update to the quantum volume experiments that allows the measurement of Circuit Layer Operations Per Second (CLOPS) and identify how both classical and quantum components play a role in improving performance. We prescribe a procedure for measuring CLOPS and use it to characterize the performance of some IBM Quantum systems.

Motivation & Objective

• Motivate the need for holistic benchmarks that reflect quantum-classical interactions in real workloads.
• Define three core performance attributes: scale (qubits), quality (quantum volume), and speed (CLOPS).
• Present and operationalize the CLOPS benchmark for measuring speed in practical quantum computing use cases.
• Describe how quantum volume and CLOPS interrelate and how runtime compilation and data transfer affect overall performance.

Proposed method

• Define and justify three performance attributes (scale, quality, speed).
• Adopt and update Quantum Volume as the quality measure and introduce a Circuit Layer Operations Per Second (CLOPS) speed benchmark.
• Describe a 100-circuit-template, parameterized CLOPS workload to capture run-time and data-transfer overheads.
• Provide a detailed procedure for measuring CLOPS, including runtime compilation, parameter updates, and data transfer times.
• Apply CLOPS to IBM Quantum systems to illustrate real-world bottlenecks and performance drivers.

Experimental results

Research questions

• RQ1 How should quantum computer performance be characterized to reflect real user workloads involving quantum-classical interactions?
• RQ2 How do scale, quality, and speed interact, and what benchmarks best capture these interactions?
• RQ3 What is the practical performance (CLOPS) of current IBM Quantum systems under a representative, parameterized circuit workload?

Key findings

• Quality is captured by Quantum Volume, which reflects coherence, gate fidelity, measurement fidelity, connectivity, and compiler effects.
• Scale is represented by the number of qubits, influencing problem size and potential to improve other metrics.
• CLOPS provides a holistic speed metric by measuring QV layers executed per second, incorporating circuit execution, runtime compilation, data transfer, and parameter updates.
• Measured CLOPS varies across devices with the same Quantum Volume (e.g., 5-qubit vs. 65-qubit systems), illustrating real-world bottlenecks beyond gate speed.
• Circuit delay and run-time compilation/data transfer dominate CLOPS, highlighting opportunities for software stack and data movement optimizations.
• Depth-1 circuit performance metrics can be derived from CLOPS to compare system speed independent of circuit depth.