Two quantum circuits can return the same measurement results and still be different programs. One can be correct and the other wrong. Most benchmarks for quantum code cannot tell them apart.
QCircuitEval grades generated quantum programs on the measurement results as well as the logic implemented, across 70 tasks and four frameworks. We built it to develop Coda and we are releasing the benchmark under the MIT license.
Coding benchmarks
Coding benchmarks give models and agents a set of programming problems and check whether the code they generate works. They have been critical in the development and scaling of large language models. HumanEval, released by OpenAI in 2021, set the template. Instead of comparing generated code to a single human-written example, HumanEval ran it against unit tests. A unit test is a small program that checks a function’s output against expected values. This shifted the focus from whether the code looked right to whether it behaved right. HumanEval also introduced the pass@k metric, which estimates the probability that at least one of k sampled solutions is correct, necessary due to the non-determinism of large language models.
No benchmark stays useful forever. As models improve, tasks can become too easy, the benchmarks get saturated, meaning that we can no longer distinguish model performance. Public benchmark problems can appear in training data known as “contamination”, allowing models to recognize or memorize them rather than solve them from scratch. To keep measuring progress, benchmarks need to become harder and their graders, the software within the benchmark to check the generated code, need to become more rigorous.
Grading quantum code
Quantum coding benchmarks start from the same idea: give a model a programming task and check whether it is correct. The difference is that determining whether a quantum program works can be more complicated than it is for classical code.
A classical function takes an input and returns a deterministic output. Quantum programs are often much more low-level, their characteristics resemble classical machine code more so than high-level programming languages like Python. A quantum program might prepare a state, produce a probability distribution, or perform measurements whose results vary from run to run.
Like classical programs, quantum programs can be written in many different ways to produce the same output. Early quantum coding benchmarks have been developed to tackle the problem of testing for correct quantum computation, while still relying on comparison against one reference (or canonical) solution. These benchmarks may permit false negatives and false positives. A false negative occurs when the grader rejects a program that results in an equivalent outcome to the reference, but takes a non-canonical form. A false positive is when the grader accepts a program where the result matches the expected outcome, but the program does not functionally produce that outcome. This means that the model took a shortcut or cheated. The Qiskit HumanEval benchmark tested generated Qiskit code, by relying on task-specific unit tests. QuanBench expanded this approach with 44 tasks and compared generated circuits with a canonical solution. QuanBench+ adapted 42 of those tasks across Qiskit, Cirq, and PennyLane, added feedback-based repair of circuits, and comparison of measurement output distributions for quantitative grading.
QCircuitEval
Our benchmark, QCircuitEval, complements prior work and uses a newly developed grader. It verifies the mathematical behavior and required construction of generated quantum programs across Qiskit, Cirq, PennyLane, and NVIDIA CUDA-Q. QCircuitEval is not tied to the canonical implementation. The grader is made up of two sub-graders: a functional grader and a structural grader. The functional grader simulates the circuit and compares what it does against a target derived from the task itself rather than a stored solution. The structural grader inspects the low-level code for example, what registers, gates, and measurements were used - evidence that the program was actually built. By combining the two, QCircuitEval checks if the program behaves as intended, while stopping shortcuts or cheating that would have been let through by other benchmarks, by inspecting the structure of the code. In other words QCircuitEval, says, “I know you got the answer right but, show me the working.”
The top performing model on QCircuitEval is GPT-5.6-sol at high reasoning effort with a score of 92.7% at pass@1. We see similar levels of performance from Grok 4.6, Gemini 3.1 Pro. Seven of the 33 various models and configurations sit above 90%. In multiple cases, changing reasoning effort does not make much difference in the overall frontier models scores. The benchmark is saturated by the latest models but still may prove beneficial for training and grading smaller models such as Gemma4. We have watched quantum coding benchmarks, including QCircuitEval, become saturated over the past few months. At the same time, we found safety filters placed on models cause them to refuse tasks they could otherwise complete.
The future
We are building a harder quantum coding benchmark for the next generation of frontier models. We are designing it with researchers across academia and industry, and we are looking for more collaborators.
Quantum computers will not produce new science on their own. Someone or something has to write the programs that run on them, and more and more often it is a large language model. If those programs are wrong in ways neither we nor the model can see, the results are wrong too, and no one finds out until much later. Frontier models need benchmarks that test whether the quantum code they write is correct. That work will matter for the next wave of discovery built on these machines.
If you’d like more detail on the grader behind QCircuitEval, check out:
QCircuitEval Results and Explainer Page: https://www.conductorquantum.com/qcircuiteval
QCircuitEval GitHub: https://github.com/conductorquantum/QCircuitEval
Conductor Quantum is an American company headquartered in San Francisco, California. We are assembling a team of hardcore engineers whose sole focus is to develop quantum superintelligence. If you would like to solve one of the hardest technological challenges of our time, this is your chance. Join us.v