Flaky CI tests with Scipy 1.11.0 on Windows Python 3.11 #10345
Description
Environment
- Qiskit Terra version: main
- Python version: Python 3.11
- Operating system: Windows
What is happening?
CI is flaky on Windows / Python 3.11 since the release of Scipy 1.11.
How can we reproduce the issue?
Example CI traceback:
test.python.providers.test_backend_v2.TestBackendV2.test_5q_ghz_22_ecr_level_3_bidirectional_False
--------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "D:\a\1\s\test-job\Lib\site-packages\ddt.py", line 220, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\test\python\providers\test_backend_v2.py", line 113, in test_5q_ghz
tqc = transpile(qc, backend, optimization_level=opt_level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\compiler\transpiler.py", line 375, in transpile
_serial_transpile_circuit(
File "D:\a\1\s\qiskit\compiler\transpiler.py", line 457, in _serial_transpile_circuit
result = pass_manager.run(circuit, callback=callback, output_name=output_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\passmanager.py", line 428, in run
return super().run(circuits, output_name, callback)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\passmanager.py", line 447, in wrapper
return meth(*meth_args, **meth_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\passmanager.py", line 154, in run
return super().run(
^^^^^^^^^^^^
File "D:\a\1\s\qiskit\passmanager\passmanager.py", line 225, in run
out_program = self._run_single_circuit(in_programs[0], callback, **metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\passmanager.py", line 173, in _run_single_circuit
out_program = pass_runner.run(input_program, callback=callback, **metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\runningpassmanager.py", line 226, in wrapper
return meth(*meth_args, **meth_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\runningpassmanager.py", line 148, in run
return super().run(
^^^^^^^^^^^^
File "D:\a\1\s\qiskit\passmanager\passrunner.py", line 235, in run
passmanager_ir = self._run_pass_generic(
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\passmanager\passrunner.py", line 190, in _run_pass_generic
passmanager_ir = self._run_pass_generic(
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\passmanager\passrunner.py", line 151, in _run_pass_generic
passmanager_ir = self._run_base_pass(
^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\runningpassmanager.py", line 177, in _run_base_pass
new_dag = pass_.run(passmanager_ir)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\transpiler\passes\optimization\consolidate_blocks.py", line 123, in run
or self.decomposer.num_basis_gates(unitary) < basis_count
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\quantum_info\synthesis\two_qubit_decompose.py", line 1414, in num_basis_gates
a, b, c = weyl_coordinates(unitary)[:]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\qiskit\quantum_info\synthesis\weyl.py", line 65, in weyl_coordinates
D = la.eigvals(Up.T @ Up)
^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\test-job\Lib\site-packages\scipy\linalg\_decomp.py", line 895, in eigvals
return eig(a, b=b, left=0, right=0, overwrite_a=overwrite_a,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\a\1\s\test-job\Lib\site-packages\scipy\linalg\_decomp.py", line 249, in eig
_check_info(info, 'eig algorithm (geev)',
File "D:\a\1\s\test-job\Lib\site-packages\scipy\linalg\_decomp.py", line 1371, in _check_info
raise LinAlgError(("%s " + positive) % (driver, info,))
numpy.linalg.LinAlgError: eig algorithm (geev) did not converge (only eigenvalues with order >= 2 have converged)
What should happen?
Reliable CI.
Any suggestions?
I suspect / hope that this is just something like Scipy's internal ARPACK build being done with a new compiler that's tweaked things a little bit. Eigensystem routines are never 100% reliable, and we call them a lot, so it's pretty likely that it just so happens that one matrix in our test suite happens to be a bit unreliable.
In the longer term, we can potentially attempt to stabilise the routines by the insertion of some very slight amounts of noise if the initial decomposition fails, but we'd first need to work out if the Scipy 1.11 is actually significantly less stable than what we've already got; it could just be an unlucky isolated failure.