Add asv benchmarks #30
Conversation
This commit adds an asv benchmark suite to the tests include in this repo. This will enable us to track the performance of the overall qiskit project over time as we make commits.
There was a problem hiding this comment.
1- I don't think we should include Qasm files here. They are quite slow to load, and quite unreadable. Instead we should have python scripts that generate those circuits (like the VQE.py and State_Tomography.py performance tests in Terra). Where do these qasms come from anyway?
2- The LiH and H2 are not used anywhere so should not be included.
|
For the qasms files sure I can remove those, but it's worth pointing out that they're not actually part of what is benchmarked they're just used to setup large circuit examples quickly without a lot of code. The only thing benchmarked is calling compile() on those circuits. For this role I think qasm works pretty well. As for the source of the files the pea_3_pi_8.qasm comes from the openqasm examples: https://github.com/Qiskit/openqasm/blob/master/examples/generic/pea_3_pi_8.qasm and the test_eoh_qasm.qasm file I actually got from you. IIRC you said it was generated by aqua and I've been using it to test performance fixes locally. Yeah, I'll remove the LiH and H2 directories. I forgot to remove them when I dropped the vqe benchmark. This patch is just a port of the work I did several months ago in terra which included a vqe benchmark based on the performance test in there. (although that doesn't work anymore) |
… add-asv-benchmarks
|
This is mostly good to go now, it feels like it's a good starting point to build off of for more benchmarks and adding more backwards compat glue code so we can compare earlier releases to what we have today. I'm still debugging one issue though before I remove the WIP tag. For a good portion of the commits when doing asv run over a range it raises either a |
The install stage was going off the rails trying failing to install the built wheel in certain scenarios. When this happened the empty namespace package from qiskit-aer was leftover and when benchmarks ran there was nothing leftover. To avoid this and simplify the install (and by extension uninstall) configuration to just rely on the default working behavior this commit removes the dependence on the qiskit-aer package. The aer package wasn't even used directly in the benchmarks since we're not actually running simulations (for obvious time reasons). Instead Aer was just being used as a backend for the transpiler. But, there is no reason we have to use Aer for that, and BasicAer works just as well.
|
This is looking good so far, it seems to avoid the install issues I was facing before. I'm running a quick benchmark on all terra commits from 0.7.0 to master to verify it works (on a quicker benchmark some of them can take a a while, like quantum volume) to verify there are no install issues. Once that passes without error I'll remove the WIP tag. |
* Add asv benchmarks This commit adds an asv benchmark suite to the tests include in this repo. This will enable us to track the performance of the overall qiskit project over time as we make commits. * Remove H2 and LiH dirs * Fix most, but not all, lint * Remove commented unused code * Add QV circuit benchmark and fix asv build and some lint * Make lint pass * Purity isn't in quantum_info in 0.7 * Add qft and random circuit benchmarks * Fix issue caused by namespace package and aer sticking around * Make depth an independent variable for qv benchmarks * Simplify install and uninstall stage The install stage was going off the rails trying failing to install the built wheel in certain scenarios. When this happened the empty namespace package from qiskit-aer was leftover and when benchmarks ran there was nothing leftover. To avoid this and simplify the install (and by extension uninstall) configuration to just rely on the default working behavior this commit removes the dependence on the qiskit-aer package. The aer package wasn't even used directly in the benchmarks since we're not actually running simulations (for obvious time reasons). Instead Aer was just being used as a backend for the transpiler. But, there is no reason we have to use Aer for that, and BasicAer works just as well. * Clear build artifacts before install * Fix lint failures for new pylint * Switch back to pip for wheel builds * Fix another lint issue * Fix hopefully the last lint issue
* Add asv benchmarks This commit adds an asv benchmark suite to the tests include in this repo. This will enable us to track the performance of the overall qiskit project over time as we make commits. * Remove H2 and LiH dirs * Fix most, but not all, lint * Remove commented unused code * Add QV circuit benchmark and fix asv build and some lint * Make lint pass * Purity isn't in quantum_info in 0.7 * Add qft and random circuit benchmarks * Fix issue caused by namespace package and aer sticking around * Make depth an independent variable for qv benchmarks * Simplify install and uninstall stage The install stage was going off the rails trying failing to install the built wheel in certain scenarios. When this happened the empty namespace package from qiskit-aer was leftover and when benchmarks ran there was nothing leftover. To avoid this and simplify the install (and by extension uninstall) configuration to just rely on the default working behavior this commit removes the dependence on the qiskit-aer package. The aer package wasn't even used directly in the benchmarks since we're not actually running simulations (for obvious time reasons). Instead Aer was just being used as a backend for the transpiler. But, there is no reason we have to use Aer for that, and BasicAer works just as well. * Clear build artifacts before install * Fix lint failures for new pylint * Switch back to pip for wheel builds * Fix another lint issue * Fix hopefully the last lint issue
Summary
This commit adds an asv benchmark suite to the tests include in this
repo. This will enable us to track the performance of the overall qiskit
project over time as we make commits.
Details and comments