Add benchmarks for circuit scheduling #1421
Conversation
|
Here is an example of the results of new benchmarks in my local env. |
|
|
@jakelishman I've updated according to your suggestions. Would you mind taking a second look? |
There was a problem hiding this comment.
Thanks for the update, and sorry for the delay in second review.
This looks fine from my perspective, but I don't know enough about the targets for the benchmark suite to give final approval. I'll ask @kdk (who's currently away) or @mtreinish to check, and particularly check if the suggestion I made in Qiskit/qiskit#1421 (comment) was appropriate - I should have made it clearer that that was a question rather than a concrete suggestion!
There was a problem hiding this comment.
This LGTM, I think having a full path scheduling benchmark makes sense just to see what the overhead is in an end-to-end benchmark. The synthetic per pass benchmarks only take you so far. The 14x14 qv is a good choice for that because it's relatively quick. But @jakelishman is correct that we do have a time budget to worry about too. I haven't checked what our current time per commit is on the nightly runs I think we'd be ok adding a couple of optimization levels to the suite. If we added like optimization level 0 and 1 (as I think those are the likely place where people are doing scheduling) that would only add maybe 30sec-1min per commit (as each benchmark does at least 4 runs sometimes more). But I think we can do this in a follow up PR as it's easy enough to add more benchmarks
| self.cmap = [[0, 1], [1, 0], [1, 2], [1, 6], [2, 1], [2, 3], [3, 2], | ||
| [3, 4], [3, 8], [4, 3], [5, 6], [5, 10], [6, 1], [6, 5], | ||
| [6, 7], [7, 6], [7, 8], [7, 12], [8, 3], [8, 7], [8, 9], | ||
| [9, 8], [9, 14], [10, 5], [10, 11], [11, 10], [11, 12], | ||
| [11, 16], [12, 7], [12, 11], [12, 13], [13, 12], [13, 14], | ||
| [13, 18], [14, 9], [14, 13], [15, 16], [16, 11], [16, 15], | ||
| [16, 17], [17, 16], [17, 18], [18, 13], [18, 17], | ||
| [18, 19], [19, 18]] |
There was a problem hiding this comment.
I'm curious which coupling map is this? Too many edges for me to visualize it in my head.
There was a problem hiding this comment.
To be honest, I just copied it from PassBenchmarks(mapping_passes) to make the comparison with other benchmarks some sense, and I don't know where it comes from.
There was a problem hiding this comment.
Heh, well I wrote those benchmarks but a long time ago. I think I would have used either the Tokyo or Singapore coupling maps there but it's just a vague memory.
Add a benchmark to track the overall time performance of circuit scheduling, which would be near to that end users actually experience when scheduling. Follow-up PR of #1421. As suggested in the original PR, optimization level 0 and 1 are added so that it would not take so much time while covering the places where people are doing scheduling. * Add overall transpilation time bench with scheduling * Improve a bit
Add benchmarks to track the time performance of circuit scheduling Add four tests, which benchmark TimeUnitConversion, ALAPSchedule, ASAPSchedule and DynamicalDecoupling passes. It would be a good time to start benchmarking those as rework on circuit scheduling passes is about to start. * Add benchmarks for scheduling passes * lint * Add one more bench for circuit scheduling * Add overall transpilation time bench with scheduling * Revert "Add overall transpilation time bench with scheduling" This reverts commit fdba349999548ecd1fb4cc9a1e1a67802ecd9008. * Remove wild-card import
…etapackage#1441) Add a benchmark to track the overall time performance of circuit scheduling, which would be near to that end users actually experience when scheduling. Follow-up PR of Qiskit/qiskit-metapackage#1421. As suggested in the original PR, optimization level 0 and 1 are added so that it would not take so much time while covering the places where people are doing scheduling. * Add overall transpilation time bench with scheduling * Improve a bit
Add benchmarks to track the time performance of circuit scheduling Add four tests, which benchmark TimeUnitConversion, ALAPSchedule, ASAPSchedule and DynamicalDecoupling passes. It would be a good time to start benchmarking those as rework on circuit scheduling passes is about to start. * Add benchmarks for scheduling passes * lint * Add one more bench for circuit scheduling * Add overall transpilation time bench with scheduling * Revert "Add overall transpilation time bench with scheduling" This reverts commit fdba349999548ecd1fb4cc9a1e1a67802ecd9008. * Remove wild-card import
…etapackage#1441) Add a benchmark to track the overall time performance of circuit scheduling, which would be near to that end users actually experience when scheduling. Follow-up PR of Qiskit/qiskit-metapackage#1421. As suggested in the original PR, optimization level 0 and 1 are added so that it would not take so much time while covering the places where people are doing scheduling. * Add overall transpilation time bench with scheduling * Improve a bit
Summary
Add benchmarks to track the time performance of circuit scheduling
Details and comments
Add four tests, which benchmark TimeUnitConversion, ALAPSchedule, ASAPSchedule and DynamicalDecoupling passes. It would be a good time to start benchmarking those as rework on circuit scheduling passes is about to start.