Skip to content

Conversation

krypt-n
Copy link
Contributor

@krypt-n krypt-n commented Apr 21, 2021

Issue

Closes #490

Tasks

  • Update CHANGELOG.md (remove if irrelevant)
  • review

Description

hommberger_200 VRPTW benchmark master at 1e32124

,Gaps,Computing times
Min,-14.89,194
First decile,-4.56,386
Lower quartile,-0.0,482
Median,2.03,684
Upper quartile,6.67,1690
Ninth decile,8.88,2070
Max,11.53,2491

The first commit 3a0e9d9 turns Matrixfrom a vector of vectors into a flat vector. This removes a layer of indirection and could perhaps improve memory locality.

,Gaps,Computing times
Min,-14.89,183
First decile,-4.56,377
Lower quartile,-0.0,478
Median,2.03,671
Upper quartile,6.67,1660
Ninth decile,8.88,2014
Max,11.53,2461

The second commit 3a0e9d9 just moves some code around and should not have an impact.

,Gaps,Computing times
Min,-14.89,186
First decile,-4.56,375
Lower quartile,-0.0,480
Median,2.03,672
Upper quartile,6.67,1654
Ninth decile,8.88,2025
Max,11.53,2465

The third commit 9cba73e removes another layer of indirection by allowing cost_wrapperto directly access the matrix array.

,Gaps,Computing times
Min,-14.89,181
First decile,-4.56,371
Lower quartile,-0.0,463
Median,2.03,664
Upper quartile,6.67,1623
Ninth decile,8.88,1994
Max,11.53,2407

My next idea was to get rid of the cast to double by turning the cost factor into a whole number. My first attempt was 510557e .

,Gaps,Computing times
Min,-14.89,162
First decile,-4.56,332
Lower quartile,-0.0,438
Median,2.03,646
Upper quartile,6.67,1445
Ninth decile,8.88,1872
Max,11.53,2369

However, it turned out that 128 as a factor did not lead to correct reported costs on the HVRP instances (the cost factors can not be exactly represented as multiples of 1/128). That's why I changed the factor to 100 and fixed an additional rounding problem in eb253e2 and 6b3e488 . Leading to these final results:

,Gaps,Computing times
Min,-14.89,166
First decile,-4.56,343
Lower quartile,-0.0,455
Median,2.03,615
Upper quartile,6.67,1537
Ninth decile,8.88,1945
Max,11.53,2243

Note that these computing times tend to fluctuate quite a bit on my machine, but I think the general trend towards lower computing times is visible here.
The last change effectively restricts the duration factor to two digits after the decimal point. That should probably be documented somewhere if this PR is merged.

@krypt-n
Copy link
Contributor Author

krypt-n commented Apr 22, 2021

But it seems like I need to find a way to make these benchmarks produce more reliable numbers

@jcoupey
Copy link
Collaborator

jcoupey commented Apr 22, 2021

Thanks a lot for this input.

I quite like the Matrix refactor in the sense that it does not affect anything regarding usage, except in cost wrapper that makes use of the internal layout.

giphy

Keeping the cost scaling part in the integer world is definitely something I should have done in the first place! Restricting the duration factor to two digits after the decimal point is definitely not a problem. This does not exactly apply to the speed_factor used in input as we use the inverse as durations_factor. But for the sake of simplicity I think we can we add that speed_factor "defaults to 1. with two digits after decimal point taken into account" in the docs.

Copy link
Collaborator

@jcoupey jcoupey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good with the single comment and docs update.

Regarding changelog, I don't think this should generate an entry as it's a change on a change, not wrt the previous release. But it would be useful to add #490 along with #394 in the first "Support for heterogeneous vehicle routing profiles" entry.

@jcoupey
Copy link
Collaborator

jcoupey commented Apr 22, 2021

On the topic of getting reliable numbers on benchmarks, the variation in computing times is probably more noticeable with small instances (homberger_200 vs homberger_1000). Maybe a good approach would be to sum computing times across a given number of runs on each instance, but that's another topic altogether.

For now we're interested in overall trends and I guess they'll be consistent across benchmark classes.

@jcoupey jcoupey merged commit e1fd844 into VROOM-Project:master Apr 23, 2021
@jcoupey
Copy link
Collaborator

jcoupey commented Apr 23, 2021

I just merged this PR both to master and to release/1.10 so it's now part of a new release candidate v1.10.0-rc.2. I'll run the usual benchmark stuff on my side and report.

@jcoupey
Copy link
Collaborator

jcoupey commented Apr 26, 2021

Running on the usual CVRP and VRPTW benchmarks across various exploration level shows a consistent improvement. The average computing time increase between master and #450 was ~17.6%. Now the increase between v1.9.0 and v1.10.0-rc.2 (including this PR) is down at 8.4% on average.

So the changes introduced here more than halved the impact on computing times for single-profile instances.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check cost computing overhead
3 participants