This is because the ITK optimizer does not reset its learning rate between start points. Discovered by @spinicist : https://discourse.itk.org/t/unexpected-behaviour-in-multistartoptimizerv4/7466 ITK fix submitted: https://github.com/InsightSoftwareConsortium/ITK/pull/5284