Thijs had some comments on the old PR https://github.com/duckdb/duckdb/pull/10585/files. I've been messing around in the Join Order Optimizer and realized that I old got my equalities incorrect and the regression / improvement reporting was flipped 🙈 .

This PR also removes comparing build side and probe side for plans and reporting a regression if any of these values increase. If the total plan cost changes, these values more than likely change as well, and I want to avoid being eager about reporting regressions especially if the execution time has improved. The plan now is only to report changes if the total plan cost has increased and the build side has increased. When this is not the case a regression is reported if there is a significant (3%) difference in execution time. It's possible that build side and probe side cardinalities have changed, but execution time has improved. For example, there may be more intermediate tuples in the join tree, but these tuples might all be in the probe side. This won't add much overhead since the tuples will be in flight most of the time. If the tuples were on the build side, that would add a overhead since they will need to be included in the hash table.

I ran this new version plan_cost_runner.py between v0.9.2 and the current main and there are no regressions for tpch and imdb.