While working on the documentation I found that sphinx-build always reread and recreated all documentation files, even if nothing changed:

updating environment: 0 added, 96 changed, 0 removed

This was due to broken dependency calculation for python .egg files and was fixed in later versions apparently. After pip install --upgrade sphinx (currently v7.2.6), incremental builds works as expected.