Background

Dokka has good unit test coverage (> 80%) and OK-ish integration tests, but sometimes dependency updates or internal refactorings can introduce bugs or barely noticeable changes in behaviour or in the output. Most of the time, it's insignificant and not even bugs, but it would not hurt to know about it in advance.

Proposal

To make sure such subtle changes do not go unnoticed, it should be possible to compare the output of the "old" version of Dokka (before the update/refactoring) and the "new" (after). If there's no regression, the diffs should be the same. If even the smallest detail changes, the diff should show it.

This will help immensely with testing Dokka's migration to the K2 compiler (#2888) and with testing the new Gradle runner (#3131).

What exactly should be compared (the HTML outputs or the serialized documentable model), and how, is to be discussed.

Notes

There are projects that already have similar tests and that we can learn from:

• Dackka (Google's Dokka plugin for Android docs) compares the diffs of their own HTML format.
• Dokkatoo compares the HTML output that Dokka's native Gradle plugin produces with its own, but only for example projects. Ideally, it would be nice to compare big user projects used in integration tests.

Related issues: #2925