Problem

Jupyter Notebooks neither have a dedicated test framework nor a CI pipeline. Which results in,

• Almost no tests written for Notebooks
• Lack of confidence whether a Notebook would run as expected when executing on a fresh kernel
• Inability to use Notebooks for any production workflow

Users tend to jump around Notebook cells while developing, resulting in an unpredictable kernel state. Given Notebooks are prone to muddled state, importance of testing & continuous integration increases multifold.

Solution

Here are the bare minimum things we need,

• Every time a Notebook change is committed, run all cells from top to bottom and make sure no cell execution results in error
• Ability to write and execute unit tests for Notebook code
• Given a set of inputs, provide ability to specify & assert that the actual outputs of certain Notebook cells matches the expected outputs

All of the above is possible today by tinkering with separate tools such as Doctest, unittest, papermill etc. We need to combine it all, fill the gaps and make an open source testing framework dedicated to testing Jupyter Notebooks.

As a last step, this testing framework can then be integrated into ReviewNB interface to run tests after each commit is pushed for users who wish to enable CI pipeline on their repos.

Feel free to upvote/downvote the issue indicating whether you think this is useful feature or not. I also welcome additional questions/comments/discussion on the issue.