There are several improvements that would be good to make, including but not limited to:

1. Forking rather than starting up a new Sage each time.
2. Customizable comparison rather than mutating doctests into new doctests (e.g. followup to better numerical accuracy testing #10952).
3. Eliminate need to write temporary files for every doctested file.
4. Constant headaches caused by the difference in doctesting library vs. non-library files.
5. The ability to drop into a debugger if a doctest raises an exception.

Robert's code is at

http://code.google.com/p/sagemath-timer/

Apply attachment: 12415_framework.patch, attachment: 12415_doctest_fixes.patch, attachment: 12415_doc.patch, attachment: 12415_review.patch, attachment: 12415_test.patch, attachment: 12415_review_review.patch, attachment: 12415_review3.patch, attachment: 12415_manifest.patch, attachment: 12415_rebase_58.patch

Apply attachment: 12415_script.patch and attachment: 12415_script_review.patch to the scripts repo

Apply attachment: 12415_spkg_bin_sage.patch to the root repo

For follow-up or other doctest-related tickets see #11337.

Depends on #13147
Depends on #13146
Depends on #13145
Depends on #12723
Depends on #12392
Depends on #12393
Depends on #12395
Depends on #12396
Depends on #12397
Depends on #12381
Depends on #12382
Depends on #12383
Depends on #12384
Depends on #11871
Depends on #13195
Depends on #13121
Depends on #13748
Depends on #13899
Depends on #12719
Depends on #5155
Depends on #14070
Depends on #14079
Depends on #14150
Depends on #14158
Depends on #14182
Depends on #14184
Depends on #14054
Depends on #14063
Depends on #13605
Depends on #14111
Depends on #14254
Depends on #14242
Depends on #14253

CC: @kini @ohanar @jhpalmieri

Component: doctest framework

Author: David Roe, Robert Bradshaw, Jeroen Demeyer

Reviewer: Jeroen Demeyer, David Roe

Merged: sage-5.9.beta0

Issue created by migration from https://trac.sagemath.org/ticket/12415