doc: Migrate from doc.txt to doc.md in Markdown Format #21220
Description
Migration Guide from doc.txt to doc.md
Why migrate in the first place?
Currently RIOT has (module) documentation that is either too extensive to be put into headers or should be accessible more generally in doc.txt files throughout the file structure. Doxygen parses these files as C-files, which means that each doc.txt file starts and ends with a C-style comment block. In this comment block, the typical Doxygen and Markdown commands can be and are used.
This has two distinctive disadvantages:
- GitHub does not parse these files. Therefore, you just have the "unparsed" plain text file if you click on a
doc.txtfile (no headlines, no tables, no images, ...). - Many of the
doc.txtfiles use the wrong header format. Usually there should be a leading asterisk on every line. If it is missing, some Markdown commands such as**bold text**can be misinterpreted if they happen to be at the start of the line.
How to do the migration?
Reference: How it should look like
This is an example for a finished doc.md file.
@defgroup boards_random-board
@ingroup boards
@brief This is a random board definition
# Headline
## Subheadline
Test
Case 1: Incorrect Header Style
Even though in this case the header style is wrong, this is the lucky case, because only a limited number of lines has to be changed.
Files for this case usually look like this:
/**
@defgroup boards_random-board
@ingroup boards
@brief This is a random board definition
# Headline
## Subheadline
Test
*/
In this case, only the first and the last line have to be removed.
Case 2: Correct Header Style
In case the correct header style was used, the migration takes a bit more effort because all the asterisks have to be removed.
HOWEVER you can use your editor's find-and-replace feature and remove * (whitespace-asterisk-whitespace) and the remove * (whitespace-asterisk). Doing it in this order will make sure that you don't have a bunch of white spaces left over at the start of each line.
Perhaps some editors even allow you to toggle the comment block.
/**
* @defgroup boards_random-board
* @ingroup boards
* @brief This is a random board definition
*
* # Headline
* ## Subheadline
* Test
*
*/
How to commit the changes?
If you are doing any changes apart from the migration, it is good practise to put the migration changes in a separate comment.
This will make it a lot easier for the maintainers to distinguish between the changes you made and the migration. Otherwise GitHub will sometimes only show "old file removed, new file added".
Furthermore it is important to use the git mv command to rename the file from doc.txt to doc.md.
History behind this
As discovered in #21218, some doc.txt files use an incorrectly formatted doc.txt file. In these files, the Doxygen documentation is essentially within a C comment. However many files omit the leading asterisk which can cause Doxygen to incorrectly parse Markdown statements for bold and italic at a line start.
Incorrect:
/**
* @defgroup ...
Markdown here
*Bold ignored* because Doxygen removes leading stars.
*/Correct:
/**
* @defgroup ...
*
* Markdown here
*
* *Bold not ignored* because Doxygen has a leading star to remove.
*/This is outlined in the Doxygen documentation as well:
https://www.doxygen.nl/manual/markdown.html#mddox_emph_spans
https://www.doxygen.nl/manual/markdown.html#mddox_stars
With this shell command find . -name "doc.txt" -exec grep -lE '^[^ /]' {} + I compiled a list of potential candidates. It searches for files that have lines which don't begin with a space or a slash. This can generate false positives, because some files have #ifdefs, but it gives something to start with:
List of potential candidates
RIOT$ find . -name "doc.txt" -exec grep -lE '^[^ /]' {} +
./cpu/msp430/doc.txt
./cpu/stm32/doc.txt
./cpu/esp32/esp-eth/doc.txt
./cpu/esp32/doc.txt
./cpu/cc26xx_cc13xx/doc.txt
./cpu/esp8266/doc.txt
./cpu/qn908x/doc.txt
./cpu/doc.txt
./cpu/rpx0xx/doc.txt
./cpu/kinetis/doc.txt
./cpu/esp_common/esp-wifi/doc.txt
./cpu/esp_common/esp-now/doc.txt
./bootloaders/riotboot_serial/doc.txt
./bootloaders/riotboot_tinyusb_dfu/doc.txt
./bootloaders/riotboot/doc.txt
./bootloaders/riotboot_dfu/doc.txt
./doc.txt
./pkg/uwb-core/doc.txt
./pkg/mynewt-core/doc.txt
./pkg/openwsn/doc.txt
./pkg/lv_drivers/doc.txt
./pkg/lvgl/doc.txt
./pkg/fatfs/doc.txt
./pkg/tinydtls/doc.txt
./boards/remote-reva/doc.txt
./boards/cc1352p-launchpad/doc.txt
./boards/esp32-heltec-lora32-v2/doc.txt
./boards/nucleo-g071rb/doc.txt
./boards/b-u585i-iot02a/doc.txt
./boards/arduino-mkrzero/doc.txt
./boards/feather-m0/doc.txt
./boards/stm32f4discovery/doc.txt
./boards/stm32f3discovery/doc.txt
./boards/seeeduino_arch-pro/doc.txt
./boards/nucleo-f070rb/doc.txt
./boards/spark-core/doc.txt
./boards/atxmega-a1-xplained/doc.txt
./boards/slstk3402a/doc.txt
./boards/gba_cartridge/doc.txt
./boards/avsextrem/doc.txt
./boards/qn9080dk/doc.txt
./boards/nucleo-f303ze/doc.txt
./boards/stm32l0538-disco/doc.txt
./boards/teensy31/doc.txt
./boards/sodaq-autonomo/doc.txt
./boards/iotlab-a8-m3/doc.txt
./boards/nucleo-f767zi/doc.txt
./boards/sipeed-longan-nano/doc.txt
./boards/arduino-zero/doc.txt
./boards/nucleo-l152re/doc.txt
./boards/feather-m0-wifi/doc.txt
./boards/nucleo-f746zg/doc.txt
./boards/arduino-nano-33-ble/doc.txt
./boards/esp32-ethernet-kit-v1_2/doc.txt
./boards/yunjia-nrf51822/doc.txt
./boards/adafruit-pybadge/doc.txt
./boards/stm32mp157c-dk2/doc.txt
./boards/stk3200/doc.txt
./boards/arduino-mkr1000/doc.txt
./boards/nrf51dongle/doc.txt
./boards/nucleo-l053r8/doc.txt
./boards/calliope-mini/doc.txt
./boards/yarm/doc.txt
./boards/microbit/doc.txt
./boards/wemos-zero/doc.txt
./boards/nucleo-l476rg/doc.txt
./boards/same54-xpro/doc.txt
./boards/p-l496g-cell02/doc.txt
./boards/esp8266-sparkfun-thing/doc.txt
./boards/feather-m0-lora/doc.txt
./boards/sodaq-explorer/doc.txt
./boards/atxmega-a3bu-xplained/doc.txt
./boards/arduino-duemilanove/doc.txt
./boards/microduino-corerf/doc.txt
./boards/adafruit-clue/doc.txt
./boards/adafruit-itsybitsy-m4/doc.txt
./boards/nrf52832-mdk/doc.txt
./boards/openmote-b/doc.txt
./boards/xg23-pk6068a/doc.txt
./boards/adafruit-metro-m4-express/doc.txt
./boards/arduino-mkrfox1200/doc.txt
./boards/arduino-nano-33-iot/doc.txt
./boards/nrf52dk/doc.txt
./boards/esp8266-esp-12x/doc.txt
./boards/sodaq-sara-sff/doc.txt
./boards/arduino-leonardo/doc.txt
./boards/nrf5340dk-app/doc.txt
./boards/nucleo-wl55jc/doc.txt
./boards/nucleo-f207zg/doc.txt
./boards/nucleo-l031k6/doc.txt
./boards/weact-f401ce/doc.txt
./boards/p-nucleo-wb55/doc.txt
./boards/nrf52840dk/doc.txt
./boards/native/doc.txt
./boards/esp32s3-wt32-sc01-plus/doc.txt
./boards/adafruit-itsybitsy-nrf52/doc.txt
./boards/bluepill-stm32f103cb/doc.txt
./boards/phynode-kw41z/doc.txt
./boards/microbit-v2/doc.txt
./boards/native64/doc.txt
./boards/omote/doc.txt
./boards/slstk3400a/doc.txt
./boards/ruuvitag/doc.txt
./boards/mcb2388/doc.txt
./boards/derfmega256/doc.txt
./boards/nucleo-f401re/doc.txt
./boards/nrf9160dk/doc.txt
./boards/cc1312-launchpad/doc.txt
./boards/nucleo-f429zi/doc.txt
./boards/acd52832/doc.txt
./boards/esp32s3-usb-otg/doc.txt
./boards/stm32f746g-disco/doc.txt
./boards/samd21-xpro/doc.txt
./boards/nucleo-f446ze/doc.txt
./boards/sltb001a/doc.txt
./boards/alientek-pandora/doc.txt
./boards/frdm-k64f/doc.txt
./boards/arduino-due/doc.txt
./boards/rpi-pico/doc.txt
./boards/rpi-pico-w/doc.txt
./boards/adafruit-grand-central-m4-express/doc.txt
./boards/stk3600/doc.txt
./boards/blackpill-stm32f103c8/doc.txt
./boards/nucleo-g070rb/doc.txt
./boards/e180-zg120b-tb/doc.txt
./boards/stm32l496g-disco/doc.txt
./boards/nucleo-l4r5zi/doc.txt
./boards/esp32s3-devkit/doc.txt
./boards/pba-d-01-kw2x/doc.txt
./boards/remote-pa/doc.txt
./boards/samr21-xpro/doc.txt
./boards/msbiot/doc.txt
./boards/z1/doc.txt
./boards/samr34-xpro/doc.txt
./boards/esp32s3-pros3/doc.txt
./boards/nucleo-f410rb/doc.txt
./boards/slwstk6220a/doc.txt
./boards/nrf52840-mdk-dongle/doc.txt
./boards/bluepill-stm32f030c8/doc.txt
./boards/esp32s2-wemos-mini/doc.txt
./boards/maple-mini/doc.txt
./boards/nucleo-f446re/doc.txt
./boards/feather-nrf52840-sense/doc.txt
./boards/i-nucleo-lrwan1/doc.txt
./boards/nucleo-g474re/doc.txt
./boards/atmega328p-xplained-mini/doc.txt
./boards/pinetime/doc.txt
./boards/weact-f411ce/doc.txt
./boards/stm32f723e-disco/doc.txt
./boards/nucleo-l552ze-q/doc.txt
./boards/lobaro-lorabox/doc.txt
./boards/opencm904/doc.txt
./boards/nucleo-f412zg/doc.txt
./boards/cc1352-launchpad/doc.txt
./boards/stm32f030f4-demo/doc.txt
./boards/nucleo-f042k6/doc.txt
./boards/olimexino-stm32/doc.txt
./boards/nrf52840dongle/doc.txt
./boards/generic-cc2538-cc2592-dk/doc.txt
./boards/samd10-xmini/doc.txt
./boards/nrf52840-mdk/doc.txt
./boards/esp32-wrover-kit/doc.txt
./boards/firefly/doc.txt
./boards/sltb009a/doc.txt
./boards/particle-boron/doc.txt
./boards/msb-430/doc.txt
./boards/nucleo-f103rb/doc.txt
./boards/stm32l476g-disco/doc.txt
./boards/usb-kw41z/doc.txt
./boards/sipeed-longan-nano-tft/doc.txt
./boards/pyboard/doc.txt
./boards/udoo/doc.txt
./boards/nucleo-f072rb/doc.txt
./boards/doc.txt
./boards/esp32c3-devkit/doc.txt
./boards/iotlab-m3/doc.txt
./boards/nucleo-u575zi-q/doc.txt
./boards/esp32-wemos-lolin-d32-pro/doc.txt
./boards/saml21-xpro/doc.txt
./boards/derfmega128/doc.txt
./boards/saml11-xpro/doc.txt
./boards/b-l072z-lrwan1/doc.txt
./boards/atmega1284p/doc.txt
./boards/nucleo-l452re/doc.txt
./boards/slstk3701a/doc.txt
./boards/particle-xenon/doc.txt
./boards/esp8266-olimex-mod/doc.txt
./boards/mbed_lpc1768/doc.txt
./boards/nucleo-g431rb/doc.txt
./boards/openlabs-kw41z-mini/doc.txt
./boards/frdm-k22f/doc.txt
./boards/stm32f769i-disco/doc.txt
./boards/nrf51dk/doc.txt
./boards/esp32-wroom-32/doc.txt
./boards/esp32s2-lilygo-ttgo-t8/doc.txt
./boards/saml10-xpro/doc.txt
./boards/ek-lm4f120xl/doc.txt
./boards/atmega8/doc.txt
./boards/nucleo-f303re/doc.txt
./boards/nucleo-l496zg/doc.txt
./boards/arduino-nano/doc.txt
./boards/remote-revb/doc.txt
./boards/nucleo-l412kb/doc.txt
./boards/blackpill-stm32f103cb/doc.txt
./boards/dwm1001/doc.txt
./boards/nucleo-f091rc/doc.txt
./boards/arduino-mkrwan1300/doc.txt
./boards/serpente/doc.txt
./boards/lsn50/doc.txt
./boards/stm32f429i-disco/doc.txt
./boards/openmote-cc2538/doc.txt
./boards/esp32-ttgo-t-beam/doc.txt
./boards/telosb/doc.txt
./boards/limifrog-v1/doc.txt
./boards/msba2/doc.txt
./boards/zigduino/doc.txt
./boards/esp32-olimex-evb/doc.txt
./boards/same51-curiosity-nano/doc.txt
./boards/stm32f7508-dk/doc.txt
./boards/samd20-xpro/doc.txt
./boards/b-l475e-iot01a/doc.txt
./boards/nucleo-f334r8/doc.txt
./boards/nucleo-f302r8/doc.txt
./boards/esp32-ethernet-kit-v1_0/doc.txt
./boards/atmega256rfr2-xpro/doc.txt
./boards/nucleo-f031k6/doc.txt
./boards/nucleo-f030r8/doc.txt
./boards/nucleo-l432kc/doc.txt
./boards/arduino-uno/doc.txt
./boards/im880b/doc.txt
./boards/cc2650-launchpad/doc.txt
./boards/nucleo-f722ze/doc.txt
./boards/common/slwstk6000b/doc.txt
./boards/common/blxxxpill/doc.txt
./boards/common/atxmega/doc.txt
./boards/common/weact-f4x1cx/doc.txt
./boards/common/doc.txt
./boards/common/nrf52/doc.txt
./boards/common/particle-mesh/doc.txt
./boards/stm32f469i-disco/doc.txt
./boards/mega-xplained/doc.txt
./boards/nucleo-f439zi/doc.txt
./boards/nucleo-l433rc/doc.txt
./boards/gd32vf103c-start/doc.txt
./boards/atxmega-a1u-xpro/doc.txt
./boards/arduino-nano-33-ble-sense/doc.txt
./boards/esp32-ethernet-kit-v1_1/doc.txt
./boards/nucleo-f303k8/doc.txt
./boards/bluepill-stm32f103c8/doc.txt
./boards/esp32-mh-et-live-minikit/doc.txt
./boards/hip-badge/doc.txt
./boards/mulle/doc.txt
./boards/openlabs-kw41z-mini-256kib/doc.txt
./boards/nucleo-l073rz/doc.txt
./boards/f4vi1/doc.txt
./boards/ikea-tradfri/doc.txt
./boards/reel/doc.txt
./boards/msb-430h/doc.txt
./boards/feather-nrf52840/doc.txt
./boards/waveshare-nrf52840-eval-kit/doc.txt
./boards/atmega328p/doc.txt
./boards/cc1350-launchpad/doc.txt
./boards/seeeduino_xiao/doc.txt
./boards/esp32c3-wemos-mini/doc.txt
./boards/airfy-beacon/doc.txt
./boards/cc2538dk/doc.txt
./boards/olimex-msp430-h1611/doc.txt
./boards/hamilton/doc.txt
./boards/particle-argon/doc.txt
./boards/avr-rss2/doc.txt
./boards/nucleo-f411re/doc.txt
./boards/nucleo-f413zh/doc.txt
./boards/slstk3401a/doc.txt
./boards/weact-f401cc/doc.txt
./boards/arduino-mega2560/doc.txt
./boards/stm32f0discovery/doc.txt
./boards/seeedstudio-gd32/doc.txt
./boards/weact-g030f6/doc.txt
./boards/e104-bt5010a-tb/doc.txt
./boards/nz32-sc151/doc.txt
./boards/sensebox_samd21/doc.txt
./boards/cc2650stk/doc.txt
./boards/nucleo-c031c6/doc.txt
./boards/thingy52/doc.txt
./boards/olimex-msp430-h2618/doc.txt
./boards/e104-bt5011a-tb/doc.txt
./boards/stk3700/doc.txt
./boards/nucleo-l011k4/doc.txt
./boards/esp32s3-box/doc.txt
./boards/esp32s2-devkit/doc.txt
./sys/malloc_thread_safe/doc.txt
./sys/net/sock/doc.txt
./sys/net/ble/doc.txt
./sys/net/ble/bluetil/doc.txt
./sys/cxx_ctor_guards/doc.txt
./sys/cpp_new_delete/doc.txt
./sys/shell/doc.txt
./drivers/ccs811/doc.txt
./drivers/sht3x/doc.txt
./drivers/atwinc15x0/doc.txt
./drivers/lsm303agr/doc.txt
This is a total of 301 files out of 481 total doc.txt files.
ChatGPT crafted a sed command that will put a * in front of every line that is not empty and a * in every empty line: sed -E '/^\/\*\*|^\* |^\*\//!{/^$/!s/^(.*)$/ * \1/}; /^\s*$/s/^.*$/ */' -i doc.txt.
This does not always work perfectly and for example the last line of the closing comment will get an asterisk as well. This is not perfect, but it does 99% of the work.
But still, this is obviously a herculean task and would create a lot of "noise" changing everything, so the best approach here is open for discussion.
IMO it would be good to address this in the documentation and to check PRs before merging to see if the doc.txt has the right format or not.
Where would be the best place to do that?
Update: It was decided that the best way going forward is to change the format from text files interpreted as C-code to plain Markdown files. The change has been documented in porting-boards.md and riotgen has been updated accordingly.