Skip to content

Conversation

miri64
Copy link
Member

@miri64 miri64 commented Feb 9, 2021

Contribution description

Currently, when the prompt is read in pyterm the space after it is ignored for the prompt and the output command just adds its own prompt. This leads to the next output always having a leading space, see e.g. this output from tests/shell using RIOT_TERMINAL=pyterm:

make: Entering directory '/home/mlenders/Repositories/RIOT-OS/RIOT2/tests/shell'
/home/mlenders/Repositories/RIOT-OS/RIOT2/dist/tools/pyterm/pyterm -p "/dev/ttyUSB1" -b "500000"
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2021-02-09 14:47:15,071 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
bufsize
2021-02-09 14:47:19,712 # bufsize
2021-02-09 14:47:19,712 # 128
> bufsize
2021-02-09 14:47:21,535 #  bufsize
2021-02-09 14:47:21,536 # 128
>

While this isn't necessarily a problem in most cases, it becomes a problem when the prompt is expected and the output of a command is empty. In that case, the space is added to the empty output, making it " ", so the prompt output command is never triggered and the prompt is added to the next command in the log output. To demonstrate I added a command empty to tests/shell that just does nothing and deactivated the command echoing using CFLAGS=-DCONFIG_SHELL_NO_ECHO=1:

empty
> empty
empty
bufsize
2021-02-09 14:54:33,753 #  > > 128
>

This fixes that problem by also reading the assumed space (we already assume the prompt, so I don't see no harm in that) and if it is not a space to skip the reading of the next char in the next iteration of the reader loop.

Testing procedure

Using the provided empty command in tests/shell now yields the expected output:

$ BOARD=iotlab-m3 RIOT_TERMINAL=pyterm CFLAGS=-DCONFIG_SHELL_NO_ECHO=1 RIOT_CI_BUILD=1 make -j --no-print-directory flash term
Building application "tests_shell" for "iotlab-m3" with MCU "stm32".

   text	   data	    bss	    dec	    hex	filename
  12388	    132	   2368	  14888	   3a28	/home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/iotlab-m3/tests_shell.elf
/home/mlenders/Repositories/RIOT-OS/RIOT/dist/tools/openocd/openocd.sh flash /home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/iotlab-m3/tests_shell.elf
### Flashing Target ###
Open On-Chip Debugger 0.10.0+dev-01089-g3bfe49266 (2020-02-26-14:18)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
jtag
Warn : Interface already configured, ignoring
trst_and_srst separate srst_nogate trst_push_pull srst_open_drain connect_assert_srst

Info : clock speed 1000 kHz
Info : JTAG tap: stm32f1x.cpu tap/device found: 0x3ba00477 (mfg: 0x23b (ARM Ltd.), part: 0xba00, ver: 0x3)
Info : JTAG tap: stm32f1x.bs tap/device found: 0x06414041 (mfg: 0x020 (STMicroelectronics), part: 0x6414, ver: 0x0)
Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints
Info : stm32f1x.cpu: external reset detected
Info : Listening on port 34845 for gdb connections
    TargetName         Type       Endian TapName            State       
--  ------------------ ---------- ------ ------------------ ------------
 0* stm32f1x.cpu       cortex_m   little stm32f1x.cpu       reset

Info : JTAG tap: stm32f1x.cpu tap/device found: 0x3ba00477 (mfg: 0x23b (ARM Ltd.), part: 0xba00, ver: 0x3)
Info : JTAG tap: stm32f1x.bs tap/device found: 0x06414041 (mfg: 0x020 (STMicroelectronics), part: 0x6414, ver: 0x0)
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x080007d0 msp: 0x20000200
Info : device id = 0x10016414
Info : flash size = 512kbytes
auto erase enabled
wrote 14336 bytes from file /home/mlenders/Repositories/RIOT-OS/RIOT/tests/shell/bin/iotlab-m3/tests_shell.elf in 0.641664s (21.818 KiB/s)

verified 12520 bytes in 0.210252s (58.152 KiB/s)

Info : JTAG tap: stm32f1x.cpu tap/device found: 0x3ba00477 (mfg: 0x23b (ARM Ltd.), part: 0xba00, ver: 0x3)
Info : JTAG tap: stm32f1x.bs tap/device found: 0x06414041 (mfg: 0x020 (STMicroelectronics), part: 0x6414, ver: 0x0)
shutdown command invoked
Done flashing
/home/mlenders/Repositories/RIOT-OS/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyUSB1" -b "500000"  
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2021-02-09 15:08:12,883 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
empty
> empty
> empty
> bufsize
2021-02-09 15:08:25,954 # 128

Naturally, tests/shell should still pass, when run as test

make -j -C tests/shell flash test

Issues/PRs references

Found in #15951, where a command with empty output lead to problems in the output parsing.

@miri64 miri64 added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Area: tools Area: Supplementary tools labels Feb 9, 2021
@miri64 miri64 requested review from maribu and MrKevinWeiss February 9, 2021 14:11
@fjmolinas
Copy link
Contributor

@miri64 do you think the first commit b663ab0 could be split?

@fjmolinas
Copy link
Contributor

While this isn't necessarily a problem in most cases,

I've definitely found this annoying, had no idea where it was coming from, just thought of it as a pyterm quirk hahaha :)

Copy link
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I would only split up some changes.

@@ -700,6 +710,16 @@ class SerCmd(cmd.Cmd):
elif c == self.serprompt and output == "":
sys.stdout.write('%c ' % self.serprompt)
sys.stdout.flush()
# we now need to read away the space we printed
Copy link
Contributor

@fjmolinas fjmolinas Feb 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment above on what is ignored for this whole section?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the > + ' '

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The > is not ignored. It is checked in the c == self.serprompt.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the comment in the latest force push. Is that what you had in mind?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand the wording now TBH

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit rambling now, but at least it should be easy to understand.

@miri64 miri64 force-pushed the pyterm/fix/space-after-prompt branch from b72f8c0 to b11bc71 Compare February 9, 2021 15:54
Copy link
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK, please squash

@fjmolinas fjmolinas added CI: run tests If set, CI server will run tests on hardware for the labeled PR CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Feb 9, 2021
@fjmolinas
Copy link
Contributor

I set up tests to be ran as well :)

Currently, when the prompt is read in `pyterm` the space after it is
ignored for the prompt and the output command just adds its own prompt.
This leads to the next output always having a leading space, see e.g.
this output from `tests/shell` using `RIOT_TERMINAL=pyterm`:

```
make: Entering directory '/home/mlenders/Repositories/RIOT-OS/RIOT2/tests/shell'
/home/mlenders/Repositories/RIOT-OS/RIOT2/dist/tools/pyterm/pyterm -p "/dev/ttyUSB1" -b "500000"
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2021-02-09 14:47:15,071 # Connect to serial port /dev/ttyUSB1
Welcome to pyterm!
Type '/exit' to exit.
bufsize
2021-02-09 14:47:19,712 # bufsize
2021-02-09 14:47:19,712 # 128
> bufsize
2021-02-09 14:47:21,535 #  bufsize
2021-02-09 14:47:21,536 # 128
>
```

While this isn't necessarily a problem in most cases, it becomes a
problem when the prompt is expected and the output of a command is
empty. In that case, the space is added to the empty output, making it
" ", so the prompt output command is never triggered and the prompt is
added to the next command in the log output. To demonstrate I added a
command `empty` to `tests/shell` that just does nothing and deactivated
the command echoing using `CFLAGS=-DCONFIG_SHELL_NO_ECHO=1`:

```
empty
> empty
empty
bufsize
2021-02-09 14:54:33,753 #  > > 128
>
```

This fixes that problem by also reading the assumed space (we already
assume the prompt, so I don't see no harm in that) and if it is not a
space to skip the reading of the next char in the next iteration of the
reader loop.
@miri64 miri64 force-pushed the pyterm/fix/space-after-prompt branch from 533401c to bd96e4b Compare February 9, 2021 16:17
@miri64
Copy link
Member Author

miri64 commented Feb 9, 2021

Squashed.

@miri64 miri64 merged commit 7bb7063 into RIOT-OS:master Feb 9, 2021
@miri64 miri64 deleted the pyterm/fix/space-after-prompt branch February 9, 2021 19:34
@kaspar030 kaspar030 added this to the Release 2021.04 milestone Apr 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: tools Area: Supplementary tools CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants