Skip to content

Add duration tracking option for each test #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Add duration tracking option for each test #127

wants to merge 1 commit into from

Conversation

gtkramer
Copy link

@gtkramer gtkramer commented Jul 18, 2018

In some validation cases, it makes sense to track the duration of each
test to understand which ones are executing quickly and which ones are
executing slowly. Trends may be built up over time to see how duration
changes given the other changes being tested.

Rebasing from harschware/bats/yaml_durations.

Because this adds a YAML block to the TAP output, the TAP producer needs
to write "TAP version 13" as the first line.

Signed-off-by: George T Kramer george.t.kramer@intel.com

@gtkramer gtkramer requested a review from a team as a code owner July 18, 2018 21:53
@gtkramer
Copy link
Author

The Contributor Guidelines mention quoting all variables. I'm not sure if I should do this, or remain consistent with the existing code's usage of _flags variables...

@gtkramer
Copy link
Author

gtkramer commented Jul 21, 2018

YAML blocks are a feature of the TAP 13 specification. As a result, "TAP version 13" should be the first line a TAP producer writes to be spec-compliant. This allows TAP consumers such a python's tap.py to process YAML blocks. I'll update the PR accordingly.

In some validation cases, it makes sense to track the duration of each
test to understand which ones are executing quickly and which ones are
executing slowly.  Trends may be built up over time to see how duration
changes given the other changes being tested.

Rebasing from harschware/bats/yaml_durations.

Because this adds a YAML block to the TAP output, the TAP producer needs
to write "TAP version 13" as the first line.

Signed-off-by: George T Kramer <george.t.kramer@intel.com>
@BraisGabin
Copy link

Related with #49 right?

@gtkramer
Copy link
Author

gtkramer commented Aug 10, 2018

@BraisGabin, yes, this is related to #49. This adds duration when outputting test results in the TAP format, but not the regular format. A YAML document was chosen because some TAP consumers, such as Jenkins with it's TAP plugin, look for duration in this structured manner to display it in a CI system. I think diagnostic information, in my experience, is more useful in the TAP format to show a backtrace when a test failed.

Is there a way this might be useful when outputting in the default, non-TAP format? Not sure if there is a way that it could be done for automated tools to parse easily.

Looks like I need to rebase. I'll work on doing this.

@mbland mbland mentioned this pull request Aug 14, 2018
@phillipao
Copy link

+1, would like to use this. @gtkramer, what's the status?

@gzhifang
Copy link

would like to use it . Can anyone merge this PR?

@sublimino
Copy link
Member

Hi @gtkramer, sorry this has been open so long.

Could you rebase master onto this branch to resolve the conflicts please?

I wonder about optional Tap 13 syntax. Perhaps we should migrate entirely to Tap 13 instead of just for one flag's output? WDYT @mbland @jasonkarns?

@yahliu
Copy link

yahliu commented Jun 17, 2020

Looking forward to use this function. Could someone please help to review and merge it?

@sublimino
Copy link
Member

@yahliu due to the time elapsed since submission (apologies!) these changes conflict with master and need rebasing to be merged.

If somebody is able to do that work we can get this merged. Feel free to discus here first, at first glance:

  • there are now multple output formats that each require this patched in
  • tests for each format
  • I'm not sure if this will conflict with other work on traps recently

@yahliu
Copy link

yahliu commented Jul 1, 2020

@sublimino I see latest code already have the time option -T, --timing, but it didn't shown in the Jenkins TAP plugin.
Screen Shot 2020-07-01 at 3 53 58 PM

Based on my checking, the TAP plugin will get keywords duration_ms, but -T, --timing have no such it. so I think it's better to enhance -T, --timing to support it.

@martin-schulze-vireso
Copy link
Member

martin-schulze-vireso commented Jul 25, 2020

I am looking into getting this to fly with the current codebase. My observations:

  1. the current option measures in full seconds, this one uses milliseconds
  2. I see big problems with maintaining both TAP and TAP 13 output in parallel, as @sublimino already said it would probably be benificial to go full TAP 13 then. However, this would require large redesigns of the test suite which relies on the old TAP format
  3. using TAP 13 internally would require a yaml parser or some very strict formatting to apply formatters

My proposal would be:

  1. update the timing code to use millisecond precision
  2. Add a TAP13 formatter for consumption by Jenkins TAP, this could even be an external project

@martin-schulze-vireso martin-schulze-vireso mentioned this pull request Jul 26, 2020
2 tasks
@martin-schulze-vireso
Copy link
Member

I am closing this since #337 has been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants