-
Notifications
You must be signed in to change notification settings - Fork 440
Add duration tracking option for each test #127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The Contributor Guidelines mention quoting all variables. I'm not sure if I should do this, or remain consistent with the existing code's usage of _flags variables... |
YAML blocks are a feature of the TAP 13 specification. As a result, "TAP version 13" should be the first line a TAP producer writes to be spec-compliant. This allows TAP consumers such a python's tap.py to process YAML blocks. I'll update the PR accordingly. |
In some validation cases, it makes sense to track the duration of each test to understand which ones are executing quickly and which ones are executing slowly. Trends may be built up over time to see how duration changes given the other changes being tested. Rebasing from harschware/bats/yaml_durations. Because this adds a YAML block to the TAP output, the TAP producer needs to write "TAP version 13" as the first line. Signed-off-by: George T Kramer <george.t.kramer@intel.com>
Related with #49 right? |
@BraisGabin, yes, this is related to #49. This adds duration when outputting test results in the TAP format, but not the regular format. A YAML document was chosen because some TAP consumers, such as Jenkins with it's TAP plugin, look for duration in this structured manner to display it in a CI system. I think diagnostic information, in my experience, is more useful in the TAP format to show a backtrace when a test failed. Is there a way this might be useful when outputting in the default, non-TAP format? Not sure if there is a way that it could be done for automated tools to parse easily. Looks like I need to rebase. I'll work on doing this. |
+1, would like to use this. @gtkramer, what's the status? |
would like to use it . Can anyone merge this PR? |
Hi @gtkramer, sorry this has been open so long. Could you rebase master onto this branch to resolve the conflicts please? I wonder about optional |
Looking forward to use this function. Could someone please help to review and merge it? |
@yahliu due to the time elapsed since submission (apologies!) these changes conflict with master and need rebasing to be merged. If somebody is able to do that work we can get this merged. Feel free to discus here first, at first glance:
|
@sublimino I see latest code already have the time option Based on my checking, the TAP plugin will get keywords |
I am looking into getting this to fly with the current codebase. My observations:
My proposal would be:
|
I am closing this since #337 has been merged. |
In some validation cases, it makes sense to track the duration of each
test to understand which ones are executing quickly and which ones are
executing slowly. Trends may be built up over time to see how duration
changes given the other changes being tested.
Rebasing from harschware/bats/yaml_durations.
Because this adds a YAML block to the TAP output, the TAP producer needs
to write "TAP version 13" as the first line.
Signed-off-by: George T Kramer george.t.kramer@intel.com