Skip to content

tools: improve testrunner #7258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 25 commits into from
Closed

Conversation

kaspar030
Copy link
Contributor

This PR finally adds a way of flexibly run tests on real hardware.

Before, there was "testrunner", which could sequentially run tests on a single board, and the beginnings of a more general approach, hidden in lwip's tests.

Testrunner can now read json files (or whole trees of them) containing test and node descriptions.

Test descriptions describe things like "need 3 nodes, two of them samr21-xpro, one of them must have 6lowpan; flash gnrc_networking on the first, gnrc_border_router on the second and nanocoap_test on the third; set up the default route of the first and third to be the link local address of the second; then run these pexpect scripts; .....".

Node descriptions describe node groups of actual nodes, like "node0 is a samr21-xpro, serial number ATMLxxxx, has 6lowpan and a mma8x5x connected, node 1 is ...". The latter can be less specific if they contain a field "murdock_queue".

There are some nifty features (including files, simple templateing) in order to reduce duplication in the description files.

The testrunner can then read in a bunch of test and node descriptions, and then run tests, either every test once, or every test on each node group that can run it. It can also create command lines usable by murdocks backend DWQ.

Another commit adds initial integration into murdock.

Currently, there's only one samr21-xpro connected to my laptop, for testing.

TODO:

  • produce some examples / getting-started hints
  • document
  • show test results in murdock output
  • find safe way of hooking up nodes to CI, add (more) nodes
  • split tests into long running / short running,

DO NOT MERGE! This is WIP.

@kaspar030 kaspar030 added Area: CI Area: Continuous Integration of RIOT components CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Area: tests Area: tests and testing framework State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet labels Jun 27, 2017
@kaspar030 kaspar030 added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Jun 27, 2017
@kaspar030
Copy link
Contributor Author

I'll be rebuilding this a lot. :)

@miri64
Copy link
Member

miri64 commented Jun 27, 2017

Test descriptions describe things like "need 3 nodes, two of them samr21-xpro, one of them must have 6lowpan;

Will there also be a way to say: "need 3 nodes, two must have IEEE 802.15.4"?

@miri64
Copy link
Member

miri64 commented Jun 27, 2017

Node descriptions describe node groups of actual nodes, like "node0 is a samr21-xpro, serial number ATMLxxxx, has 6lowpan and a mma8x5x connected, node 1 is ...". The latter can be less specific if they contain a field "murdock_queue".

How does a board "have 6LoWPAN"?

@kaspar030
Copy link
Contributor Author

Will there also be a way to say: "need 3 nodes, two must have IEEE 802.15.4"?
How does a board "have 6LoWPAN"?

Sure!

This is implemented using simple tags.

@miri64
Copy link
Member

miri64 commented Jun 27, 2017

How does a board "have 6LoWPAN"?

Sure!

But don't you mean it "has an IEEE 802.15.4 radio"? I'm not sure if there are already 6LoWPAN-integrated devices ;).

@kaspar030
Copy link
Contributor Author

But don't you mean it "has an IEEE 802.15.4 radio"? I'm not sure if there are already 6LoWPAN-integrated devices ;).

I badly misquoted. :) Yeah, specifying 802.15.4 instead of sixlowpan makes a lot more sense. Luckily tag names can be freely chosen. :)

@kaspar030 kaspar030 added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Jun 27, 2017
@kaspar030 kaspar030 force-pushed the improve_testrunner branch from f5c690e to b70d771 Compare June 27, 2017 21:40
@kaspar030 kaspar030 removed the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Jun 29, 2017
@kYc0o kYc0o mentioned this pull request Oct 30, 2017
11 tasks
@smlng smlng mentioned this pull request Nov 22, 2017
@smlng smlng removed this from the Release 2018.01 milestone Jan 12, 2018
@miri64
Copy link
Member

miri64 commented Mar 27, 2018

I guess this is superseded by #8801 now? If not, just reopen.

@miri64 miri64 closed this Mar 27, 2018
@kaspar030
Copy link
Contributor Author

I guess this is superseded by #8801 now?

Nope, they're complementary. But I'll reopen when there are news.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: CI Area: Continuous Integration of RIOT components Area: tests Area: tests and testing framework CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants