When executing UI tests on Travis we do not execute the UI tests that are defined in the submodule plugins. On the contrary, we do execute unit, integration and system tests of these plugins. I think we should be consistent and also include UI tests of plugins that are added in piwik/piwik as a submodule.

When working on these plugins it makes it easier to see if the UI tests of this plugin actually work and when working on Piwik we can see if we break any of these plugins. The ones that are added are some kind of special plugins that are kinda core but we want to provide updates separately etc.

Of course we also have Travis jobs in the plugins that are submodules. The problem is that they run against beta versions of Piwik. This means to make tests work for a plugin we need to release a new beta after each change in Piwik, and then change the PIWIK_LATEST_STABLE_TEST_TARGET variable in .travis.yml of the plugin.

I am currently developing the CustomDimensions plugin and I cannot let Travis test the UI tests. To test the UI tests in CustomDimensions Travis we would need to release a new beta version of Piwik. To release a new beta that can be used by this plugin we would need to merge a PR in piwik/piwik that includes some changes in Piwik core that are needed to make this plugin work. However, we cannot really merge since we don't know whether the UI tests work etc.

This issue is to discuss whether we should include these plugins in UI tests or not. And if want to include them, we should probably remove the --core flag in https://github.com/piwik/travis-scripts/blob/ab737e8de918bfc06c1fc5234071f60e3be51a60/travis.sh#L76