[RDY] Enable functional tests in Appveyor #5225

equalsraf · 2016-08-15T11:32:52Z

Most of the functional tests need to be adjusted to run in windows. But we can already enable some tests in Appveyor. ~~For now I've only used eval/printf_spec.lua.~~

~~The tests output junit that is passed onto Appveyor and the results are visible in the "Tests" tab for each job.~~

~~This is what happens when tests fail you can click in the failing tests to see the trace for each error.~~

The first commit comes from #4771. Because luv in windows needs libuv 1.9.

bfredl · 2016-08-15T13:15:42Z

.ci/msys_test.bat

@@ -0,0 +1,7 @@
+:: FIXME(equalsraf) currently not all tests work on Windows
+:: for now just run what we can
+mingw32-make functionaltest TEST_FILE=test\functional\eval\printf_spec.lua VERBOSE=1 || goto :error


Should we use a test tag for this? That would enable whatever granularity we want, and would put the information what is supported directly at the tests.

You mean tag describe() with #windows as we verify that each test runs on windows. Might be a bother later on, when almost all tests would be tagger, but its a good way to go about it. TEST_FILTER might also work.

I'm trying to get this to work but TEST_TAG=windows doesn't seem to have any effect (no tests are run). I assumed the only thing I had to do was add #windows in describe but maybe I was wrong.

I'm trying to get this to work but TEST_TAG=windows doesn't seem to have any effect (no tests are run). I assumed the only thing I had to do was add #windows in describe but maybe I was wrong.

That's the way it supposed to be supposed to be (and it works for me locally on linux).

If we expect more than say half of the test files to work on windows soon, it might be better to blacklist the non-working with pending(). This will mean we gradually remove blacklists as more and more tests work on windows, which might feel better as the tests look cleaner as windows coverage improves.

Also a single pending() can be used to short circuit an entire test file , as in the python3_spec.

Problem with using pending() like that is that for it would likely end up being used in most tests. And the extra call to pending() seems expensive.

That's the way it supposed to be supposed to be (and it works for me locally on linux).

I see whats happening now. In tests that do the early checks for pending, like python3_spec the tag is only matched after evaluating the pending check - turns out the check itself causes it to fail in windows :D. There seem to be other cases where early evaluation of the test specs causes failures, but its hard to debug because there is no output.

Ah the other culprit seems to be luajit crashes when running certain test groups (cannot pinpoint individual test), assert in lj_gc.c Line 666 IIIRC.

Ah the other culprit seems to be luajit crashes when running certain test groups

You mean it crashes even when it "runs" a test group in search of requested tagged tests? (I assume before_each() won't run if the describe has no tests to be run)

Problem with using pending() like that is that for it would likely end up being used in most tests. And the extra call to pending() seems expensive.

I suppose it would be a pending per file to start with, which is a lot less expensive than "run" all the describes in search of tagged tests. These would be removed as windows support/coverage improves, not being end up with, except for features/tests we decide to not support on windows, which we are supposed to block on windows using pending().

You mean it crashes even when it "runs" a test group in search of requested tagged tests?

Yes even running busted -l to enumerate the tests causes this. Because this code is not a test and is right at the top of the file (but it does spawn Neovim for the checks).

equalsraf · 2016-08-15T23:01:55Z

FYI seeing failures with busted tests because of lunarmodules/busted#528

Edit: Nevermind just fetched master :D

equalsraf · 2016-08-16T15:13:37Z

IIIRC if the ouput is set as junit and tests block we get nothing that helps debugging. Is it possible to have multiple output handlers in busted?

equalsraf · 2016-08-16T17:03:12Z

It seems Appveyor just had a little fit processing junit, maybe its best to drop it all together. Here is the output in gtest. Several tests are failing, but at least I have marked as pending all the tests that caused the build to block (maybe it was related to the use of Screen).

justinmk · 2016-08-16T17:36:30Z

test/functional/ui/syntax_conceal_spec.lua

+    pending('FIXME: Windows', function() end)
+    return
+  end
+end


This block could go in a helpers.pending_if_win32() function.

Not sure I can call pending in that context, https://ci.appveyor.com/project/equalsraf/neovim/build/1114/job/t2e8u4yf440fyuqp#L8667 - helpers.lua

the return is still needed in the test file itself, unfortunately. Something like this maybe:

if helpers.pending_if_win32() then return end

equalsraf · 2016-08-17T23:28:13Z

Had to rebase over #5198 to get rid of some of the errors.

I might have disabled some tests that could have passed, tried disabling everything that used Screen and shell or external commands. Some of those were causing errors others blocked the build. In total I marked 97 files as pending,

I'm pretty sure I spotted some bugs in the following tests, but for now they are disabled too (full log)

functional\ex_cmds\recover_spec.lua @ 27: :preserve after_each
getcwd_spec.lua @ 10: getcwd after_each
normal\K_spec.lua @ 27: K invokes non-prefixed

@justinmk the only way I was able to write a little shortcut for pending was to pass pending as argument to the check.

I ended up disabling the junit/appveyor integration and leaving the output as gtest. Time would be best spent writing a busted output handler.

equalsraf · 2016-08-17T23:50:07Z

Appveyor/Windows (updated)

1222 tests from 213 test files ran. (45805.39 ms total)
[  PASSED  ] 1161 tests.
[ SKIPPED  ] 61 tests, listed below:

Travis/Linux

1666 tests from 213 test files ran. (95024.03 ms total)
[  PASSED  ] 1660 tests.
[ SKIPPED  ] 6 tests, listed below:

Busted now builds on Windows, remove the check. In Windows the binary is called busted.bat.

The argument quotes in the luv build recipe did not work in Windows.

Most functional tests don't work on Windows yet, for now enable a subset of the tests in Appveyor builds.

equalsraf · 2016-08-29T22:48:05Z

Ping, anyone up for this one?

fwalch · 2016-08-30T22:05:06Z

CMakeLists.txt

@@ -498,6 +498,7 @@ if(BUSTED_PRG)
        -DTEST_DIR=${CMAKE_CURRENT_SOURCE_DIR}/test
        -DBUILD_DIR=${CMAKE_BINARY_DIR}
        -DTEST_TYPE=unit
+        -DCMAKE_SYSTEM_NAME=${CMAKE_SYSTEM_NAME}


Do we really need to pass this into the "target CMake"? Isn't it just defined there by default as well?

CMake script invocation with -P does not pass on the variables in the current script.

Okay, but CMAKE_SYSTEM_NAME is a builtin variable, no? I mean, we don't define it manually in the current script, either.

Edit: nevermind, apparently this is only defined when CMake is run on CMakeLists.txt, not if CMake is run "standalone".

fwalch · 2016-08-30T22:17:37Z

Some minor comments, otherwise LGTM 👍

In Windows Lua's os.tmpname() returns relative paths starting with \s, prepend them with $TEMP to generate a valid path. In OS X os.tmpname() returns paths in '/tmp' but they should be in '/private/tmp'. We cannot use os_name() for platform detection because some tests use tempname() before nvim is spawned, instead use one of the following: 1. Set SYSTEM_NAME environment variable before calling the tests, it is set from CMAKE_SYSTEM_NAME(i.e. uname -s or 'Windows') 2. Call uname -s 3. Assume windows

equalsraf · 2016-08-31T10:59:59Z

Fixed all comments. Several screen tests were failing in the GCOV Travis job. But it seems ok after a couple retries.

Tagging RDY

bfredl · 2016-08-31T19:39:22Z

LGTM 👍

There are some "pure" screen tests I wouldn't expect to fail on windows, like bufhl, but those could be gone through in a follow-up PR, no need to block this.

FEATURES: 0b5a7e4 neovim#4432 API: external UIs can render custom popupmenu c6ac4f8 neovim#4934 API: call any API method from vimscript 31df051 neovim#4568 API: nvim_call_atomic(): multiple calls in a single request b268ba3 neovim#5424 API: nvim_win_get_number(), nvim_tabpage_get_number() e7e2844 has("nvim-1.2.3") checks for a specific Nvim version 522b885 neovim#5295, neovim#5493 `:CheckHealth` checks tmux, terminfo, performance 719dae2 neovim#5384 events: allow event processing in getchar() f25797f neovim#5386 API: metadata: Nvim version & API level 22dfe69 neovim#5389 API: metadata: "since", "deprecated_since" 605e743 Added QuickFixLine highlight group CHANGES: 4af6ec7 neovim#5253 perf: Disable clipboard in do_cmdline() 6e9f329 neovim#5299 perf: Skip foldUpdate() in insert-mode. 9d4fcec neovim#5426 perf: Do not auto-update folds for some foldmethods. eeec0ca neovim#5419 tui: Default to normal-mode cursor shape. FIXES: e838452 neovim#5436 tui: Fix "weird characters" / "bleeding termcodes" 10a54ad neovim#5243 signal_init: Always unblock SIGCHLD. bccb49b neovim#5316 eval.c: Fix memory leak for detached pty job 626065d neovim#5227 tchdir: New tab should inherit CWD. cd321b7 neovim#5292 getcwd(): Return empty string if CWD is invalid. 6127eae shada: Fix non-writeable ShaDa directory handling ca65514 neovim#2789 system(): Respect shellxescape, shellxquote 2daf54e neovim#4874 Restore vim-like tab dragging 0c536b5 neovim#5319 syntax.c: Support bg/fg special color-names. 3c53371 neovim#4972 from justinmk/schedule-ui_refresh 68bcb32 neovim#4789 tui.c: Do not wait for tui loop on teardown. c8b6ec2 neovim#5409 v:count broken in command-line window 6bc3bce neovim#5461 fix emoji display 51937e1 neovim#5470 fix :terminal with :argadd, :argu 79d77da neovim#5481 external UIs: opening multiple files from command-line 657ba62 neovim#5501 rplugin: resolve paths in manifest file 6a6f188 neovim#5502 system('foo &', 'bar'): Show error, don't crash. 1ff162c neovim#5515 os_nodetype: open fd with O_NONBLOCK 2a6c5bb neovim#5450 modeline: Handle version number overflow. 0ade1bb neovim#5225 CI tests now run against Windows!

equalsraf force-pushed the windows-functionaltests branch 2 times, most recently from 0d4c5c5 to e5b2cee Compare August 15, 2016 12:03

marvim added the RFC label Aug 15, 2016

bfredl reviewed Aug 15, 2016
View reviewed changes

equalsraf force-pushed the windows-functionaltests branch from e5b2cee to 139af83 Compare August 15, 2016 23:07

equalsraf changed the title ~~[RFC] Enable functional tests in Appveyor~~ [WIP/RFC] Enable functional tests in Appveyor Aug 16, 2016

equalsraf force-pushed the windows-functionaltests branch from 3755cf4 to 477a704 Compare August 16, 2016 13:44

equalsraf force-pushed the windows-functionaltests branch from 252aa9c to 0091510 Compare August 16, 2016 16:38

equalsraf force-pushed the windows-functionaltests branch from 0091510 to 3340036 Compare August 16, 2016 17:09

justinmk reviewed Aug 16, 2016
View reviewed changes

equalsraf mentioned this pull request Aug 16, 2016

Windows: improvements #5229

Closed

14 tasks

equalsraf force-pushed the windows-functionaltests branch 12 times, most recently from 14e7507 to da628ce Compare August 17, 2016 22:46

equalsraf changed the title ~~[WIP/RFC] Enable functional tests in Appveyor~~ [RFC] Enable functional tests in Appveyor Aug 17, 2016

equalsraf force-pushed the windows-functionaltests branch from da628ce to d7bcacb Compare August 18, 2016 09:22

equalsraf added 4 commits August 26, 2016 08:09

third-party: Build busted in Windows

bed6104

Busted now builds on Windows, remove the check. In Windows the binary is called busted.bat.

third-party: Windows fix for luv build recipe

703cf31

The argument quotes in the luv build recipe did not work in Windows.

Appveyor: Enable functional tests

0089ef6

Most functional tests don't work on Windows yet, for now enable a subset of the tests in Appveyor builds.

Mark some functional tests as pending in Windows

39c628d

equalsraf force-pushed the windows-functionaltests branch from a6ec725 to ad60868 Compare August 26, 2016 07:22

fwalch reviewed Aug 30, 2016
View reviewed changes

equalsraf force-pushed the windows-functionaltests branch 7 times, most recently from 95586bf to 9ce81f7 Compare August 31, 2016 10:32

equalsraf changed the title ~~[RFC] Enable functional tests in Appveyor~~ [RDY] Enable functional tests in Appveyor Aug 31, 2016

marvim added RDY and removed RFC labels Aug 31, 2016

bfredl merged commit 0ade1bb into neovim:master Aug 31, 2016

jszakmeister removed the RDY label Aug 31, 2016

equalsraf deleted the windows-functionaltests branch August 31, 2016 20:29

justinmk mentioned this pull request Oct 10, 2016

Update to libuv 1.9.1 #4771

Closed

Uh oh!

[RDY] Enable functional tests in Appveyor #5225

[RDY] Enable functional tests in Appveyor #5225

Uh oh!

Conversation

equalsraf commented Aug 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

equalsraf commented Aug 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

equalsraf commented Aug 16, 2016

Uh oh!

equalsraf commented Aug 16, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

equalsraf commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

equalsraf commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

equalsraf commented Aug 29, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fwalch Aug 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fwalch commented Aug 30, 2016

Uh oh!

equalsraf commented Aug 31, 2016

Uh oh!

bfredl commented Aug 31, 2016

Uh oh!

Uh oh!

equalsraf commented Aug 15, 2016 •

edited

Loading

equalsraf commented Aug 15, 2016 •

edited

Loading

equalsraf commented Aug 17, 2016 •

edited

Loading

equalsraf commented Aug 17, 2016 •

edited

Loading

fwalch Aug 30, 2016 •

edited

Loading