CcrRepositoryIT#testFollowerMappingIsUpdated can fail with assertion error #37887
Description
Unfortunately the failure doesn't reproduce for me locally.
Link to the build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/202
Command to reproduce:
./gradlew :x-pack:plugin:ccr:internalClusterTest \
-Dtests.seed=20B21185CE3EF1E3 \
-Dtests.class=org.elasticsearch.xpack.ccr.CcrRepositoryIT \
-Dtests.method="testFollowerMappingIsUpdated" \
-Dtests.security.manager=true \
-Dtests.locale=ar-TN \
-Dtests.timezone=Atlantic/Jan_Mayen \
-Dcompiler.java=11 \
-Druntime.java=8
Relevant excerpt from the logs:
1> [2019-01-25T21:16:59,592][INFO ][o.e.c.m.MetaDataCreateIndexService] [leader0] [index1] creating index, cause [api], templates [], shards [1]/[0], mappings [doc]
1> [2019-01-25T21:16:59,624][INFO ][o.e.c.r.a.AllocationService] [leader0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index1][0]] ...]).
1> [2019-01-25T21:16:59,643][INFO ][o.e.x.c.CcrRepositoryIT ] [testFollowerMappingIsUpdated] ensure green leader indices [index1]
1> [2019-01-25T21:16:59,684][WARN ][o.e.r.b.FileRestoreContext] [followerd3] [[index2][0]] [_latest_/_latest_] failed to delete file [extra0] during snapshot cleanup
1> [2019-01-25T21:16:59,694][INFO ][o.e.c.m.MetaDataMappingService] [leader0] [index1/yHsi0cZ5TpmL9dnQq2hcDQ] update_mapping [doc]
1> [2019-01-25T21:16:59,728][INFO ][o.e.c.r.a.AllocationService] [followerm2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index2][0]] ...]).
1> [2019-01-25T21:16:59,766][INFO ][o.e.c.m.MetaDataDeleteIndexService] [leader0] [index1/yHsi0cZ5TpmL9dnQq2hcDQ] deleting index
1> [2019-01-25T21:16:59,816][INFO ][o.e.c.m.MetaDataDeleteIndexService] [followerm2] [index2/sXK7-Cy5SSqjmw9hdCu6rQ] deleting index
1> [2019-01-25T21:16:59,837][INFO ][o.e.x.c.CcrRepositoryIT ] [testFollowerMappingIsUpdated] after test
FAILURE 0.40s J3 | CcrRepositoryIT.testFollowerMappingIsUpdated <<< FAILURES!
> Throwable #1: java.lang.AssertionError: expected:<2> but was:<1>
> at __randomizedtesting.SeedInfo.seed([20B21185CE3EF1E3:2B41836BA05FE850]:0)
> at org.elasticsearch.xpack.ccr.CcrRepositoryIT.testFollowerMappingIsUpdated(CcrRepositoryIT.java:342)
> at java.lang.Thread.run(Thread.java:748)