Elasticsearch 8.x support #3571
Description
Requirement - what kind of business use case are you trying to solve?
Elastic recently released Elasticsearch 8.0, by now they've released 8.1.
Problem - what in Jaeger blocks you from solving the requirement?
The ES index can't be created anymore, I think the legacy template api part is now completely deprecated:
jaeger-collector | {"level":"info","ts":1646824639.4679213,"caller":"config/config.go:205","msg":"Elasticsearch detected","version":8}
jaeger-collector | {"level":"info","ts":1646824639.4697242,"caller":"zapgrpc/zapgrpc.go:129","msg":"Deprecation warning: 299 Elasticsearch-8.1.0-3700f7679f7d95e36da0b43762189bab189bc53a \"Legacy index templates are deprecated in favor of composable templates.\""}
jaeger-collector | {"level":"fatal","ts":1646824639.4699457,"caller":"./main.go:84","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template [type=parse_exception]","stacktrace":"main.main.func1\n\t./main.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.3.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}The https://www.elastic.co/guide/en/elasticsearch/reference/8.1/migrating-8.0.html lists now (somewhere in the middle)
The [create or update index template API](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/indices-templates-v1.html)’s template parameter has been removed.
Details
In 6.0, we deprecated the template parameter in create or update index template requests in favor of using index_patterns. Support for the template parameter is now removed in 8.0.
Impact
Use the create or update index template API's index_patterns parameter. Requests that include the template parameter will return an error.I think the code falls back to the unversioned mappings if it does not find a specific mapping
So, in my opinion the code should not fall back to the jaeger-span.json mapping but instead use the
jaeger-span-7.json for newer versions, e.g. use the -7 jsons for 8 etc.
e.g. change line 42 from == 7 to >=7
jaeger/plugin/storage/es/mappings/mapping.go
Lines 40 to 46 in 7872d1b
Workaround
Force the ES Version to 7 via --es.version=7