[Bug]: ElasticSearch storage not deduplicating multiply-archived spans #6001
Description
What happened?
As an SRE deploying Jaeger at a medium-sized engineering organization (the Wikimedia Foundation), I want to enable trace archival support, for the usual benefits discussed in #57:
Users often ask for this feature. The trace retention period is usually short, and often people would put a link to the trace into some issue tracker to improve service performance, but by the time someone starts working on the ticket the trace has already expired.
However, spans are written to our Elasticsearch(/OpenSearch) archival storage regardless of whether or not they already exist. When archived traces are read from archival storage and displayed, this leads to spans that are duplicated exponentially by however many times 'archive' was clicked.
We're running Jaeger on Kubernetes using a slight modification of the official Helm chart (if curious see chart and values)
Steps to reproduce
- Configure jaeger + OpenSearch similar to ours, with archive tracing
- Send some tracing data
- Archive the trace
- Archive the trace again
- Restart jaeger-query with
--es.max-span-ageyounger than the trace is - Refresh the UI page for the trace
Expected behavior
Pressing the 'archive' button on a trace that already exists in archive storage would not mutate anything in archive storage.
Relevant log output
No response
Screenshot
When expanded, the SpanIDs for these traces show as:
- 8d3589e43d599843
- 8d3589e43d599843_1
- 8d3589e43d599843_2
- 8d3589e43d599843_3
All child spans are similarly duplicated.
If you examine the OpenSearch index directly, you find that the spans are duplicate documents with everything identical except the index-internal _id field:
Additional context
No response
Jaeger backend version
v1.56.0
SDK
Envoy sending OpenTelemetry
Pipeline
Envoy -> otelcol -> jaeger-collector
Stogage backend
OpenSearch 2.7.0
Operating system
Linux
Deployment model
Kubernetes
Deployment configs
Args:
--es-archive.enabled=true
--es-archive.index-date-separator=.
--es-archive.num-replicas=2
--es-archive.num-shards=1
--es-archive.tags-as-fields.all=true
--es-archive.tls.enabled=true
--es.index-date-separator=.
--es.max-span-age=756h0m0s
--es.num-replicas=2
--es.num-shards=1
--es.tags-as-fields.all=true
--es.tls.enabled=true
[...]
Environment:
SPAN_STORAGE_TYPE: elasticsearch
ES_SERVER_URLS: https://logs-api.svc.eqiad.wmnet:443
ES_USERNAME: jaeger-prod
ES_PASSWORD: <set to the key 'password' in secret 'main-jaeger-elasticsearch'> Optional: false
ES_ARCHIVE_SERVER_URLS: https://logs-api.svc.eqiad.wmnet:443
ES_ARCHIVE_USERNAME: jaeger-prod
ES_ARCHIVE_PASSWORD: <set to the key 'password' in secret 'main-jaeger-elasticsearch'> Optional: false
QUERY_BASE_PATH: /