Skip to content

Conversation

yurishkuro
Copy link
Member

@yurishkuro yurishkuro commented Jul 3, 2025

Which problem is this PR solving?

User reported panic

I am running into this error “recoveryhandler/zap.go:22 runtime error: invalid memory address or nil pointer dereference” while querying traces using Jaeger v2.7.0: It only happens on one service type (Traefik), so I am puzzled why it happens, since all traces come through the same OpenTelemetry Collector. Does the attached log provide any insight into what happens (I am not a developer)?

github.com/gorilla/handlers@v1.5.2/recovery.go:76
runtime.gopanic
    runtime/panic.go:792
runtime.panicmem
    runtime/panic.go:262
runtime.sigpanic
    runtime/signal_unix.go:925
go.opentelemetry.io/collector/pdata/pcommon.Slice.Len
    go.opentelemetry.io/collector/pdata@v1.33.0/pcommon/slice.go:48
github.com/jaegertracing/jaeger/internal/jptrace.GetWarnings
    github.com/jaegertracing/jaeger/internal/jptrace/warning.go:27
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.transferWarningsToModelSpans
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:150
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1BatchesFromTraces
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:23
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.modelTraceFromOtelTrace
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:108
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.func1
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:57
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.V1TracesFromSeq2.AggregateTraces.func2.func3
    github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:33
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore.(*TraceReader).FindTraces.func1
    github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore/reader.go:88
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.AggregateTraces.func2
    github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:21
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:52
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.(*SpanReader).FindTraces
    github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/spanreader.go:83
github.com/jaegertracing/jaeger/cmd/query/app/querysvc.QueryService.FindTraces
    github.com/jaegertracing/jaeger/cmd/query/app/querysvc/query_service.go:102
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search

Description of the changes

  • Do not assume that the span warning is always a slice (hypothesis here is that it might have been created as a slice originally but when stored to ES and loaded back it came back as a string).

How was this change tested?

  • Unit test, not reproducable

Signed-off-by: Yuri Shkuro <github@ysh.us>
Comment on lines 108 to 113
func TestGetWarnings_EmptySpan(t *testing.T) {
span := ptrace.NewSpan()
span.Attributes().PutStr(WarningsAttribute, "warning-1")
actual := GetWarnings(span)
assert.Nil(t, actual)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test name TestGetWarnings_EmptySpan doesn't match what the test is actually doing. The span isn't empty - it has a string warning attribute set.

More importantly, there's a discrepancy between the test expectation and the updated implementation. The test expects nil to be returned, but according to the new code in warning.go, when a string attribute is set, it should return a slice containing that string ([]string{"warning-1"}).

Either:

  1. Update the test expectation to assert.Equal(t, []string{"warning-1"}, actual) to match the implementation, or
  2. Rename the test to better describe what it's testing (e.g., TestGetWarnings_StringAttribute)

The current test doesn't properly validate the new fallback behavior for non-slice warning attributes.

Suggested change
func TestGetWarnings_EmptySpan(t *testing.T) {
span := ptrace.NewSpan()
span.Attributes().PutStr(WarningsAttribute, "warning-1")
actual := GetWarnings(span)
assert.Nil(t, actual)
}
func TestGetWarnings_StringAttribute(t *testing.T) {
span := ptrace.NewSpan()
span.Attributes().PutStr(WarningsAttribute, "warning-1")
actual := GetWarnings(span)
assert.Equal(t, []string{"warning-1"}, actual)
}

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

yurishkuro and others added 2 commits July 3, 2025 17:30
Signed-off-by: Yuri Shkuro <github@ysh.us>
Copy link

codecov bot commented Jul 3, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.19%. Comparing base (2fe4b58) to head (3205448).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7293      +/-   ##
==========================================
- Coverage   96.21%   96.19%   -0.03%     
==========================================
  Files         374      374              
  Lines       22686    22691       +5     
==========================================
- Hits        21828    21827       -1     
- Misses        641      645       +4     
- Partials      217      219       +2     
Flag Coverage Δ
badger_v1 9.80% <0.00%> (-0.01%) ⬇️
badger_v2 1.78% <75.00%> (+0.01%) ⬆️
cassandra-4.x-v1-manual 12.96% <0.00%> (-0.01%) ⬇️
cassandra-4.x-v2-auto 1.77% <75.00%> (+0.01%) ⬆️
cassandra-4.x-v2-manual 1.77% <75.00%> (+0.01%) ⬆️
cassandra-5.x-v1-manual 12.96% <0.00%> (-0.01%) ⬇️
cassandra-5.x-v2-auto 1.77% <75.00%> (+0.01%) ⬆️
cassandra-5.x-v2-manual 1.77% <75.00%> (+0.01%) ⬆️
elasticsearch-6.x-v1 20.68% <0.00%> (-0.02%) ⬇️
elasticsearch-7.x-v1 20.73% <0.00%> (-0.02%) ⬇️
elasticsearch-8.x-v1 20.91% <0.00%> (-0.02%) ⬇️
elasticsearch-8.x-v2 1.78% <75.00%> (+0.01%) ⬆️
grpc_v1 11.33% <0.00%> (-0.01%) ⬇️
grpc_v2 1.78% <75.00%> (+0.01%) ⬆️
kafka-3.x-v1 10.16% <0.00%> (-0.01%) ⬇️
kafka-3.x-v2 1.78% <75.00%> (+0.01%) ⬆️
memory_v2 1.78% <75.00%> (+0.01%) ⬆️
opensearch-1.x-v1 20.78% <0.00%> (-0.02%) ⬇️
opensearch-2.x-v1 20.78% <0.00%> (-0.02%) ⬇️
opensearch-2.x-v2 1.78% <75.00%> (+0.01%) ⬆️
query 1.78% <75.00%> (+0.01%) ⬆️
tailsampling-processor 0.49% <0.00%> (-0.01%) ⬇️
unittests 95.06% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@yurishkuro yurishkuro added this pull request to the merge queue Jul 3, 2025
Merged via the queue into jaegertracing:main with commit a6ef817 Jul 3, 2025
60 checks passed
@yurishkuro yurishkuro deleted the fix-warning branch July 3, 2025 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants