Skip to content

Conversation

iypetrov
Copy link
Contributor

Which problem is this PR solving?

Description of the changes

  • Added simple retry logic with exponential backoff to the Exists method in ILMClient to improve resilience. This helps prevent random test failures and mitigates issues caused by transient network errors.

How was this change tested?

  • All existing tests for ILMClient passed.
  • An additional unit test was added to verify that if the HTTP request fails a few times, the client correctly retries and eventually completes the Exists operation successfully once the service becomes healthy.

Checklist

@iypetrov iypetrov requested a review from a team as a code owner May 14, 2025 05:52
@iypetrov iypetrov requested a review from mahadzaryab1 May 14, 2025 05:52
@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch 2 times, most recently from ee9eef8 to 3ccc1ea Compare May 14, 2025 06:34
@yurishkuro yurishkuro requested a review from Copilot May 15, 2025 23:15
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses a flaky test by introducing retry logic with exponential backoff in the ILMClient.Exists method. The changes aim to improve resilience against transient network issues.

  • Added a test (TestExists_Retries) to verify the retry behavior.
  • Implemented exponential backoff retry logic in the ILMClient.Exists method.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
internal/storage/elasticsearch/client/ilm_client.go Added retry loop with exponential backoff for Exists
internal/storage/elasticsearch/client/ilm_client_test.go Introduced a new unit test to validate retry logic

@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from 377cd99 to 0695551 Compare May 16, 2025 05:01
Copy link

codecov bot commented May 17, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.19%. Comparing base (bfa0b2c) to head (d81cad9).
Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7120      +/-   ##
==========================================
- Coverage   96.20%   96.19%   -0.02%     
==========================================
  Files         362      362              
  Lines       21853    21868      +15     
==========================================
+ Hits        21024    21036      +12     
- Misses        620      622       +2     
- Partials      209      210       +1     
Flag Coverage Δ
badger_v1 9.87% <ø> (ø)
badger_v2 1.90% <ø> (ø)
cassandra-4.x-v1-manual 14.85% <ø> (ø)
cassandra-4.x-v2-auto 1.89% <ø> (ø)
cassandra-4.x-v2-manual 1.89% <ø> (ø)
cassandra-5.x-v1-manual 14.85% <ø> (ø)
cassandra-5.x-v2-auto 1.89% <ø> (ø)
cassandra-5.x-v2-manual 1.89% <ø> (ø)
elasticsearch-6.x-v1 20.69% <ø> (ø)
elasticsearch-7.x-v1 20.77% <ø> (ø)
elasticsearch-8.x-v1 20.95% <ø> (ø)
elasticsearch-8.x-v2 1.90% <ø> (ø)
grpc_v1 11.41% <ø> (ø)
grpc_v2 1.90% <ø> (ø)
kafka-3.x-v1 10.14% <ø> (ø)
kafka-3.x-v2 1.90% <ø> (ø)
memory_v2 1.90% <ø> (ø)
opensearch-1.x-v1 20.82% <ø> (ø)
opensearch-2.x-v1 20.82% <ø> (ø)
opensearch-2.x-v2 1.90% <ø> (ø)
query 1.90% <ø> (ø)
tailsampling-processor 0.51% <ø> (ø)
unittests 95.01% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from 97dcaad to 4046f17 Compare May 19, 2025 04:02
@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from 988991f to e996f54 Compare May 20, 2025 10:13
@yurishkuro yurishkuro enabled auto-merge May 20, 2025 20:28
iypetrov added 3 commits May 21, 2025 07:10
…efix,_[ROLLOVER=true] (jaegertracing#6838)

Signed-off-by: Ilia Yavorov Petrov <ilia.yavorov.petrov@gmail.com>
Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
…/cenkalti/backoff/v5

Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
auto-merge was automatically disabled May 21, 2025 04:10

Head branch was pushed to by a user without write access

@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from 5bdecc6 to 26b58f3 Compare May 21, 2025 04:10
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@yurishkuro yurishkuro changed the title [Fix] Flaky Test: OpenSearch TestIndexCleaner/RemoveRolloverIndices_pr… Add retries to ILM client May 22, 2025
@yurishkuro
Copy link
Member

need to run make generate-mocks

iypetrov added 2 commits May 22, 2025 07:10
Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from c033f6b to dc8374a Compare May 22, 2025 04:10
iypetrov and others added 2 commits May 27, 2025 00:19
Signed-off-by: Ilia Petrov <ilia.yavorov.petrov@gmail.com>
@iypetrov iypetrov force-pushed the fix-flaky-test-openseach-testindexcleaner-remove_rollover-indices_prefix-rollover-true branch from 3f0a99b to d81cad9 Compare May 26, 2025 21:19
@yurishkuro yurishkuro enabled auto-merge May 26, 2025 21:45
@yurishkuro yurishkuro added this pull request to the merge queue May 26, 2025
Merged via the queue into jaegertracing:main with commit 2f2b6fb May 26, 2025
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky Test: OpenSeach TestIndexCleaner/RemoveRolloverIndices_prefix,_[ROLLOVER=true]
2 participants