Skip to content

Fix panic in shard cleanup task when timeout subtraction overflows #6086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 28, 2025
Merged

Fix panic in shard cleanup task when timeout subtraction overflows #6086

merged 1 commit into from
Feb 28, 2025

Conversation

timvisee
Copy link
Member

If the elapsed timeout is more than the timeout duration, this previously panicked.

Trace:

ERROR qdrant::startup: Panic backtrace: 

2025-02-28 10:49:18.518	
   0: qdrant::startup::setup_panic_hook::{{closure}}
2025-02-28 10:49:18.518	
   1: std::panicking::rust_panic_with_hook
2025-02-28 10:49:18.518	
   2: std::panicking::begin_panic_handler::{{closure}}
2025-02-28 10:49:18.518	
   3: std::sys::backtrace::__rust_end_short_backtrace
2025-02-28 10:49:18.518	
   4: rust_begin_unwind
2025-02-28 10:49:18.518	
   5: core::panicking::panic_fmt

2025-02-28 10:49:18.518	
   6: core::option::expect_failed
2025-02-28 10:49:18.518	
   7: collection::collection::clean::ShardCleanTasks::await_task::{{closure}}
2025-02-28 10:49:18.518	
   8: actix_web::handler::handler_service::{{closure}}::{{closure}}
2025-02-28 10:49:18.518	
   9: <actix_web::resource::Resource<T> as actix_web::service::HttpServiceFactory>::register::{{closure}}::{{closure}}
2025-02-28 10:49:18.518	
  10: <actix_utils::future::either::Either<L,R> as core::future::future::Future>::poll
2025-02-28 10:49:18.518	
  11: <qdrant::actix::auth::AuthMiddleware<S> as actix_service::Service<actix_web::service::ServiceRequest>>::call::{{closure}}
2025-02-28 10:49:18.518	
  12: <actix_web_extras::middleware::condition::ConditionMiddlewareFuture<E,D> as core::future::future::Future>::poll
2025-02-28 10:49:18.518	
  13: <actix_cors::middleware::CorsMiddleware<S> as actix_service::Service<actix_web::service::ServiceRequest>>::call::{{closure}}
2025-02-28 10:49:18.518	
  14: <qdrant::actix::actix_telemetry::ActixTelemetryService<S> as actix_service::Service<actix_web::service::ServiceRequest>>::call::{{closure}}
2025-02-28 10:49:18.518	
  15: <actix_service::map_err::MapErrFuture<A,Req,F,E> as core::future::future::Future>::poll
2025-02-28 10:49:18.518	
  16: actix_http::h1::dispatcher::InnerDispatcher<T,S,B,X,U>::poll_response
2025-02-28 10:49:18.518	
  17: <actix_http::h1::dispatcher::Dispatcher<T,S,B,X,U> as core::future::future::Future>::poll
2025-02-28 10:49:18.518	
  18: <actix_server::service::StreamService<S,I> as actix_service::Service<(actix_server::worker::WorkerCounterGuard,actix_server::socket::MioStream)>>::call::{{closure}}
2025-02-28 10:49:18.518	
  19: tokio::runtime::task::raw::poll
2025-02-28 10:49:18.518	
  20: tokio::task::local::LocalSet::tick
2025-02-28 10:49:18.518	
  21: tokio::task::local::LocalSet::run_until::{{closure}}
2025-02-28 10:49:18.518	
  22: std::sys::backtrace::__rust_begin_short_backtrace
2025-02-28 10:49:18.518	
  23: core::ops::function::FnOnce::call_once{{vtable.shim}}
2025-02-28 10:49:18.518	
  24: std::sys::pal::unix::thread::Thread::new::thread_start
2025-02-28 10:49:18.518	
  25: <unknown>
2025-02-28 10:49:18.518	
  26: <unknown>

2025-02-28 10:49:18.518	
    
2025-02-28 10:49:18.518	
2025-02-28T09:49:18.518411Z ERROR qdrant::startup: Panic occurred in file /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/time.rs at line 1138: overflow when subtracting durations

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

@timvisee timvisee merged commit badaa5c into dev Feb 28, 2025
17 checks passed
@timvisee timvisee deleted the fix-clean-shard-timeout-panic branch February 28, 2025 16:09
timvisee added a commit that referenced this pull request Mar 21, 2025
@timvisee
Fix panic in shard cleanup task when timeout subtraction overflows (#…
39aaab5 
…6086)
@timvisee timvisee mentioned this pull request Mar 21, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@agourlay agourlay agourlay approved these changes

@generall generall generall approved these changes

@coszio coszio Awaiting requested review from coszio

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@timvisee @agourlay @generall