Skip to content

Fix deadlock in consensus manager #5987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 14, 2025
Merged

Fix deadlock in consensus manager #5987

merged 1 commit into from
Feb 14, 2025

Conversation

agourlay
Copy link
Member

Found by lockbud

[2025-02-13T20:25:26Z WARN  lockbud::callbacks] [
      {
        "ConflictLock": {
          "bug_kind": "ConflictLock",
          "possibility": "Possibly",
          "diagnosis": [
            {
              "first_lock_type": "ParkingLotRead(content_manager::consensus::persistent::Persistent)",
              "first_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:984:13: 984:23 (#0)",
              "second_lock_type": "ParkingLotMutex(content_manager::consensus::consensus_wal::ConsensusOpWal)",
              "second_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:1012:13: 1012:28 (#0)",
              "callchains": []
            },
            {
              "first_lock_type": "ParkingLotMutex(content_manager::consensus::consensus_wal::ConsensusOpWal)",
              "first_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:965:27: 965:42 (#0)",
              "second_lock_type": "ParkingLotRead(content_manager::consensus::persistent::Persistent)",
              "second_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:967:21: 967:43 (#0)",
              "callchains": []
            }
          ],
          "explanation": "Locks mutually wait for each other to form a cycle"
        }
      },
      {
        "ConflictLock": {
          "bug_kind": "ConflictLock",
          "possibility": "Possibly",
          "diagnosis": [
            {
              "first_lock_type": "ParkingLotRead(content_manager::consensus::persistent::Persistent)",
              "first_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:984:13: 984:23 (#0)",
              "second_lock_type": "ParkingLotMutex(content_manager::consensus::consensus_wal::ConsensusOpWal)",
              "second_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:1012:13: 1012:28 (#0)",
              "callchains": []
            },
            {
              "first_lock_type": "ParkingLotMutex(content_manager::consensus::consensus_wal::ConsensusOpWal)",
              "first_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:955:13: 955:22 (#0)",
              "second_lock_type": "ParkingLotRead(content_manager::consensus::persistent::Persistent)",
              "second_lock_span": "lib/storage/src/content_manager/consensus_manager.rs:956:13: 956:23 (#0)",
              "callchains": []
            }
          ],
          "explanation": "Locks mutually wait for each other to form a cycle"
        }
      }
    ]

@agourlay agourlay marked this pull request as ready for review February 13, 2025 20:56
timvisee
timvisee approved these changes Feb 14, 2025
View reviewed changes
Copy link
Member

@timvisee timvisee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find! Thanks! 🙌

Here's an example where we locked wal first, then persistent:

qdrant/lib/storage/src/content_manager/consensus_manager.rs

Lines 955 to 956 in 16734b9

let wal_guard = self.wal.lock();
let persistent = self.persistent.read();

Your change matches that order 👍

@timvisee timvisee requested a review from ffuugoo February 14, 2025 10:07
@agourlay agourlay merged commit 2381082 into dev Feb 14, 2025
17 checks passed
@agourlay agourlay deleted the fix-deadlock-in-consensus-manger branch February 14, 2025 10:47
timvisee pushed a commit that referenced this pull request Feb 17, 2025
@agourlay @timvisee
Fix deadlock in consensus manager (#5987)
3c1dfc1
@timvisee timvisee mentioned this pull request Feb 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timvisee timvisee timvisee approved these changes

@ffuugoo ffuugoo ffuugoo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@agourlay @timvisee @ffuugoo