Skip to content

Conversation

sipa
Copy link
Member

@sipa sipa commented May 23, 2024

Extracted from #30126.

This adds a VecDeque data type, inspired by std::deque, but backed by a single allocated memory region used as a ring buffer instead of a linked list of arrays. This gives better memory locality and less allocation overhead, plus better guarantees (some C++ standard library implementations, though not libstdc++ and libc++, use a separate allocation per element in a deque).

It is intended for the candidate set search queue in #30126, but may be useful as a replacement for std::deque in other places too. It's not a full drop-in replacement, as I did not add iteration support which is unnecessary for the intended use case, but nothing prevents adding that if needed.

Everything is tested through a simulation-based fuzz test that compares the behavior with normal std::deque equivalent operations, both for trivially-copyable/destructible types and others.

@DrahtBot
Copy link
Contributor

DrahtBot commented May 24, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK cbergqvist, hebasto, instagibbs, glozow

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #30126 (Low-level cluster linearization code by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipa sipa force-pushed the 202405_ringbuffer branch from 017272d to c2a4915 Compare May 24, 2024 15:34
@sipa sipa force-pushed the 202405_ringbuffer branch 2 times, most recently from 9c4861a to 315dd68 Compare May 24, 2024 17:31
@sipa sipa changed the title util: add RingBuffer util: add VecDeque May 24, 2024
@sipa sipa force-pushed the 202405_ringbuffer branch from 315dd68 to f1148c5 Compare May 24, 2024 17:35
@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/25390426376

@sipa sipa force-pushed the 202405_ringbuffer branch from f1148c5 to 82258f0 Compare May 25, 2024 23:58
@sipa
Copy link
Member Author

sipa commented May 26, 2024

Apparently std::is_trivially_default_constructible_v<T> does not imply you can just memset 0 to construct the objects (or at least, I can't find evidence of that). So I've dropped that branch.

@hebasto
Copy link
Member

hebasto commented May 27, 2024

Concept ACK.

@hebasto
Copy link
Member

hebasto commented May 27, 2024

It's not a full drop-in replacement...

Then, perhaps, it's a good chance to avoid size_t for parameter and return types in the interface?

See: Signed and Unsigned Types in Interfaces

@sipa
Copy link
Member Author

sipa commented May 27, 2024

@hebasto I do prefer to stay compatible with the std::deque interface, even if just for matching behavior users would expect.

size_t old_pos = m_offset;
for (size_t new_pos = 0; new_pos < m_size; ++new_pos) {
std::construct_at<T>(new_buffer + new_pos, std::move(*(m_buffer + old_pos)));
std::destroy_at<T>(m_buffer + old_pos);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to skip this call if std::is_trivially_destructible_v<T>?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, since that case doesn't let us avoid the loop.

VecDeque& operator=(const VecDeque& other)
{
clear();
Reallocate(other.m_size);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The move assignment operator preserves the capacity. Maybe do the same here for consistency?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to match std::vector behavior here (technically, spec doesn't say anything about the capacity of a copied element, but common implementations make it just big enough to hold the copied data, I believe).

@sipa sipa force-pushed the 202405_ringbuffer branch 2 times, most recently from 5e9abaf to 03cfb86 Compare May 28, 2024 12:55
@sipa sipa force-pushed the 202405_ringbuffer branch from 03cfb86 to a5a27ba Compare May 29, 2024 19:52
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK e4ecb82, I agree with changes since my recent review.

@sipa sipa force-pushed the 202405_ringbuffer branch from e4ecb82 to b8793da Compare June 6, 2024 17:40
@sipa
Copy link
Member Author

sipa commented Jun 6, 2024

I've added some doxygen comments on VecDeque.

@sipa sipa force-pushed the 202405_ringbuffer branch from b8793da to fcdd357 Compare June 6, 2024 17:44
@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 6, 2024

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/25906521916

@sipa sipa force-pushed the 202405_ringbuffer branch from fcdd357 to ee253ca Compare June 6, 2024 18:35
@instagibbs
Copy link
Member

reACK ee253ca

reviewed via git range-diff master e4ecb82 ee253ca

@DrahtBot DrahtBot requested a review from hebasto June 6, 2024 18:52
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK ee253ca, code deduplication using emplace_{back,front} is really nice.

Copy link
Contributor

@cbergqvist cbergqvist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK ee253ca

@DrahtBot DrahtBot removed the CI failed label Jun 6, 2024
sipa added 2 commits June 6, 2024 17:06
This is an STL-like container that interface-wise looks like std::deque, but
is backed by a (fixed size, with vector-like capacity/reserve) circular buffer.
@sipa sipa force-pushed the 202405_ringbuffer branch from ee253ca to 7b8eea0 Compare June 6, 2024 21:06
Copy link
Contributor

@cbergqvist cbergqvist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 7b8eea0

@DrahtBot DrahtBot requested review from hebasto and instagibbs June 6, 2024 22:03
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 7b8eea0, I've verified changes since my recent review with

git range-diff master ee253ca7dea9bed01d4c1800760477ef06310df8 7b8eea067f188c0b0e52ef21b01aedd37667a237

@instagibbs
Copy link
Member

reACK 7b8eea0

reviewed via git range-diff master ee253ca 7b8eea0

Copy link
Member

@glozow glozow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 7b8eea0

@glozow glozow merged commit feab351 into bitcoin:master Jun 7, 2024
fanquake added a commit that referenced this pull request Jul 11, 2024
26a7f70 ci: enable self-assignment clang-tidy check (Cory Fields)
32b1d13 refactor: add self-assign checks to classes which violate the clang-tidy check (Cory Fields)

Pull request description:

  See comment here: #30161 (comment)

  Our code failed these checks in three places, which have been fixed up here. Though these appear to have been harmless, adding the check avoids the copy in the self-assignment case so there should be no downside.

  ~Additionally, minisketch failed the check as well. See bitcoin-core/minisketch#87
  Edit: Done

  After fixing up the violations, turn on the aggressive clang-tidy check.

  Note for reviewers: `git diff -w` makes this trivial to review.

ACKs for top commit:
  hebasto:
    ACK 26a7f70, I have reviewed the code and it looks OK.
  TheCharlatan:
    ACK 26a7f70

Tree-SHA512: 74d8236a1b5a698f2f61c4740c4fc77788b7f882c4b395acc4e6bfef1ec8a4554ea8821a26b14d70cfa6c8e2e9ea305deeea3fbf323967fa19343c007a53c5ba
@hebasto
Copy link
Member

hebasto commented Jul 14, 2024

Ported to the CMake-based build system in hebasto#264.

@bitcoin bitcoin locked and limited conversation to collaborators Jul 14, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants