Skip to content

Conversation

taniabogatsch
Copy link
Contributor

There are currently two known bugs for 16kB blocks causing corrupted compressed storage (roaring bitmap compression and FSST compression). I do not have a reproducer, only the .db files - with these exceptions, we prevent crashes, and we can (ideally) detect corrupted data when its being written, instead of later down the line.

@Tishj, I've also pushed our 'fix':

	auto unaligned_data_size = NumericCast<idx_t>(data_ptr - base_ptr);
	auto data_size = AlignValue(unaligned_data_size);
	data_ptr += data_size - unaligned_data_size;

@duckdb-draftbot duckdb-draftbot marked this pull request as draft July 4, 2025 14:25
@Mytherin Mytherin marked this pull request as ready for review July 5, 2025 08:10
@Mytherin Mytherin merged commit 42f1667 into duckdb:v1.3-ossivalis Jul 5, 2025
51 checks passed
@Mytherin
Copy link
Collaborator

Mytherin commented Jul 5, 2025

Thanks!

@taniabogatsch taniabogatsch deleted the roaring branch July 7, 2025 08:36
krlmlr added a commit to krlmlr/duckdb-r that referenced this pull request Jul 9, 2025
Add internal exceptions to compression paths to prevent segmentation violations (duckdb/duckdb#18151)
krlmlr added a commit to krlmlr/duckdb-r that referenced this pull request Jul 9, 2025
Add internal exceptions to compression paths to prevent segmentation violations (duckdb/duckdb#18151)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants