Block based encryption #17275

ccfelius · 2025-04-28T10:05:49Z

This PR adds block-based encryption. In block-based encryption all blocks - except the main database header - are encrypted. To ensure that encryption is randomised, each block is prefixed with a 12-byte nonce. For the encryption mode AES-GCM the nonce is followed by a 16-byte tag, which is calculated after encrypting the actual data and checksum of the corresponding block. Upon decryption, this tag is calculated and checked against the stored tag (pretty much like how a checksum works).

Encryption Algorithm support
For now we only support AES-GCM with a fixed key length of 32 bytes (aes-gcm-256) in Mbed TLS. This is a randomised encryption algorithm that also performs authentication by means of a calculated tag.

Usage

Encrypt an existing database

ATTACH 'existing_unencrypted.duckdb' as unencrypted;
ATTACH 'encrypted.duckdb' AS encrypted (ENCRYPTION_KEY 'asdf');
COPY FROM DATABASE unencrypted TO encrypted;

Decrypt an existing database

ATTACH 'encrypted.duckdb' AS encrypted (ENCRYPTION_KEY 'asdf');
ATTACH 'new_unencrypted.duckdb' as unencrypted;
COPY FROM DATABASE encrypted TO unencrypted;

Reencrypt an existing database

ATTACH 'existing_encrypted.duckdb' AS existing_encrypted (ENCRYPTION_KEY 'asdf');
ATTACH 'new_encrypted.duckdb' AS new_encrypted (ENCRYPTION_KEY 'xxxx');
COPY FROM DATABASE existing_encrypted TO new_encrypted;

Write to an encrypted database

ATTACH 'encrypted.duckdb' AS encrypted (ENCRYPTION_KEY 'asdf');
CREATE OR REPLACE TABLE encrypted.tbl AS SELECT * FROM range(10) t(i);
SELECT SUM(i) FROM encrypted.tbl;

Performance

The benchmarks are performed on a Mac M2 with 16GB RAM. All queries are performed with TPC-H scale factor 10, unless specified otherwise.

Operation	Time
`COPY FROM DATABASE unencrypted TO new_unencrypted`	10.639 s
`COPY FROM DATABASE unencrypted TO encrypted`	12.688 s
`COPY FROM DATABASE encrypted TO unencrypted`	12.382 s
`COPY FROM DATABASE encrypted TO new_encrypted`	14.528 s
`SELECT SUM(l_suppkey) FROM lineitem` unencrypted	0.067 s
`SELECT SUM(l_suppkey) FROM lineitem` encrypted	0.252 s
(SF1) `SELECT SUM(l_suppkey) FROM lineitem` unencrypted	0.009 s
(SF1) `SELECT SUM(l_suppkey) FROM lineitem` encrypted	0.027s

Encryption incurs significant overhead. With using Mbed TLS, for a simple aggregate in an encrypted file the overhead is around ~3x for TPC-H SF1 Lineitem table, and around ~4x for SF10.

Tests

For all above use-cases, I've added corresponding SQLLogic tests.

Other tests
To ensure that the database is actually encrypted, we can (i) calculate the entropy with ent encrypted.duckdb (ii) search for existing strings that we know, e.g. strings encrypted.duckdb | grep "AIRMAIL", or (iii) inspect the file ourselves with hexdump. For an encrypted database in TPC-H SF10, the entropy is 8 bits per byte, which means that the file is (almost) completely random. Also, existing strings such as AIRMAIL cannot be found. For an unencrypted TPC-H SF10 file the calculated entropy is around 7.7 - which is also pretty high - but this is because of compression.

Internals

Key management
A user can input a key of any length, and this will be reduced or expanded to a 32-byte key. This is done by a key derivation function (kdf). For hashing the password we use SHA256, which we could replace with a more secure kdf. To replace the kdf however we will have to expand the (stripped) Mbed TLS library. The derived key is managed by the EncryptionKeyManager, and the derived key id is stored in the EncryptionOptions. As an extra security measurement, we lock the encryption key with mlock/virtuallock to avoid that the key is ever swapped to disk.

Wrong Key Detection
To distinguish between a checksum failure and a wrong key, we store a canary in the database header. The plaintext of this canary (DUCKKEY) is known, and the canary will be encrypted along with the database. If we then later read an encrypted database, we first decrypt the canary and check whether the decrypted canary value is correct. If the used key is not correct, the database will throw an error.

Metadata
If the database is encrypted, the first flag of the main database header is set to 1. Additionally, we thus store a (i) canary and (ii) 8-bytes of additional metadata. The first byte of the additional metadata indicates the kdf used, the second byte is for the usage of AAD, the third byte indicates the cipher, and the last 4 bytes identify the key_length. This additional metadata is not used yet, but we will need this later when we start supporting multiple kdfs, key lengths and ciphers (such as AES CTR and AES CBC).

carlopi · 2025-04-28T11:38:40Z

src/storage/single_file_block_manager.cpp

+
+	if (flags[0] == MainHeader::ENCRYPTED_DATABASE_FLAG) {
+		ser.WriteData(encryption_metadata, ENCRYPTION_METADATA_LEN);
+		ser.WriteData(encrypted_canary, CANARY_BYTE_SIZE);
+	}
+


I don't think this is desirable, if something needs to be written, it needs either to be after the LibraryVersion/SourceID pair OR it needs to be written as part of the flags (say flags[1] and flags[2]).

This is not strictly needed for duckdb itself, but I think it's handier if there is some consistency on fixed positions where those fields are, say for external tools like file or for past duckdb versions (that can't be aware of this logic).

Thanks for your suggestions! I thought about putting this in the flags but was unsure if we could use them for this purpose. We might want to use them for other purposes in the future.

I'll look into it :)

carlopi · 2025-04-28T11:51:06Z

src/storage/storage_manager.cpp

+		state.AddString("IBd2nLfyDoWYZy6R81DVYxxdM7CAsOcX"); // random salt
+	}


We currently never use the salt, so this is always used. Unsure if that's supposed to stay, if so, I would somehow add some versioning to encryption itself, since changing the default salt needs to be implemented as: add new salt, keeping the old one around, and select which one to use.

Or even a note saying this needs to stay around, and it's not meant to be changed (that default salt is not changed is probably best tested via adding a checked in encrypted duckdb file, and testing it can always be opened).

I can put a salt version in the encryption metadata, since there is still some room for it. Let's say that version 1 is then the fixed salt and version 2 is a uniquely generated salt (which also needs to be stored in the main database header). And the checked encrypted db file is necessary indeed, thanks!

I decided to just generate a new salt for every user key input, it's more secure that way anyways.

Mytherin

Thanks for the PR! Looks good - exciting to get this in. Some comments below.

Mytherin · 2025-05-01T11:20:31Z

src/include/duckdb/storage/single_file_block_manager.hpp

@@ -49,10 +107,17 @@ class SingleFileBlockManager : public BlockManager {

 	FileOpenFlags GetFileFlags(bool create_new) const;
 	//! Creates a new database.
-	void CreateNewDatabase();
+	void CreateNewDatabase(string *encryption_key = nullptr);


nit: we're trying to get rid of raw C-style pointers everywhere - can this be optional_ptr<string> instead?

Mytherin · 2025-05-01T11:21:20Z

src/include/duckdb/storage/storage_info.hpp

@@ -103,12 +121,42 @@ struct MainHeader {
 		return flags[0] == MainHeader::ENCRYPTED_DATABASE_FLAG;
 	}

+	void SetEncryptionMetadata(const data_t *source) {


nit: const_data_ptr_t

Mytherin · 2025-05-01T11:26:58Z

src/storage/single_file_block_manager.cpp

-	// read the buffer from disk
-	block.Read(*handle, location);
+	if (options.encryption_options.encryption_enabled && !skip_block_header) {
+		DecryptBuffer(block, delta);


I think this logic also needs to be added to SingleFileBlockManager::ReadBlocks. Perhaps we can unify ReadAndChecksum / ReadBlocks more so that they can go through the same code-path here?

ReadBlocks is triggered when doing remote reads or when ALTERNATIVE_VERIFY=1 is set. If it is not triggered with alternative verify we're likely not stress testing enough. Can we perhaps add a test that runs TPC-H over an encrypted database (see e.g. test/sql/tpch/tpch_sf1.test_slow).

Totally overlooked this! thanks, will change it :)

Mytherin · 2025-05-01T11:29:30Z

test/sql/copy/encryption/encrypted_to_unencrypted.test_slow

+require skip_reload
+
+# workaround - alternative verify always forces the latest storage
+require no_alternative_verify


I don't quite follow why this require is necessary here - I think this might be hiding some bugs (see comments above). Can we remove it?

Good catch - it stems from the previous block header size PR and never removed it

Mytherin · 2025-05-01T11:32:32Z

third_party/mbedtls/mbedtls_wrapper.cpp

+
+#ifdef DEBUG
+	duckdb::RandomEngine random_engine(1);
+#else
 	duckdb::RandomEngine random_engine(duckdb::Timestamp::GetCurrentTimestamp().value);


Not directly related to this PR - but what is the reason we're seeding explicitly with the current timestamp here? Initializing RandomEngine with no seed seeds it using more cryptographically secure methods when available (SYS_getrandom). That seems like a better behavior here?

I think you are right. The best thing to do is just to remove it then

Mytherin · 2025-05-01T11:35:18Z

src/parser/parsed_data/attach_info.cpp

@@ -7,6 +7,13 @@

 namespace duckdb {

+bool ValidKey(const string &key) {
+	if (key == "true" || key == "false" || key.empty() || key.size() < 4) {


Could you expand on why these requirements are present for keys?

Keys should not be empty, and for some reason empty strings evaluated to true. Key size < 4 is not a strict requirement, but in general smaller user inputs are less secure

I think we should be able to detect the empty key scenario by looking at the type (i.e. ENCRYPTION_KEY passes in a boolean). Instead of calling ToString() on whatever value we have we can throw an error if the encryption key is not a string (or blob?)

Ah yes, that sounds like a better approach

Mytherin · 2025-05-01T11:36:09Z

src/storage/single_file_block_manager.cpp


 #include <algorithm>
 #include <cstring>

+#if defined(_WIN32)
+#include <windows.h>


Can we include windows.h through duckdb/common/windows.hpp instead?

Mytherin · 2025-05-01T11:38:36Z

src/storage/single_file_block_manager.cpp

+#if defined(_WIN32)
+	VirtualLock(const_cast<void *>(static_cast<const void *>(derived_key.data())), derived_key.size());
+#else
+	mlock(derived_key.data(), derived_key.size());


Given we are going through lengths here not to page out the memory of the encryption key - I think we should not use a std::string to store the encryption key. It is very easy to accidentally copy over an std::string which would then negate this protection.

Can we perhaps rewrite the keys to use a separate class that is not copy constructible (and that calls mlock/munlock itself perhaps on memory that it is allocating to hold the key?).

Sounds good!

Mytherin

Thanks for the updates! Looks great - some comments

Mytherin · 2025-06-10T08:59:43Z

src/parser/parsed_data/attach_info.cpp

 			storage_options.block_header_size = DEFAULT_ENCRYPTION_BLOCK_HEADER_SIZE;
 			storage_options.encryption = true;
+			// set storage version to v1.3.0
+			storage_options.storage_version = SerializationCompatibility::FromString("v1.3.0").serialization_version;


We should only set this if it is not specified by the user itself - otherwise the encryption key option overrides the user-provided setting.

i.e. this should throw an error:

ATTACH '...' (STORAGE_VERSION 'v1.1', ENCRYPTION_KEY '...');

And this should use storage version v1.4, not v1.3:

ATTACH '...' (STORAGE_VERSION 'v1.14, ENCRYPTION_KEY '...');

Mytherin · 2025-06-10T09:01:01Z

src/parser/parsed_data/attach_info.cpp

@@ -16,8 +16,23 @@ StorageOptions AttachInfo::GetStorageOptions() const {
 			// even though the corresponding option we expose to the user is called "block_size".
 			storage_options.block_alloc_size = entry.second.GetValue<uint64_t>();
 		} else if (entry.first == "encryption_key") {
+			auto user_key = entry.second.GetValue<string>();


GetValue<string>() does a cast to string. I think we might want to look at the actual type of the value here instead, i.e. enforce it to be VARCHAR or BLOB?

That also means the check below for "true" is not required. We get "true" because we cast a BOOLEAN to string.

It makes sense. Maybe I'm missing someting - but for example, with ATTACH '__TEST_DIR__/encrypted.duckdb' AS encrypted (ENCRYPTION_KEY); this also evaluates to a VARCHAR with value true. So maybe we will need to keep the check for "true"?

Seems to pass in a boolean for me?

$ > build/debug/duckdb -c "ATTACH 'encrypted.duckdb' AS encrypted (ENCRYPTION_KEY);" Binder Error: "true" is not a valid key. A key must be of type VARCHAR

Mytherin · 2025-06-10T09:04:47Z

src/include/duckdb/storage/single_file_block_manager.hpp

 	//! Loads an existing database. We pass the provided block allocation size as a parameter
 	//! to detect inconsistencies with the file header.
-	void LoadExistingDatabase();
+	void LoadExistingDatabase(optional_ptr<string> encryption_key = nullptr);


Should this be a string?

I also wonder - do we need to pass this into these individual methods? Can't we pass it in using the StorageManagerOptions in the constructor?

Mytherin · 2025-06-10T09:07:50Z

src/include/duckdb/storage/single_file_block_manager.hpp

+	//! indicates whether the db is encrypted
+	bool encryption_enabled = false;
+	//! Whether Additional Authenticated Data is used
+	bool aad = false;


Can we call this additional_authenticated_data for clarity?

src/include/duckdb/storage/single_file_block_manager.hpp

Mytherin · 2025-06-10T09:10:07Z

src/common/file_buffer.cpp

@@ -100,6 +100,10 @@ void FileBuffer::Resize(BlockManager &block_manager) {
 	ResizeInternal(block_manager.GetBlockSize(), block_manager.GetBlockHeaderSize());
 }

+void FileBuffer::Resize(const BlockManager &block_manager) {
+	ResizeInternal(block_manager.GetBlockSize(), block_manager.GetBlockHeaderSize());


nit: I think we can just remove the method above with the non-const BlockManager now

Mytherin · 2025-06-10T09:10:49Z

src/include/duckdb/storage/storage_options.hpp

+	//! Encryption algorithm (default = GCM)
+	string encryption_cipher = "gcm";
+	//! encryption key
+	string encryption_key;


Similar - should this be a string?

Yes, this in on purpose. This is not the actual key, but this is the user input, which can be of any length and is almost instantly deleted because another 32-byte key is derived (which is locked and stored in bytes in an EncryptionKey object).

Since this an input key, and is only around for a short period, its memory is also not locked. This value in the StorageOptions is basically just used to pass on the input key to the single file block manager, until we are able to generate or extract the salt from the main database header which we use to derive the final key. This string is then immediately wiped.

delete temp test file

Mytherin

Thanks! One last comment then this is good to go

Mytherin · 2025-06-16T11:47:25Z

src/include/duckdb/storage/storage_options.hpp

+	//! Encryption algorithm (default = GCM)
+	string encryption_cipher = "gcm";
+	//! encryption key
+	string user_key;


Can we make this a shared_ptr, with maybe a fixme to change to a unique_ptr in the future?

Mytherin · 2025-06-17T06:54:28Z

Thanks!

Block based encryption (duckdb/duckdb#17275) Do not dispatch JDBC/ODBC jobs in release CI runs (duckdb/duckdb#17937)

Block based encryption (duckdb/duckdb#17275) Do not dispatch JDBC/ODBC jobs in release CI runs (duckdb/duckdb#17937) Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

bump iceberg to latest main [chore] Fix amalgamation build in progress_bar (duckdb/duckdb#18910) Bump inet & aws (duckdb/duckdb#18899) fix: refine query ETA display and Kalman filter stability (duckdb/duckdb#18880) Bump httpfs to v1.4-andium branch (duckdb/duckdb#18898) Encryption now encoded as a bit, centralizing in set/getter (duckdb/duckdb#18897) Add callback for when an extension fails to load, and also log this (duckdb/duckdb#18894) Keep base data scan state alive in ColumnData::Update call (duckdb/duckdb#18893) Expected errors 2053 (duckdb/duckdb#18892) fixing auto-specifying ciphers and remove double storage (duckdb/duckdb#18891) Add rowsort to upsert_default.test (duckdb/duckdb#18890) bump aws and iceberg (duckdb/duckdb#18889) [chore] Bump config test/configs/compressed_in_memory.json to new format (duckdb/duckdb#18888) [Dev] Fix footgun in `string_t::SetSizeAndFinalize` (duckdb/duckdb#18885) Json: no reinterpret<size_t*> (duckdb/duckdb#18886) [C API] Result schema of prepared statements (duckdb/duckdb#18779) Add `COPY (FORMAT BLOB)` to Andium too :^) (duckdb/duckdb#18884) Avoid automatically checkpointing if the database instance has been invalidated (duckdb/duckdb#18881) Update spatial+vss+sqlsmith in preparation for v1.4 (duckdb/duckdb#18882) Internal duckdb/duckdb#5796: Window Progress (duckdb/duckdb#18860) [Test] Small fixes to concurrent attach/detach test (duckdb/duckdb#18862) Update ducdkb iceberg hash (duckdb/duckdb#18873) Storage fuzzing + several fixes (duckdb/duckdb#18876) Bump mbedtls to v3.6.4 (duckdb/duckdb#18871) [minor] Incompatible DB error message: add newline (duckdb/duckdb#18861) Bump & remove patches for delta, avro, excel, encodings, fts (duckdb/duckdb#18869) Add a FORCE_DEBUG flag to force `-DDEBUG`, similar to FORCE_ASSERT (duckdb/duckdb#18872) Expected errors 2053 (duckdb/duckdb#18864) update duckdb azure extension ref for 1.4.0 (duckdb/duckdb#18868) Hold segment lock during GetColumnSegmentInfo (duckdb/duckdb#18859) Centralize attached database paths in a DatabaseFilePathManager which is shared across databases created through the same DBInstanceCache (duckdb/duckdb#18857) Add more encryption modes CTR and CBC (duckdb/duckdb#18619) Bump Ducklake (duckdb/duckdb#18825) No more `wal_encryption` flag (duckdb/duckdb#18851) Fix `NULL` path for `json_each`/`json_tree` (duckdb/duckdb#18852) Make ATTACH OR REPLACE atomic, keep list of used databases in MetaTransaction (duckdb/duckdb#18850) WAL <> DB File Match Fixes (duckdb/duckdb#18849) Add test_env to unit tester (duckdb/duckdb#18847) Merge ossivalis into main (duckdb/duckdb#18844) Bump MySQL/Postgres/SQLite (duckdb/duckdb#18848) Add OnBeginExtensionLoad callback (duckdb/duckdb#18842) Ignore null verification for statistics on structs (duckdb/duckdb#18813) Document storage version flag in CLI + minor rendering fix (duckdb/duckdb#18841) Add the `VARIANT` LogicalType (duckdb/duckdb#18609) Avoid printing '99 hours', given in most cases that means estimate is… (duckdb/duckdb#18839) Don't notify Py pkg when override git describe is set (duckdb/duckdb#18843) Add support for reading/writing native parquet geometry types (duckdb/duckdb#18832) Fix/run function in transaction (duckdb/duckdb#18741) Avoid expensive checkpoints and write amplification by appending row groups, and limiting vacuum operations for the last number of row groups (duckdb/duckdb#18829) fix: silence warnings about signed/unsigned conversions. (duckdb/duckdb#18835) fix: sanitize input for enable_logging (duckdb/duckdb#18830) Ensure a WAL file matches the DB file and checkpoint iteration (duckdb/duckdb#18823) Fix: Preserve database configuration flags for tab completion in DuckDB shell (duckdb/duckdb#18482) Extensions.yml: Pass down save_cache to inner workflows (duckdb/duckdb#18828) Fix format-fix runs on Linux (duckdb/duckdb#18827) Re-add accidentally removed check if copy_from is supported (duckdb/duckdb#18824) [Fix] Bug in fixed-size buffer when throwing out-of-memory (duckdb/duckdb#18769) For BC reasons - keep VARINT as alias for BIGNUM (duckdb/duckdb#18821) Add callback to get a list of copy options, use this to provide suggestions and to erase options from import that are only used during exporting (duckdb/duckdb#18812) fix: Add COLLATE NOCASE support to strpos function (duckdb/duckdb#18819) [CI] install libcurl4-openssl-dev with apt-get (duckdb/duckdb#18811) Provide failing file name in Parquet reader error messages (duckdb/duckdb#18814) Test runner: Expand '{UUID}' into a random UUID (duckdb/duckdb#18809) Expected errors 2053 (duckdb/duckdb#18810) Add support for non-aggregate window functions (duckdb/duckdb#18788) Fix some unindented interactions between `EMPTY_RESULT_PULLUP` and `MATERIALIZED` CTEs (duckdb/duckdb#18805) Typed macro parameters (duckdb/duckdb#18786) Internal duckdb/duckdb#3273: Hashed Sort Callbacks (duckdb/duckdb#18796) [chore] Fixup tidy-check on src/logging/log_manager.cpp by passing const & (duckdb/duckdb#18801) Fixup progress_bar: avoid converting doubles into int32_t unchecked (duckdb/duckdb#18800) Issue duckdb/duckdb#18767: Ignore Timestamp Offsets (duckdb/duckdb#18794) bump httpfs so it includes curl option (duckdb/duckdb#18691) Improve autocomplete suggestions (duckdb/duckdb#18773) Remove everything python-package related (duckdb/duckdb#18789) Support expressions as COPY file target (duckdb/duckdb#18795) fix: coalesce query progress updates to reduce terminal writes (duckdb/duckdb#18672) Task Scheduler: track exact task count, and re-signal on dequeue failure if there are tasks left (duckdb/duckdb#18792) Internal duckdb/duckdb#3273: Parallel Window Masks (duckdb/duckdb#18731) fix: improve speed of GetValue() for STRUCT type (duckdb/duckdb#18785) Add `memory_limit` parameter to `benchmark_runner`/`test_runner.py` (duckdb/duckdb#18790) Treat ENABLE_EXTENSION_AUTOINSTALL as the BOOL that it is (duckdb/duckdb#18778) Move row id logic to separate RowIdColumnData class instead of inlining it into the RowGroup (duckdb/duckdb#18780) Improve error messages for merge / vector reference (duckdb/duckdb#18777) Use microsecond resolution for printing the current timestamp (duckdb/duckdb#18776) Add `file_size_bytes` (de-)serialization (duckdb/duckdb#18775) Propagate `DUCKDB_*_VERSION` in extensions and tests (duckdb/duckdb#18774) [Test Fix] Forward output to file (duckdb/duckdb#18772) [CI] Adjust test configs post logger PR (duckdb/duckdb#18771) Revert "Use 1-based indexing for SQL-based JSON array extraction" (duckdb/duckdb#18758) Make `duckdb_log` return a TIMESTAMP_TZ (duckdb/duckdb#18768) Fix Path Typo in Extension's CMake Warning Message (duckdb/duckdb#18766) Fix index resolution when querying table with index via view (duckdb/duckdb#18319) Fix radix partitioning with more than 10 bits (duckdb/duckdb#18761) Add support for auto-globbing within a directory: if no matches are found for a specific path, we retry with `/**/*.[ext]` appended (duckdb/duckdb#18760) Refactor read_blob and read_text to use MultiFileFunction. (duckdb/duckdb#18706) Add missing expected errors to the test cases (next chunk) (duckdb/duckdb#18753) Minor logging fixes and more benchmarking (duckdb/duckdb#18755) Extensions.yml should also check converted_to_draft (duckdb/duckdb#18754) [Profiling] Add Profiling to Write Function (duckdb/duckdb#18724) Fixing lazy polars execution on query result (duckdb/duckdb#18749) Remove separate WAL encryption flag (duckdb/duckdb#18750) Add leak suppressions to nightly runs (duckdb/duckdb#18748) Append using a SQL query, instead of directly appending to a base table, and support user-provided queries through the QueryAppender (duckdb/duckdb#18738) removed placeholder client directories for node and jdbc, its been > 1 yr (duckdb/duckdb#18757) Add missing expected errors to the test cases (duckdb/duckdb#18746) Add OS X notarization for DuckDB CLI and libduckdb.dylib (duckdb/duckdb#18747) Use correct type for pushing collations in subqueries (duckdb/duckdb#18744) Merge ossivalis into main (duckdb/duckdb#18719) Secrets: if serialization_type is not specified, assume it's a key value secret (duckdb/duckdb#18743) [C API] Function to set a copy callback for bind data (duckdb/duckdb#18739) fix timetravel for default tables (duckdb/duckdb#18240) [unittest] SkipLoggingSameError() to make unittester report one failure per case (duckdb/duckdb#18270) Use 1-based indexing for SQL-based JSON array extraction (duckdb/duckdb#18735) Add (CSV) file logger (duckdb/duckdb#17692) feat: enhance .tables command with schema disambiguation and filtering (duckdb/duckdb#18641) Internal duckdb/duckdb#5669: Loop Join Thresholds (duckdb/duckdb#18733) Fix PIVOT in multiple statements (duckdb/duckdb#18729) Minor fixes for other catalogs - mostly checking `IsDuckTable()` for unsupported operations (duckdb/duckdb#18720) Added support for blob<->uuid conversions (duckdb/duckdb#18027) #Fix 18558: add row_group scan fast path (duckdb/duckdb#18686) Improved grammar generation script (duckdb/duckdb#18716) Correctly throw an error when too few columns are supplied in MERGE INTO INSERT (duckdb/duckdb#18715) [Profiling] Add Profiling to Read Function (duckdb/duckdb#18661) Fix issue with materialized CTE optimization in flatten_dependent_join (duckdb/duckdb#18714) Add Option to Allocate Using an Arena in `string_t` (duckdb/duckdb#17992) Internal duckdb/duckdb#3273: Hashed Sort States (duckdb/duckdb#18690) Python-style positional/named arguments for macro's (duckdb/duckdb#18684) [Fix] Correctly handle table and index chunks in WAL replay buffering (duckdb/duckdb#18700) Make ART construction iterative via ARTBuilder (duckdb/duckdb#18702) Correctly handle collations for IN (subquery) (duckdb/duckdb#18698) Hold row group lock for entire call of MoveToCollection (duckdb/duckdb#18694) Expected errors 2053 (duckdb/duckdb#18695) Issue duckdb/duckdb#18457: DateTrunc Simplification Warnings (duckdb/duckdb#18687) [Python SQLLogicTest] Add `test/sql/pragma/profiling/test_profiling_all.test` to the SKIPPED_TESTS set (duckdb/duckdb#18689) Make sure parse errors are wrapped in ErrorData (duckdb/duckdb#18682) Internal duckdb/duckdb#5366: Window State Arguments (duckdb/duckdb#18676) Expected errors 2053 (duckdb/duckdb#14213) Add `date_trunc()` simplification rules (duckdb/duckdb#18457) Fix the issue where delta_for isn't used in bitpacking when for is unavailable (duckdb/duckdb#18616) fix error message related to wrong memory unit (duckdb/duckdb#18671) Grab lock and double-check that column is not loaded in MoveToCollection (duckdb/duckdb#18677) Correctly allocate uncompressed string data in ZSTD for many giant strings (duckdb/duckdb#18678) Internal duckdb/duckdb#5662: IEJoin Test Plans (duckdb/duckdb#18680) [ Python SQLLogic Tester ] Add `MERGE_INTO` to `statement.type` enum in `result.py` (duckdb/duckdb#18675) Internal duckdb/duckdb#5366: Window Interrupt Arguments (duckdb/duckdb#18651) Correctly set weights in reservoir sample when switch to slow sampling (duckdb/duckdb#18563) [Dev] Add script to create patch from changes in an extension repository (duckdb/duckdb#18620) Python test runner: Fix hash comparison error output (duckdb/duckdb#18626) [CI] skip building encodings extension in InvokeCI (duckdb/duckdb#18655) CLI: Make ETA more of an estimate, and support large_row_rendering for footers (duckdb/duckdb#18656) Merge ossivalis into main (duckdb/duckdb#18644) Python test runner: Fix result check for `COPY ... RETURN_STATS` queries (duckdb/duckdb#18625) Add 1.4 release codename (duckdb/duckdb#18652) Change arrow() to export record batch reader (duckdb/duckdb#18642) bump spatial (on main) (duckdb/duckdb#18197) bump avro to v1.4 (duckdb/duckdb#18434) Make more configs into generic settings (duckdb/duckdb#18592) Add "Hash Zero" verification CI run (duckdb/duckdb#18623) feat: add ETA to progress bar in DuckDB CLI (duckdb/duckdb#18575) wrap httplib ::max() call in `WIN_32` check (duckdb/duckdb#18590) [ART] ART::Erase refactoring (duckdb/duckdb#18595) [CSV Sniffer] Fix type detection issue with union and empty columns (duckdb/duckdb#18606) Add Field IDS to multi file reader for positional deletes (duckdb/duckdb#18617) Re-add `hugeint` to `__internal_compress_string` (duckdb/duckdb#18622) Adjust filter pushdown to latest polars release (duckdb/duckdb#18624) parquet/parquet_multi_file_info.cpp: fix move from stack (duckdb/duckdb#18634) Issue duckdb/duckdb#18631: Streaming Windowed Quantile (duckdb/duckdb#18636) Fix serialization backwards compatability for varargs functions (duckdb/duckdb#18596) [Profiling] Add client context into more read functions (duckdb/duckdb#18514) [CI] Don't zip and upload Code Coverage tests results when Code Coverage got cancelled (duckdb/duckdb#18607) [Test] Fix test case and a benchmark (duckdb/duckdb#18610) Update README.md (duckdb/duckdb#18614) correctly setting log transaction id in ThreadContext (duckdb/duckdb#18536) [Fix] Hidden test failure in test_struct_update.test (duckdb/duckdb#18598) Increment storage version to enable `DICT_FSST` in benchmark file (duckdb/duckdb#18588) fix hidden merge conflict (duckdb/duckdb#18589) Adds a function for updating and adding values in a struct (duckdb/duckdb#15533) Pushdown filters on coalesced outer join keys compared for equality under the join condition (duckdb/duckdb#18169) fix: libduckdb.so missing soversion (duckdb/duckdb#18305) String dictionary hash cache (duckdb/duckdb#18580) Force `LIST`/`ARRAY` child vectors on a Parquet single page (duckdb/duckdb#18578) fix: use thousands separator and decimal for row counts in`duckbox` output format (duckdb/duckdb#18564) Flip left/right delim join based on cardinalities (duckdb/duckdb#18552) [Fix] Adjust shrink threshold back to original count > SHRINK_THRESHOLD (duckdb/duckdb#18582) [CSV Sniffer] Fixing bug of not properly setting skipped rows from sniffer (duckdb/duckdb#18555) fix: add formatting to explain row counts (duckdb/duckdb#18566) Delete FUNDING.json Update FUNDING.json Create FUNDING.json [Indexes] Buffer-managed indexes part 3: segment handle for Node48 and Node256 (duckdb/duckdb#18567) Rename the Varint type to Bignum (duckdb/duckdb#18547) Add compile option standalone-debug for clang (duckdb/duckdb#17433) Fixing compilation with -std=cpp23 (duckdb/duckdb#18557) [easy] [no-op] Minor optimization on iterator lookup (duckdb/duckdb#15349) optimize/parquet: generate movable types for parquet (duckdb/duckdb#18510) Check if `heap_block_ids` is empty before getting start/end when destroying chunks in `TupleDataCollection` (duckdb/duckdb#18556) Implement special-case `VARCHAR` to `JSON[]` casts and vice versa (duckdb/duckdb#18541) [ART] Node::Free refactoring (duckdb/duckdb#18544) [Fix] Follow-up PR to only delete unique row IDs (duckdb/duckdb#18545) Restore missing `test/configs/small_block_size.json` file (duckdb/duckdb#18507) Unittester: Add the `--sort-style` parameter that allows a fallback comparison where results are sorted according to a given sort-style (duckdb/duckdb#18542) Allow overriding openssl version for FIPS compliance (duckdb/duckdb#18499) fix: improve handling variant nulls and nested types (duckdb/duckdb#18538) Add support for explicit clean-up routine in test config, and exit multi-statement execution when an error is encountered (duckdb/duckdb#18539) Use global index, not local id when creating filters in `MultiFileColumnMapper` (duckdb/duckdb#18537) Add `StatementVerifier` for `EXPLAIN` (duckdb/duckdb#18529) Add CAPI to retrieve client context for table functions (duckdb/duckdb#18520) fix: support both field orders for variant struct (duckdb/duckdb#18532) [Varint] Negation, Subtraction and Over/under-flow checking (duckdb/duckdb#18477) ALP test: skip TPC-DS 67 - it is not consistent with floating point numbers (duckdb/duckdb#18528) Consistently detect JSON schema indepent of number of threads (duckdb/duckdb#18522) Internal duckdb/duckdb#16560: Numeric TRUNC Precision (duckdb/duckdb#18511) Dynamically determine dictionary size limit in Parquet writer (if unset) (duckdb/duckdb#18356) Fix incorrect character encoding in GetLastErrorAsString on Windows (duckdb/duckdb#18431) Fix: Write the salt together with the HT offset when determining the value for key comparison (duckdb/duckdb#18374) When tracking evicted_data_per_tag, track actual size on disk after temp file compression (duckdb/duckdb#18521) Adding WITH ORDINALITY to DuckDB (duckdb/duckdb#16581) ParserException for Pragma with named parameters (duckdb/duckdb#18506) Temporarily excluding `Build Pyodide wheel` for Python 3.11 because it fails to build `WASM` wheels (duckdb/duckdb#18508) Remove `immediate_transaction_mode` from DB config options (duckdb/duckdb#18516) Allow expressions to be used in ATTACH / COPY options (duckdb/duckdb#18515) Fix several bugs/fuzzer issues (duckdb/duckdb#18503) Fix: Remove overly strict assertion on empty string value (duckdb/duckdb#18504) Change UNICODE to UTF8 (duckdb/duckdb#17586) Merge ossivalis (duckdb/duckdb#18502) fix: add missing space in AttachInfo::ToString() (duckdb/duckdb#18500) julia: config improvements (duckdb/duckdb#17585) [Profiling] Add client context into read functions (duckdb/duckdb#18438) Fix accidental internal exception in type transformation (duckdb/duckdb#18492) add delta linux back to ci (duckdb/duckdb#18491) Change ctrl-a/ctrl-e to move to start/end of line, not buffer (duckdb/duckdb#18490) Unify `ON CONFLICT` and `MERGE INTO` (duckdb/duckdb#18480) Internal duckdb/duckdb#5384: Window Sorting Polish (duckdb/duckdb#18484) re-nable extensions in invokeci (duckdb/duckdb#18476) SUM and + Operator for Varints (duckdb/duckdb#18424) Internal duckdb/duckdb#5366: WindowDeltaScanner (duckdb/duckdb#18468) Merge ossivalis (duckdb/duckdb#18456) Bump postgres to latest main (duckdb/duckdb#18464) Internal duckdb/duckdb#5385: WindowMergeSortTree Sort Update (duckdb/duckdb#18461) Add support for generic settings, and move many settings over to generic settings (duckdb/duckdb#18447) Buffer index appends during WAL replay (duckdb/duckdb#18313) Internal duckdb/duckdb#5384: WindowDistinctAggregator Sort Update (duckdb/duckdb#18442) Add support for "template" types (duckdb/duckdb#18410) Update pyodide build to 0.28.0 (duckdb/duckdb#18446) Parquet: add row-group ordinal during writing encryption (duckdb/duckdb#18433) Include pyodide build configuration (duckdb/duckdb#18183) [Fix] Block size nightly (duckdb/duckdb#18425) Internal duckdb/duckdb#5368: WindowNaiveAggregator Sort Update (duckdb/duckdb#18409) Internal duckdb/duckdb#5367: SortedAggregateFunction Sort Update (duckdb/duckdb#18408) Refactor extension CI to use extension-ci-tools (duckdb/duckdb#18361) Correctly fetch only base column data in ColumnData::FetchUpdateData (duckdb/duckdb#18423) feat: remove anything following `?` in database name (duckdb/duckdb#18417) Merge `v1.3-ossivalis` in `main` (duckdb/duckdb#18401) Add support for table_constraints of AdbcConnectionGetObjects() (duckdb/duckdb#18181) Add DuckLake back in (duckdb/duckdb#18405) Internal duckdb/duckdb#5294: TIME_NS C API (duckdb/duckdb#18215) Remove incorrect assertion (duckdb/duckdb#18404) [ Python SQLLogic Tester ] Add `MERGE_INTO` statement to duckdb python (duckdb/duckdb#18402) CI: Add separate job for discussion mirroring (duckdb/duckdb#18407) Wrap runner.ExecuteFile, otherwise cleanup is not properly performed (duckdb/duckdb#18400) Internal duckdb/duckdb#3273: Window Hashed Sort (duckdb/duckdb#18337) Store extra metadata blocks in RowGroupPointer, and only flush dirty Metadata blocks (duckdb/duckdb#18398) CI: Fix Discussion mirroring (duckdb/duckdb#18397) Record whether or not cross products are implicit or not, and use this for converting queries back to SQL (duckdb/duckdb#18394) Correct and consistent integer arithmetic error messages (duckdb/duckdb#18393) Re-use metadata of unaltered row groups when checkpointing a table (duckdb/duckdb#18395) Approx database count system function (duckdb/duckdb#18392) Re-use table metadata when table is not altered during checkpoint (duckdb/duckdb#18390) Bump httpfs (duckdb/duckdb#18388) Uncomment skipped decimal REE tests (duckdb/duckdb#18372) Re-enable but deprecate CORE_EXTENSIONS in CMakeLists.txt (duckdb/duckdb#18377) Add missing ninja to workflow file (duckdb/duckdb#18373) Merge `v1.3-ossivalis` into `main` (duckdb/duckdb#18364) Pass `AttachOptions` to `attach` method, and turn `StorageExtensionInfo` into an `optional_ptr` (duckdb/duckdb#18368) Split up out-of-tree extensions into separate files, and allow out-of-tree extensions to be built using BUILD_EXTENSIONS={ext_name} (duckdb/duckdb#18357) Python external dispatch param fixes (duckdb/duckdb#18359) Revert "[unittest] - fix doubled error headers on `Unexpected failure`" (duckdb/duckdb#18355) Add support for checkpointing in-memory tables (duckdb/duckdb#18348) [C API] Expose expressions and use them in scalar function binding (duckdb/duckdb#18142) Extend PEG parser grammar (duckdb/duckdb#18221) [unittest] - fix doubled error headers on `Unexpected failure` (duckdb/duckdb#18314) Fix condition indexes in join filter pushdown (duckdb/duckdb#18341) download Real Nest data in quiet mode (duckdb/duckdb#18346) Fix debug error in join order optimizer (duckdb/duckdb#18344) Aarch64 backport (duckdb/duckdb#18345) add the from-table-function as parameter to copy-from-bind (duckdb/duckdb#18004) feat: making Parquet write RowGroup.total_compressed_size (duckdb/duckdb#18307) Make storage-version a test parameter (duckdb/duckdb#18324) New Arrow C-API (duckdb/duckdb#18246) feat: Parquet extension add row_group_compressed_size (duckdb/duckdb#18294) Merge ossivalis into main (duckdb/duckdb#18272) SHOW TABLES FROM <qualified_name> (duckdb/duckdb#18179) Add target for installing Python deps. (duckdb/duckdb#18285) Use `FromEpochSeconds` instead of `FromTimeT` in `FileSystem::GetLastModifiedTime` (duckdb/duckdb#18281) [Fix] Adjust test to run with different block sizes (duckdb/duckdb#18277) Use DuckDB cast infrastructure in fmt for new uhugeint/hugeint code (duckdb/duckdb#18275) Use set for row ID scanning during index scans (duckdb/duckdb#18274) Add support for RETURNING to MERGE INTO (duckdb/duckdb#18271) Support HUGEINT in printf and format (duckdb/duckdb#13277) Expanded autocomplete suggestions (duckdb/duckdb#18243) [Parquet] Add read support for the `VARIANT` LogicalType (with shredded encoding) (duckdb/duckdb#18224) Reduce copy in Vector::Reinterpret (duckdb/duckdb#18264) Fixes for gcc 15 (duckdb/duckdb#18261) Fix dictionary-related assertions (duckdb/duckdb#18260) Allow for static libs from extension dependencies to be bundled (duckdb/duckdb#18226) disable WebAssembly duckdb-wasm builds job in NightlyTests triggered by 'workflow_dispatch' event (duckdb/duckdb#18129) Bunch of loosely connected test/CI fixes (duckdb/duckdb#18254) update run_extension_medata_tests.sh (duckdb/duckdb#17976) fixes for some minor llvm 20 complaints (duckdb/duckdb#18257) Fix integer overflow in sequence vector (duckdb/duckdb#18245) Add type safety to `FlatVector::GetData<T>`, `ConstantVector::GetData<T>` and `UnifiedVectorFormat::GetData<T>` (duckdb/duckdb#18256) Slightly higher memory limit for test (duckdb/duckdb#18235) Improve descriptions of thresholds vars affecting join algorithm selection (duckdb/duckdb#17377) Add support for geoarrow encoded geometries in geoparquet files. (duckdb/duckdb#17942) Dictionary functions (duckdb/duckdb#18127) Better `NULL` handling in `TupleDataLayout` (duckdb/duckdb#18069) Track `DataChunk` memory usage in various places (duckdb/duckdb#18191) [Parquet] Add read support for the `VARIANT` LogicalType (duckdb/duckdb#18187) Bugduckdb/duckdb#18163 Fix STDDEV_SAMP undeterminism (duckdb/duckdb#18210) Internal duckdb/duckdb#5264: NLJ Not Distinct (duckdb/duckdb#18216) Improve Parquet reader `NULL` statistics and compress all-`NULL` columns using `CompressedMaterialization` (duckdb/duckdb#18217) Get type of encoded `SortKey` from `TupleDataLayout` (duckdb/duckdb#18218) ci(pyodide): enable WASM exceptions on the latest pyodide build (duckdb/duckdb#18173) Temporary file encryption (duckdb/duckdb#18208) More internal-linkage (duckdb/duckdb#18177) Two-rowID-leaf support in the conflict manager and general refactoring (duckdb/duckdb#18194) [Parquet][Dev] Update the vendored `parquet.thrift` to `3ce0760` (duckdb/duckdb#18195) Parquet reader logging (duckdb/duckdb#18172) Merge `v1.3-ossivalis` into `main` (duckdb/duckdb#18188) [Profiling] Move the client context into more write functions (duckdb/duckdb#17875) Check if `GetLastSegment` is not `nullptr` in `ColumnData::RevertAppend` (duckdb/duckdb#18171) Reduce lock contention for the instance cache (duckdb/duckdb#18079) fix bug with allowed_paths (duckdb/duckdb#18176) Avoid `realloc` in CSV writer (duckdb/duckdb#18174) fix typo (duckdb/duckdb#18165) Resolve some small build issues (duckdb/duckdb#18162) Implement `replace_type` function (duckdb/duckdb#18077) Issue duckdb/duckdb#17683: TIME_NS Compilation (duckdb/duckdb#18053) Add support for AdbcConnectionGetObjects(table_type) (duckdb/duckdb#18066) Detect when updates have no effect, and skip performing the actual updates if we encounter these nop updates (duckdb/duckdb#18144) Add support for `MERGE INTO` (duckdb/duckdb#18135) Improve sort key comparison performance (duckdb/duckdb#18131) set ::error:: annotations for test runners (duckdb/duckdb#18072) Internal duckdb/duckdb#3273: Window Task Generation (duckdb/duckdb#18113) Update description of 'arrow_lossless_conversion' (duckdb/duckdb#18046) [chore] Merge v1.3-ossivalis on main (duckdb/duckdb#18109) ci: build duckdb against the latest emscripten (duckdb/duckdb#18110) Don't throw `InternalException` in `Sort::Sink` (duckdb/duckdb#18105) TPC-DS: Use BIGINT fields (duckdb/duckdb#18098) [CI] don't run jobs on draft PRs (duckdb/duckdb#18016) Fix correlated subquery unnest fail (duckdb/duckdb#18092) [CSV Reader] Prohibit options delim and sep in same read_csv call (duckdb/duckdb#18096) Add start/end offset percentage options to Python test runner (duckdb/duckdb#18091) Avoid running DraftPR.yml until timeout if token is missing (duckdb/duckdb#18090) Unittest: Configure skip error messages (duckdb/duckdb#18087) Switch to Optional for type hints in polars lazy dataframe function (duckdb/duckdb#18078) Issue duckdb/duckdb#18071: Temporal inf -inf (duckdb/duckdb#18083) Fix some scaling issues (duckdb/duckdb#17985) Unittester: add `on_new_connection` + `on_load` + `skip_tests` options (duckdb/duckdb#18042) Use `timestamp_t` instead of `time_t` for file last modified time (duckdb/duckdb#18037) Add support for class-based expression iteration (duckdb/duckdb#18070) fix star expr exclude error (duckdb/duckdb#18063) Adding WAL encryption (duckdb/duckdb#17955) Avoid adding commands read from a file to the shell history (duckdb/duckdb#18057) Remove match-case statements from polars_io.py (duckdb/duckdb#18052) Merge ossivalis into main (duckdb/duckdb#18036) Add ppc64le spin-wait instruction (duckdb/duckdb#17837) Unittest: Add skip_compiled option that can be used to skip built-in C++ tests (duckdb/duckdb#18034) [Explain] Add the YAML format for EXPLAIN statements (duckdb/duckdb#17572) Remove Linux (32 Bit) job (duckdb/duckdb#18012) [Chore] Minor conflict manager refactoring (duckdb/duckdb#18015) Fix duckdb/duckdb#18007: correctly execute expressions with pivot operator (duckdb/duckdb#18020) c-api to copy vector with selection (duckdb/duckdb#17870) Add support to produce Polars Lazy Dataframes (duckdb/duckdb#17947) Implement consumption and production of Arrow Binary View (duckdb/duckdb#17975) Rework extension loading to go through thread-safe ExtensionManager (duckdb/duckdb#17994) Issue duckdb/duckdb#5123: make_timestamp_ms (duckdb/duckdb#17908) Flag to disable database invalidation (duckdb/duckdb#17938) [Fix] Reset profiling info before preparing a query (duckdb/duckdb#17940) Issue duckdb/duckdb#5144: AsOf Join Threshold (duckdb/duckdb#17979) [CI] Skip some workflows when updating out of tree extensions SHA (duckdb/duckdb#17949) Merge v1.3-ossivalis into main (duckdb/duckdb#17973) [nested] Allow fixed-size arrays to be unnested (duckdb/duckdb#17968) Unit Tester Configuration (duckdb/duckdb#17972) [Nested] Optimize structs in `LIST_VALUE` (duckdb/duckdb#17169) Enable building spatial and encodings extensions (duckdb/duckdb#17960) [Nested] Add `struct_position` and `struct_contains` functions (duckdb/duckdb#17819) Visual Studio 17 (2022) fixes (duckdb/duckdb#17948) [CI Nightly Fix] Skip logging test if not standard block size (duckdb/duckdb#17957) Add v1.3-ossivalis to Cross version workflow (duckdb/duckdb#17906) Unittester failures summary (duckdb/duckdb#16833) Block based encryption (duckdb/duckdb#17275) Do not dispatch JDBC/ODBC jobs in release CI runs (duckdb/duckdb#17937) fix use after free in adbc on invalid stmt (duckdb/duckdb#17927) Fix empty BP block when writing parquet (duckdb/duckdb#17929) Leverage `VectorType` in `ColumnDataCollection` (duckdb/duckdb#17881) Merge v1.3 into main (duckdb/duckdb#17897) Make CTE Materialization the Default Instead of Inlining (duckdb/duckdb#17459) Use an arena linked list for the physical operator children (duckdb/duckdb#17748) Reword GenAI policy (duckdb/duckdb#17895) Issue duckdb/duckdb#17861: FILL Argument Types (duckdb/duckdb#17888) Update function descriptions and examples for list, array, lambda functions (duckdb/duckdb#17886) Add GenAI policy (duckdb/duckdb#17882) Issue duckdb/duckdb#17849: Test FILL Duplicates (duckdb/duckdb#17869) Add STRUCT to MAP cast function (duckdb/duckdb#17799) Issue duckdb/duckdb#17040: FILL Secondary Sorts (duckdb/duckdb#17821) Issue duckdb/duckdb#17153: Window Order Columns (duckdb/duckdb#17835) julia: add missing methods from C-API (duckdb/duckdb#17733) Function Serialization: adapt to removal of overloads by explicitly casting if argument types have changed (duckdb/duckdb#17864) [Indexes] Buffer-managed indexes part 2: segment handle for base nodes (duckdb/duckdb#17828) duckdb/duckdb#17853 Enable flexible page sizes and update Android NDK to r27 in workflow. (duckdb/duckdb#17854) Internal duckdb/duckdb#4991: Remove Epoch_MS(MS) (duckdb/duckdb#17816) Add `duckdb_type` column to parquet_schema (duckdb/duckdb#17852) Merge v1.3 into main (duckdb/duckdb#17851) Fix ICE with Windows ARM64 (duckdb/duckdb#17844) fix: escape using_columns on JoinRef::ToString (duckdb/duckdb#17839) Merge130 (duckdb/duckdb#17833) Replace string for const data ptr in encryption api (duckdb/duckdb#17825) Pushdown pivot filter (duckdb/duckdb#17801) Merge v1.3 into main (duckdb/duckdb#17806) Add qualified parameter to Python GetTableNames API (duckdb/duckdb#17797) Fix propagatesNullValues for case expr (duckdb/duckdb#17796) [Profiling] Propagate the ClientContext into file handle writes (duckdb/duckdb#17754) Ensure we use the same layout in `RadixPartitionedHashTable` and `GroupedAggregateHashTable` (duckdb/duckdb#17790) [Julia] api docs improvements (duckdb/duckdb#15645) [Indexes] Buffer-managed indexes part 1: segment handles (duckdb/duckdb#17758) Mark Upper/LowerComparisonType as const (duckdb/duckdb#17773) Support glibc 2.28 environments (duckdb/duckdb#17776) Pass `ExtensionLoader` when loading extensions, change extension entry function (duckdb/duckdb#17772) Expose file_size_bytes and footer_size in parquet_file_metadata (duckdb/duckdb#17750) [CAPI] Expose ErrorData (duckdb/duckdb#17722) Rename decorator from test_nulls to null_test_parameters (duckdb/duckdb#17760) re-add httpfs apply_patches (duckdb/duckdb#17755) Deprecate windows-2019 runners (duckdb/duckdb#17745) csv_scanner: correct code comment (duckdb/duckdb#17735) Adding additional authenticated data for encryption (duckdb/duckdb#17508) [SQLLogicTester] Introduce `reset label <query label>` in the tester (duckdb/duckdb#17729) Fix windows-2025 build errors (duckdb/duckdb#17726) Aggregation performance (duckdb/duckdb#17718) fix linux extension ci (duckdb/duckdb#17720) Correctly setting the delim offset (duckdb/duckdb#17716) Sorting followup (duckdb/duckdb#17717) Revert "set default for MAIN_BRANCH_VERSIONING to false" (duckdb/duckdb#17708) ClientBufferManager wrapper to access the client context in the buffer manager (duckdb/duckdb#17699) Issue duckdb/duckdb#17040: FILL Window Function (duckdb/duckdb#17686) Merge v1.3-ossivalis into main (duckdb/duckdb#17690) New Sorting Implementation (duckdb/duckdb#17584) Output hashes in unittest and fix order (duckdb/duckdb#17664) Enable profiling output for all operator types (duckdb/duckdb#17665) [C API] Expose duckdb_scalar_function_bind_get_extra_info (duckdb/duckdb#17666) Add rowsort in generate_series test duckdb/duckdb#43 (duckdb/duckdb#17675) bump DuckDB_jll to v1.3.0 (duckdb/duckdb#17677) C API tidying (duckdb/duckdb#17623) fix extension troubleshooting link (duckdb/duckdb#17616) Move query profiler's EndQuery after commit/rollback (duckdb/duckdb#17595) Update function descriptions and examples (duckdb/duckdb#17132) Add support for ToSqlString for union types (duckdb/duckdb#17513) Remove redundant code path in the ConflictManager (duckdb/duckdb#17562) change exception type to not be an internal exception (duckdb/duckdb#17551) Python package devexp improvements (duckdb/duckdb#17483)

carlopi reviewed Apr 28, 2025

View reviewed changes

duckdb-draftbot marked this pull request as draft April 29, 2025 11:50

ccfelius marked this pull request as ready for review April 30, 2025 12:27

ccfelius added Ready For Review and removed Ready For Review labels Apr 30, 2025

duckdb-draftbot marked this pull request as draft April 30, 2025 19:01

ccfelius marked this pull request as ready for review April 30, 2025 19:12

Mytherin reviewed May 1, 2025

View reviewed changes

Mytherin added Changes Requested and removed Ready For Review labels May 1, 2025

duckdb-draftbot marked this pull request as draft May 15, 2025 12:21

ccfelius force-pushed the block_based_encryption branch from fadb5fb to a49992c Compare June 6, 2025 15:23

adding block based encryption (rebased)

ec13fa7

ccfelius force-pushed the block_based_encryption branch from 2f26705 to ec13fa7 Compare June 9, 2025 19:24

ccfelius marked this pull request as ready for review June 9, 2025 19:25

ccfelius added Ready For Review and removed Changes Requested labels Jun 9, 2025

Mytherin reviewed Jun 10, 2025

View reviewed changes

ccfelius added 3 commits June 10, 2025 18:06

Refactor changes

b25ccf7

delete temp test file

fix test

21ae356

fix storage version

9290037

duckdb-draftbot marked this pull request as draft June 10, 2025 16:40

Fix in test

71cba3c

ccfelius marked this pull request as ready for review June 10, 2025 19:47

ccfelius removed the Ready For Review label Jun 10, 2025

Decode base64 encoded keys

d7a5bf3

duckdb-draftbot marked this pull request as draft June 10, 2025 20:58

ccfelius added 2 commits June 10, 2025 23:41

Merge branch 'main' into block_based_encryption

a5d9e60

include conversion exception header

2e854e8

ccfelius marked this pull request as ready for review June 11, 2025 04:16

remove resize with const blockmanager

b34e136

duckdb-draftbot marked this pull request as draft June 11, 2025 05:38

ccfelius marked this pull request as ready for review June 11, 2025 05:38

move user key to encryption config

e11662d

duckdb-draftbot marked this pull request as draft June 11, 2025 15:16

ccfelius marked this pull request as ready for review June 11, 2025 19:28

ccfelius added the Ready For Review label Jun 12, 2025

ccfelius mentioned this pull request Jun 13, 2025

Encryption: adding -key and -master_key for command line #17921

Closed

Mytherin reviewed Jun 16, 2025

View reviewed changes

Use shared_ptr instead of string

181fe3c

duckdb-draftbot marked this pull request as draft June 16, 2025 12:13

ccfelius marked this pull request as ready for review June 16, 2025 12:13

tidy fix

d7cb382

duckdb-draftbot marked this pull request as draft June 16, 2025 12:45

ccfelius added 2 commits June 16, 2025 15:23

pass by reference

51326e0

tidy fix attempt

df25f23

ccfelius marked this pull request as ready for review June 16, 2025 14:43

ccfelius added Ready To Merge and removed Ready For Review labels Jun 17, 2025

Mytherin merged commit 37736e2 into duckdb:main Jun 17, 2025
52 checks passed

ccfelius mentioned this pull request Jun 17, 2025

Encryption: adding -key for the command line #17950

Merged

github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request Jun 21, 2025

vendor: Update vendored sources to duckdb/duckdb@37736e2

fa5abb5

Block based encryption (duckdb/duckdb#17275) Do not dispatch JDBC/ODBC jobs in release CI runs (duckdb/duckdb#17937)

github-actions bot mentioned this pull request Jun 21, 2025

vendor: Update vendored sources to duckdb/duckdb@37736e216e12100e498c5bc4d972afa643c30e0e duckdb/duckdb-r#1285

Merged

hpvd mentioned this pull request Jul 22, 2025

Encryption at rest? tursodatabase/turso#21

Open

		state.AddString("IBd2nLfyDoWYZy6R81DVYxxdM7CAsOcX"); // random salt
		}

Block based encryption #17275

Block based encryption #17275

Uh oh!

Conversation

ccfelius commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Performance

Tests

Internals

Uh oh!

carlopi Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mytherin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mytherin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ccfelius Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

ccfelius commented Apr 28, 2025 •

edited

Loading

carlopi Apr 28, 2025 •

edited

Loading

Mytherin left a comment •

edited

Loading

ccfelius Jun 10, 2025 •

edited

Loading