[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile #12164

Tishj · 2024-05-21T12:26:56Z

Title kind of says it all

STANDARD_VECTOR_SIZE translates to the STANDARD_VECTOR_SIZE define in vector_size.hpp
BLOCK_ALLOC_SIZE translates to the (new) DUCKDB_BLOCK_ALLOC_SIZE define in storage_info.hpp, this is then used to set Storage::BLOCK_ALLOC_SIZE.

I looked at the test_block_sizes.py script for the BLOCK_ALLOC_SIZE
BLOCK_ALLOC_SIZE was by default set to DEFAULT_BLOCK_ALLOC_SIZE

The script only replaces that one instance, but DEFAULT_BLOCK_ALLOC_SIZE is also referenced in src/storage/single_file_block_manager.cpp, I'm not sure if that should also be changed?

…he Makefile

Tishj · 2024-05-21T12:30:12Z

As a follow up, we can rework the vector size and block size tests we have that rely on manually editing the files
I've committed those changes by accident more than once, I figured this was cleaner and less error prone

taniabogatsch · 2024-05-21T14:25:54Z

Cool! Can we trigger the nightly runs on this? I want to make sure that this does not break the block size nightly.

taniabogatsch

I just had a look at this. Currently, you cannot compile with any BLOCK_ALLOC_SIZE. Have a look at the changes in feature. There is a minimum block allocation size, and a test that ensures the block allocation size is a power of two (and maybe others, there is a verify function in storage_info). Maybe we should add these here, similar to this.

#if (STANDARD_ROW_GROUPS_SIZE < STANDARD_VECTOR_SIZE)
#error Row groups must be able to hold at least one vector
#endif

Tishj · 2024-05-21T14:52:03Z

Yea there are requirements, the compilation will fail if those are not met, I ran into those when I tried a random BLOCK_ALLOC_SIZE, as it needs to be a multiple of Storage::SECTOR_SIZE.

STANDARD_VECTOR_SIZE has a similar requirement, needing to be a power of two (though that doesn't seem to be entirely enforced, because 12 was allowed(?))

Those are respected even when set this way

taniabogatsch · 2024-05-21T14:59:33Z

Ah, great!

carlopi

I think also changing/removing scripts/test_block_sizes.py and scripts/test_vector_sizes.py to use standard Make / CMake instead of replacements would be cool, and allow to test this PR.

That said, it's already an improvement as is.

Changes to nightly-tests have to be reverted (I think), so the "request changes".

.github/workflows/NightlyTests.yml

Tishj · 2024-05-21T16:26:47Z

I changed the test_block_sizes.py and test_vector_sizes.py files, seeing as I have to make another commit anyways to revert the nightly test stuff

I'll push it after the relevant test finishes to show that everything works

Tishj · 2024-05-21T18:24:08Z

@taniabogatsch it didn't complete successfully, but I'm not sure the failures are related to these changes?
https://github.com/duckdb/duckdb/actions/runs/9177094312/job/25234470190?pr=12164

…le_constants

Tishj · 2024-05-22T12:37:30Z

Seems to pass the nightly tests:
https://github.com/Tishj/duckdb/actions/runs/9188051251/job/25267556702
https://github.com/Tishj/duckdb/actions/runs/9188051251/job/25267558237

…rable_constants

taniabogatsch · 2024-05-27T11:49:50Z

Could you also add this? We have some places where we implicitly assume this, I know you said earlier that it will fail. But I'd rather have it explicitly in the storage_info.hpp, where we declare them.

#if (DUCKDB_BLOCK_ALLOC_SIZE & (DUCKDB_BLOCK_ALLOC_SIZE - 1) != 0)
#error The duckdb block allocation size must be a power of two
#endif

Also not sure if this will fail if it exceeds the maximum value. As it could still be a power of two. But we have some places where we just assume this maximum when casting. I've added more checks here, but it would be nice to include this also for DUCKDB_BLOCK_ALLOC_SIZE.

#if (DUCKDB_BLOCK_ALLOC_SIZE > 2147483647)
#error The duckdb block allocation size must not exceed the maximum value of a 32-bit signed integer
#endif

And maybe also this?

#if (DUCKDB_BLOCK_ALLOC_SIZE < MIN_BLOCK_ALLOC_SIZE)
#error The duckdb block allocation size must be greater or equal than the minimum block allocation size
#endif

…rable_constants

Mytherin · 2024-05-31T11:01:31Z

Thanks! Great idea

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [duckdb/duckdb](https://redirect.github.com/duckdb/duckdb) | minor | `v1.0.0` -> `v1.1.0` | --- ### Release Notes <details> <summary>duckdb/duckdb (duckdb/duckdb)</summary> ### [`v1.1.0`](https://redirect.github.com/duckdb/duckdb/releases/tag/v1.1.0): DuckDB 1.1.0 "Eatoni" [Compare Source](https://redirect.github.com/duckdb/duckdb/compare/v1.0.0...v1.1.0) This release of DuckDB is named "Eatoni" after Eaton's pintail (Anas Eatoni) from the southern Indian Ocean. Please also refer to the announcement blog post: https://duckdb.org/2024/09/09/announcing-duckdb-110 #### What's Changed - Add feature changes back in by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/11146](https://redirect.github.com/duckdb/duckdb/pull/11146) - Make `MultiFileReader` filename configurable by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/11178](https://redirect.github.com/duckdb/duckdb/pull/11178) - \[Dev] Fix compilation issues on `feature` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11082](https://redirect.github.com/duckdb/duckdb/pull/11082) - add query() and query_table() functions by [@chrisiou](https://redirect.github.com/chrisiou) in [https://github.com/duckdb/duckdb/pull/10586](https://redirect.github.com/duckdb/duckdb/pull/10586) - \[Block Size] Move the block allocation size into the block manager by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/11176](https://redirect.github.com/duckdb/duckdb/pull/11176) - LIMIT pushdown below PROJECT by [@jeewonhh](https://redirect.github.com/jeewonhh) in [https://github.com/duckdb/duckdb/pull/11112](https://redirect.github.com/duckdb/duckdb/pull/11112) - BUGFIX: IN () filter with one argument should translate to = filter. by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/11473](https://redirect.github.com/duckdb/duckdb/pull/11473) - Regression Script should calculate micro benchmark differences with the correct base branch by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/11762](https://redirect.github.com/duckdb/duckdb/pull/11762) - Pushdown filters on window partitions by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/10932](https://redirect.github.com/duckdb/duckdb/pull/10932) - Arrow ListView Type by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/10766](https://redirect.github.com/duckdb/duckdb/pull/10766) - Add scalar function support to the C API by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/11786](https://redirect.github.com/duckdb/duckdb/pull/11786) - Add TopN optimization in physical plan mapping by [@kryonix](https://redirect.github.com/kryonix) in [https://github.com/duckdb/duckdb/pull/11290](https://redirect.github.com/duckdb/duckdb/pull/11290) - Join-dependent filter derivation by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/11272](https://redirect.github.com/duckdb/duckdb/pull/11272) - Implement `ROW_GROUPS_PER_FILE` for Parquet by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/11249](https://redirect.github.com/duckdb/duckdb/pull/11249) - Prefer Final projected columns on probe side if cardinalities are similar by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/11109](https://redirect.github.com/duckdb/duckdb/pull/11109) - Propagate unused columns to distinct on by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/11006](https://redirect.github.com/duckdb/duckdb/pull/11006) - Separate eviction queues by `FileBufferType` by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/11417](https://redirect.github.com/duckdb/duckdb/pull/11417) - Disable false positive for vector size nightly in test by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/11953](https://redirect.github.com/duckdb/duckdb/pull/11953) - Rework jemalloc extension by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/11891](https://redirect.github.com/duckdb/duckdb/pull/11891) - Tweak jemalloc config by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12034](https://redirect.github.com/duckdb/duckdb/pull/12034) - Httpfs test to nightly by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12196](https://redirect.github.com/duckdb/duckdb/pull/12196) - Removed three reinterpret casts and some rewriting by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12200](https://redirect.github.com/duckdb/duckdb/pull/12200) - Begin Profiling Rework to move towards Modularity by [@maiadegraaf](https://redirect.github.com/maiadegraaf) in [https://github.com/duckdb/duckdb/pull/11101](https://redirect.github.com/duckdb/duckdb/pull/11101) - \[CLI] Add highlighting + limited auto-complete for shell dot commands by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12201](https://redirect.github.com/duckdb/duckdb/pull/12201) - Skip test to fix block size nightly and add more explicit error checking by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12211](https://redirect.github.com/duckdb/duckdb/pull/12211) - Remove BLOCK_ALLOC_SIZE from the column segment files by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/11474](https://redirect.github.com/duckdb/duckdb/pull/11474) - \[Julia] - Added optional `schema` input argument to `DuckDB.Appender` constructor by [@curtd](https://redirect.github.com/curtd) in [https://github.com/duckdb/duckdb/pull/12174](https://redirect.github.com/duckdb/duckdb/pull/12174) - Fix Mark Index in the Bound Join Ref by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12263](https://redirect.github.com/duckdb/duckdb/pull/12263) - Fix for CI Regression Failure by [@maiadegraaf](https://redirect.github.com/maiadegraaf) in [https://github.com/duckdb/duckdb/pull/12273](https://redirect.github.com/duckdb/duckdb/pull/12273) - 🦆 by [@samansmink](https://redirect.github.com/samansmink) in [https://github.com/duckdb/duckdb/pull/12303](https://redirect.github.com/duckdb/duckdb/pull/12303) - Disable `JEMALLOC_RETAIN` by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12185](https://redirect.github.com/duckdb/duckdb/pull/12185) - Enforce compression extensions for CSV Files by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/11903](https://redirect.github.com/duckdb/duckdb/pull/11903) - Make spuriously failing test more robust by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12306](https://redirect.github.com/duckdb/duckdb/pull/12306) - Add new extensions to issue template by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12313](https://redirect.github.com/duckdb/duckdb/pull/12313) - \[Fix] Block size nightly run by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12283](https://redirect.github.com/duckdb/duckdb/pull/12283) - Spell Check | Nothing Major | Corrected base_scanner.cpp by [@nj7](https://redirect.github.com/nj7) in [https://github.com/duckdb/duckdb/pull/12282](https://redirect.github.com/duckdb/duckdb/pull/12282) - add duckdb_bind_timestamp_tz function to C API by [@karlseguin](https://redirect.github.com/karlseguin) in [https://github.com/duckdb/duckdb/pull/12151](https://redirect.github.com/duckdb/duckdb/pull/12151) - \[Python] Add some date/datetime functions to pyspark api by [@mariotaddeucci](https://redirect.github.com/mariotaddeucci) in [https://github.com/duckdb/duckdb/pull/12075](https://redirect.github.com/duckdb/duckdb/pull/12075) - Fixes to Windows workflow and ubuntu\_18 action by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12308](https://redirect.github.com/duckdb/duckdb/pull/12308) - \[Extension Dev] Forward declare re2 in `hive_partitioning.hpp` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12302](https://redirect.github.com/duckdb/duckdb/pull/12302) - add expected errors to test/sql/copy/per_thread_output.test by [@hmeriann](https://redirect.github.com/hmeriann) in [https://github.com/duckdb/duckdb/pull/12280](https://redirect.github.com/duckdb/duckdb/pull/12280) - Issue [#12287](https://redirect.github.com/duckdb/duckdb/issues/12287): ICU Strptime Lists by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12295](https://redirect.github.com/duckdb/duckdb/pull/12295) - Issue [#12171](https://redirect.github.com/duckdb/duckdb/issues/12171): Streaming Window FILTER by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12250](https://redirect.github.com/duckdb/duckdb/pull/12250) - \[Python] Update the Connection wrapper generation, now generates c++ code by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12216](https://redirect.github.com/duckdb/duckdb/pull/12216) - Use iterator buffer position when storing buffer handles by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12315](https://redirect.github.com/duckdb/duckdb/pull/12315) - Bump Julia client to v0.10.3 by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12323](https://redirect.github.com/duckdb/duckdb/pull/12323) - Fix [#12286](https://redirect.github.com/duckdb/duckdb/issues/12286) - in the MetadataManager, prefer to allocate new blocks if the next free block id is smaller than the currently used metadata block by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12318](https://redirect.github.com/duckdb/duckdb/pull/12318) - \[Fix] Only read file size if file handle still exists by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12319](https://redirect.github.com/duckdb/duckdb/pull/12319) - Add support for APPEND argument to hive partitioned write by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12262](https://redirect.github.com/duckdb/duckdb/pull/12262) - Remove all reinterpret casts from the transformer by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12320](https://redirect.github.com/duckdb/duckdb/pull/12320) - Additional check for overlapping CTE names by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12305](https://redirect.github.com/duckdb/duckdb/pull/12305) - \[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12164](https://redirect.github.com/duckdb/duckdb/pull/12164) - \[Upsert] Fix issue with lambdas in `DO UPDATE SET` expressions by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11866](https://redirect.github.com/duckdb/duckdb/pull/11866) - \[Python] Fix scoping issue for `pandas_analyze_sample` setting by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11706](https://redirect.github.com/duckdb/duckdb/pull/11706) - Support REGEX matches expected error message by [@hmeriann](https://redirect.github.com/hmeriann) in [https://github.com/duckdb/duckdb/pull/12327](https://redirect.github.com/duckdb/duckdb/pull/12327) - Allow run_fuzzer to reduce multi statements. by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/12278](https://redirect.github.com/duckdb/duckdb/pull/12278) - Fix [#12328](https://redirect.github.com/duckdb/duckdb/issues/12328) - when flattening STRUCT vectors with NULL values, we need to flatten the children recursively as well by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12332](https://redirect.github.com/duckdb/duckdb/pull/12332) - Make `dbgen` generate data in parallel by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12337](https://redirect.github.com/duckdb/duckdb/pull/12337) - dbgen: skip parallel generation if DUCKDB_NO_THREADS is set by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12341](https://redirect.github.com/duckdb/duckdb/pull/12341) - Add prefix prefix_front_back. to get prefix_front\_ and prefix_back\_ by [@liujiayi771](https://redirect.github.com/liujiayi771) in [https://github.com/duckdb/duckdb/pull/12344](https://redirect.github.com/duckdb/duckdb/pull/12344) - Issue [#12171](https://redirect.github.com/duckdb/duckdb/issues/12171): Streaming Windowed DISTINCT by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12311](https://redirect.github.com/duckdb/duckdb/pull/12311) - Update README by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12357](https://redirect.github.com/duckdb/duckdb/pull/12357) - \[CSV Reader] \[Skip Option] Tests and fixes by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12213](https://redirect.github.com/duckdb/duckdb/pull/12213) - Adjust BM25 score in FTS extension to prevent negative scores by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12356](https://redirect.github.com/duckdb/duckdb/pull/12356) - Fix typos by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12360](https://redirect.github.com/duckdb/duckdb/pull/12360) - Fix [#12293](https://redirect.github.com/duckdb/duckdb/issues/12293) - accept NULL values in generate_series with timestamp by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12367](https://redirect.github.com/duckdb/duckdb/pull/12367) - Fix [#12335](https://redirect.github.com/duckdb/duckdb/issues/12335): avoid calling fsync when writing Parquet files, instead just close the file by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12371](https://redirect.github.com/duckdb/duckdb/pull/12371) - Fix parameters passed down to other workflows in OnTag.yml by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12369](https://redirect.github.com/duckdb/duckdb/pull/12369) - \[Python] Fixes for the SQLLogicTest runner implementation by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12372](https://redirect.github.com/duckdb/duckdb/pull/12372) - Bump julia to v1.0.0 by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12379](https://redirect.github.com/duckdb/duckdb/pull/12379) - Fix [#11921](https://redirect.github.com/duckdb/duckdb/issues/11921) - varchar -> timestamp casts are not invertible by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12376](https://redirect.github.com/duckdb/duckdb/pull/12376) - Upgrade utf8proc - and move our custom extensions out of utf8proc itself by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12373](https://redirect.github.com/duckdb/duckdb/pull/12373) - change max_queries number back to 2000 by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/12375](https://redirect.github.com/duckdb/duckdb/pull/12375) - Remove sqlsmith extension by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/12300](https://redirect.github.com/duckdb/duckdb/pull/12300) - Reorder semi and anti joins. by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/11815](https://redirect.github.com/duckdb/duckdb/pull/11815) - Issue [#12351](https://redirect.github.com/duckdb/duckdb/issues/12351): implicit cast to `TIMESTAMP_MS`, `TIMESTAMP_S`, `TIMESTAMP_NS` from `DATE` values by [@akoshchiy](https://redirect.github.com/akoshchiy) in [https://github.com/duckdb/duckdb/pull/12352](https://redirect.github.com/duckdb/duckdb/pull/12352) - Issue [#10023](https://redirect.github.com/duckdb/duckdb/issues/10023): Approx_Count_Distinct Memory Usage by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12355](https://redirect.github.com/duckdb/duckdb/pull/12355) - Fix a small typo in dev instructions for swift setup by [@gjmwoods](https://redirect.github.com/gjmwoods) in [https://github.com/duckdb/duckdb/pull/12383](https://redirect.github.com/duckdb/duckdb/pull/12383) - Release lock before returning `BufferHandle` in `StandardBufferManager::Pin` by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12391](https://redirect.github.com/duckdb/duckdb/pull/12391) - Remote attach autoload by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12393](https://redirect.github.com/duckdb/duckdb/pull/12393) - Add JSON type to Parquet reader/writer by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12222](https://redirect.github.com/duckdb/duckdb/pull/12222) - Add `RETURN_FILES` parameter to `COPY TO` by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12220](https://redirect.github.com/duckdb/duckdb/pull/12220) - Updated JoinHashTable to use linear probing to resolve hash collisions by [@gropaul](https://redirect.github.com/gropaul) in [https://github.com/duckdb/duckdb/pull/11472](https://redirect.github.com/duckdb/duckdb/pull/11472) - \[Benchmark Runner] Add `--disable-timeout` flag by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12387](https://redirect.github.com/duckdb/duckdb/pull/12387) - Don't replace unicode spaces within `$$` quotes in query strings by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12405](https://redirect.github.com/duckdb/duckdb/pull/12405) - \[Python] Fix fatal exception caused by empty Pandas Categorical objects. by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12370](https://redirect.github.com/duckdb/duckdb/pull/12370) - Release CSV Blocks when acquiring new blocks if single threaded by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12409](https://redirect.github.com/duckdb/duckdb/pull/12409) - Add support for prefetching multiple adjacent blocks in a single batched read when attaching to remote databases by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12413](https://redirect.github.com/duckdb/duckdb/pull/12413) - MatchRegex() fixed to do not return false positive result by [@hmeriann](https://redirect.github.com/hmeriann) in [https://github.com/duckdb/duckdb/pull/12396](https://redirect.github.com/duckdb/duckdb/pull/12396) - Expected errors 2053 by [@hmeriann](https://redirect.github.com/hmeriann) in [https://github.com/duckdb/duckdb/pull/12392](https://redirect.github.com/duckdb/duckdb/pull/12392) - \[C-API] Catch exception in `duckdb_execute_prepared` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12414](https://redirect.github.com/duckdb/duckdb/pull/12414) - Combining LIST_CONCAT and CONCAT binding by [@maiadegraaf](https://redirect.github.com/maiadegraaf) in [https://github.com/duckdb/duckdb/pull/12317](https://redirect.github.com/duckdb/duckdb/pull/12317) - \[Appender] Add `AppendDefault` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11905](https://redirect.github.com/duckdb/duckdb/pull/11905) - \[Python Dev] Push CTE internally for every (python) replacement scan that occurred. by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12161](https://redirect.github.com/duckdb/duckdb/pull/12161) - Improve compiler compatibility by [@krlmlr](https://redirect.github.com/krlmlr) in [https://github.com/duckdb/duckdb/pull/12401](https://redirect.github.com/duckdb/duckdb/pull/12401) - Write zero-length list offsets for NULL values when serializing vectors by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12423](https://redirect.github.com/duckdb/duckdb/pull/12423) - Get column statistics if Logical Get has a statistics function by [@jeewonhh](https://redirect.github.com/jeewonhh) in [https://github.com/duckdb/duckdb/pull/12424](https://redirect.github.com/duckdb/duckdb/pull/12424) - jemalloc: Identify GNU source code properly by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12420](https://redirect.github.com/duckdb/duckdb/pull/12420) - Avoid parallelizing LIMIT clauses when the query plan is simple by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12433](https://redirect.github.com/duckdb/duckdb/pull/12433) - Prefetch metadata blocks for remote files by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12437](https://redirect.github.com/duckdb/duckdb/pull/12437) - \[Jupyter] Remove width limit on the BoxRenderer config by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12443](https://redirect.github.com/duckdb/duckdb/pull/12443) - Revert [#10865](https://redirect.github.com/duckdb/duckdb/issues/10865) by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12426](https://redirect.github.com/duckdb/duckdb/pull/12426) - inline delta by [@samansmink](https://redirect.github.com/samansmink) in [https://github.com/duckdb/duckdb/pull/12435](https://redirect.github.com/duckdb/duckdb/pull/12435) - Account for *tagged* dollar-quoted strings when stripping unicode spaces by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12421](https://redirect.github.com/duckdb/duckdb/pull/12421) - Work-around for broken github windows runner by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12447](https://redirect.github.com/duckdb/duckdb/pull/12447) - Prevents clearing of the types of the LogicalExecute operator by [@NiclasHaderer](https://redirect.github.com/NiclasHaderer) in [https://github.com/duckdb/duckdb/pull/12436](https://redirect.github.com/duckdb/duckdb/pull/12436) - Add support for BEGIN TRANSACTION READ ONLY by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12202](https://redirect.github.com/duckdb/duckdb/pull/12202) - Make `range` and `generate_series` table in-out functions, and fix several issues with table in-out functions by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12431](https://redirect.github.com/duckdb/duckdb/pull/12431) - Issue [#12412](https://redirect.github.com/duckdb/duckdb/issues/12412): AsOf Filter Push by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12448](https://redirect.github.com/duckdb/duckdb/pull/12448) - \[Fix] Block Size Nightly by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12427](https://redirect.github.com/duckdb/duckdb/pull/12427) - \[ART] Remove Flatten and template key generation by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12428](https://redirect.github.com/duckdb/duckdb/pull/12428) - \[Python] Clean up internals of `execute` / `executemany` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12434](https://redirect.github.com/duckdb/duckdb/pull/12434) - By default attach remote databases as READ_ONLY by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12461](https://redirect.github.com/duckdb/duckdb/pull/12461) - Fix [#11837](https://redirect.github.com/duckdb/duckdb/issues/11837): use internal physical type for FIRST/LAST/ANY_VALUE instead of logical type by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12462](https://redirect.github.com/duckdb/duckdb/pull/12462) - Issue [#12464](https://redirect.github.com/duckdb/duckdb/issues/12464): Windowed Order By All by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12470](https://redirect.github.com/duckdb/duckdb/pull/12470) - Specialize `list_value` for primitive types for significantly improved performance by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12468](https://redirect.github.com/duckdb/duckdb/pull/12468) - \[Dev] Remove dead code from `PhysicalBatchCopyToFile` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12459](https://redirect.github.com/duckdb/duckdb/pull/12459) - Disable Windows extensions CI until Github actions runners are fixed by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12479](https://redirect.github.com/duckdb/duckdb/pull/12479) - \[Fix] access_mode now lives in AttachOptions by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12482](https://redirect.github.com/duckdb/duckdb/pull/12482) - Internal [#2186](https://redirect.github.com/duckdb/duckdb/issues/2186): Nanosecond Functionality by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12440](https://redirect.github.com/duckdb/duckdb/pull/12440) - \[C-API] Fix leak in `duckdb_create_config` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12465](https://redirect.github.com/duckdb/duckdb/pull/12465) - \[Python] No longer scan the entire frame lineage in a replacement scan, added option to disable (python) replacements entirely by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12425](https://redirect.github.com/duckdb/duckdb/pull/12425) - throw binder error for comment on system catalog by [@samansmink](https://redirect.github.com/samansmink) in [https://github.com/duckdb/duckdb/pull/12486](https://redirect.github.com/duckdb/duckdb/pull/12486) - Parquet reader performance by [@lnkuiper](https://redirect.github.com/lnkuiper) in [https://github.com/duckdb/duckdb/pull/12478](https://redirect.github.com/duckdb/duckdb/pull/12478) - Operators the Optimizer can skip by [@Tmonster](https://redirect.github.com/Tmonster) in [https://github.com/duckdb/duckdb/pull/12489](https://redirect.github.com/duckdb/duckdb/pull/12489) - Fixes clang conversion warnings by [@TinyTinni](https://redirect.github.com/TinyTinni) in [https://github.com/duckdb/duckdb/pull/12467](https://redirect.github.com/duckdb/duckdb/pull/12467) - Avoid creating internal schemas as non-internal when reading old database files by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12456](https://redirect.github.com/duckdb/duckdb/pull/12456) - Allow parquet encryption/decryption keys to be passed in as base64 encoded strings by [@elefeint](https://redirect.github.com/elefeint) in [https://github.com/duckdb/duckdb/pull/12445](https://redirect.github.com/duckdb/duckdb/pull/12445) - \[Block Size] Introducing CompressionInfo by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12481](https://redirect.github.com/duckdb/duckdb/pull/12481) - add the number of filtered files to explain by [@samansmink](https://redirect.github.com/samansmink) in [https://github.com/duckdb/duckdb/pull/12488](https://redirect.github.com/duckdb/duckdb/pull/12488) - Implement Map Type Detection for JSON Reader by [@ZiyaZa](https://redirect.github.com/ZiyaZa) in [https://github.com/duckdb/duckdb/pull/11285](https://redirect.github.com/duckdb/duckdb/pull/11285) - \[Dev] Remove busy-spin from `ClientContext::ExecuteTaskInternal` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12483](https://redirect.github.com/duckdb/duckdb/pull/12483) - Pluggable collations by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12492](https://redirect.github.com/duckdb/duckdb/pull/12492) - \[Dev] Don't fail `make generate-files` if the python code generation fails by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12500](https://redirect.github.com/duckdb/duckdb/pull/12500) - Optimize `EXTRACT(year/month/day FROM date/timestamp)` by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12499](https://redirect.github.com/duckdb/duckdb/pull/12499) - \[Fix] Remove BLOCK_ALLOC_SIZE in the single file block manager by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12502](https://redirect.github.com/duckdb/duckdb/pull/12502) - Revert Windows CI fixes by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12510](https://redirect.github.com/duckdb/duckdb/pull/12510) - Fix [#12467](https://redirect.github.com/duckdb/duckdb/issues/12467) changes to covariance calculation by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12515](https://redirect.github.com/duckdb/duckdb/pull/12515) - \[Python] Fix reading strided `datetime` and `timedelta` columns by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12519](https://redirect.github.com/duckdb/duckdb/pull/12519) - Add method for decoding sort keys, and use this in min/max for arbitrary types by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12520](https://redirect.github.com/duckdb/duckdb/pull/12520) - Reduce allocations & use predication in ColumnSegment::FilterSelection by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12521](https://redirect.github.com/duckdb/duckdb/pull/12521) - Skip only built-in optimizers by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12522](https://redirect.github.com/duckdb/duckdb/pull/12522) - Improve min/max performance for strings and fallback types by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12524](https://redirect.github.com/duckdb/duckdb/pull/12524) - Move arg_min/arg_max to use sort keys by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12525](https://redirect.github.com/duckdb/duckdb/pull/12525) - Move FIRST/LAST/ANY_VALUE to use sort keys by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12526](https://redirect.github.com/duckdb/duckdb/pull/12526) - CMake: use GNUInstallDirs as defaults for INSTALL\_{BIN,LIB,INCLUDE}\_DIR by [@paparodeo](https://redirect.github.com/paparodeo) in [https://github.com/duckdb/duckdb/pull/12509](https://redirect.github.com/duckdb/duckdb/pull/12509) - More formatting and fix to stddev by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12516](https://redirect.github.com/duckdb/duckdb/pull/12516) - Linux Extensions CI: Attempt at fix missing dependencies by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12429](https://redirect.github.com/duckdb/duckdb/pull/12429) - Fix checkouts by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12366](https://redirect.github.com/duckdb/duckdb/pull/12366) - Etag if none match for extension install by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12333](https://redirect.github.com/duckdb/duckdb/pull/12333) - \[Block Size] FixedSizeAllocator, MetadataManager, PartialBlockManager by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12514](https://redirect.github.com/duckdb/duckdb/pull/12514) - \[Python] Skip the PandasAnalyzer if dtype is `'string'` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12511](https://redirect.github.com/duckdb/duckdb/pull/12511) - \[StreamQueryResult] Batched variant of the StreamQueryResult collector by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11494](https://redirect.github.com/duckdb/duckdb/pull/11494) - Move many tests to slow by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12534](https://redirect.github.com/duckdb/duckdb/pull/12534) - Add support for `arg_min(ANY, ANY)` by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12532](https://redirect.github.com/duckdb/duckdb/pull/12532) - Avoid overriding types in PrepareTypeForCast when not required by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12539](https://redirect.github.com/duckdb/duckdb/pull/12539) - Support all types in `histogram` function by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12538](https://redirect.github.com/duckdb/duckdb/pull/12538) - \[Python] Remove busy-spin during execution by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12512](https://redirect.github.com/duckdb/duckdb/pull/12512) - \[Block Size] String space constant by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12537](https://redirect.github.com/duckdb/duckdb/pull/12537) - Use string_t instead of std::string in histogram by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12545](https://redirect.github.com/duckdb/duckdb/pull/12545) - Add support for binned histograms by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12548](https://redirect.github.com/duckdb/duckdb/pull/12548) - \[Upsert] Fix RETURNING for `DO NOTHING` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12554](https://redirect.github.com/duckdb/duckdb/pull/12554) - Build Android Binaries by [@hannes](https://redirect.github.com/hannes) in [https://github.com/duckdb/duckdb/pull/12550](https://redirect.github.com/duckdb/duckdb/pull/12550) - \[CI] Remove pyarrow version lock by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12566](https://redirect.github.com/duckdb/duckdb/pull/12566) - \[Dev] Change tests: np.NaN -> np.nan by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12565](https://redirect.github.com/duckdb/duckdb/pull/12565) - Internal [#2017](https://redirect.github.com/duckdb/duckdb/issues/2017): DECIMAL Downcast Rounding by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12036](https://redirect.github.com/duckdb/duckdb/pull/12036) - Issue [#12204](https://redirect.github.com/duckdb/duckdb/issues/12204): Summarize Temporal Quantiles by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12297](https://redirect.github.com/duckdb/duckdb/pull/12297) - Internal [#2186](https://redirect.github.com/duckdb/duckdb/issues/2186): Nanosecond StrTimeFormat by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12551](https://redirect.github.com/duckdb/duckdb/pull/12551) - Add support for `equi_width_bins` function to compute histogram boundaries by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12574](https://redirect.github.com/duckdb/duckdb/pull/12574) - add support for casting 'yes'/'no' strings to boolean values by [@chrisiou](https://redirect.github.com/chrisiou) in [https://github.com/duckdb/duckdb/pull/12501](https://redirect.github.com/duckdb/duckdb/pull/12501) - Julia: Add chunked results with Tables.partitions() by [@frankier](https://redirect.github.com/frankier) in [https://github.com/duckdb/duckdb/pull/12395](https://redirect.github.com/duckdb/duckdb/pull/12395) - \[PySpark] - Allow spark session range by [@mariotaddeucci](https://redirect.github.com/mariotaddeucci) in [https://github.com/duckdb/duckdb/pull/12346](https://redirect.github.com/duckdb/duckdb/pull/12346) - \[PySpark] Implement subset drop duplicates by [@mariotaddeucci](https://redirect.github.com/mariotaddeucci) in [https://github.com/duckdb/duckdb/pull/12348](https://redirect.github.com/duckdb/duckdb/pull/12348) - ICU noaccent collation by [@tiagokepe](https://redirect.github.com/tiagokepe) in [https://github.com/duckdb/duckdb/pull/12170](https://redirect.github.com/duckdb/duckdb/pull/12170) - Implement Brotli compression for Parquet reading & writing by [@hannes](https://redirect.github.com/hannes) in [https://github.com/duckdb/duckdb/pull/12103](https://redirect.github.com/duckdb/duckdb/pull/12103) - \[FriendlySQL] Unpacked COLUMNS() Expression by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/11872](https://redirect.github.com/duckdb/duckdb/pull/11872) - \[PySpark] Implement UDFRegistration.register method on PySpark api by [@mariotaddeucci](https://redirect.github.com/mariotaddeucci) in [https://github.com/duckdb/duckdb/pull/12179](https://redirect.github.com/duckdb/duckdb/pull/12179) - \[Python] Don't use `np.nan`, deprecated alias starting with NumPy 2.0 by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12583](https://redirect.github.com/duckdb/duckdb/pull/12583) - Add `bind_expression` callback to scalar function, and use it to turn `typeof` into a `BoundConstantExpression` by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12580](https://redirect.github.com/duckdb/duckdb/pull/12580) - Add `can_cast_implicitly` scalar function by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12581](https://redirect.github.com/duckdb/duckdb/pull/12581) - Add support for `histogram` and `histogram_values` table macro, and add support for default table macros (similar to how we support default macros) by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12590](https://redirect.github.com/duckdb/duckdb/pull/12590) - build: swap libclang for cxxheaderparser by [@Mause](https://redirect.github.com/Mause) in [https://github.com/duckdb/duckdb/pull/12567](https://redirect.github.com/duckdb/duckdb/pull/12567) - \[C-API] Add `table_description` struct to query various information about the table. by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12460](https://redirect.github.com/duckdb/duckdb/pull/12460) - Change new micro benchmark script to only look for `.benchmark` files by [@maiadegraaf](https://redirect.github.com/maiadegraaf) in [https://github.com/duckdb/duckdb/pull/12598](https://redirect.github.com/duckdb/duckdb/pull/12598) - Add HTTP error code to extension install failures by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12608](https://redirect.github.com/duckdb/duckdb/pull/12608) - Separate WAL write from commit, and allow writing to the WAL without holding the transaction lock by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12261](https://redirect.github.com/duckdb/duckdb/pull/12261) - Add `OwningStringMap` - and rework `histogram` and `mode` functions to use this by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12601](https://redirect.github.com/duckdb/duckdb/pull/12601) - Feature [#1272](https://redirect.github.com/duckdb/duckdb/issues/1272): Window Executor State by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12573](https://redirect.github.com/duckdb/duckdb/pull/12573) - Add support for any type to `mode` aggregate by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12619](https://redirect.github.com/duckdb/duckdb/pull/12619) - WAL - when dropping a table, also delete any transaction local storage associated with that table by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12603](https://redirect.github.com/duckdb/duckdb/pull/12603) - \[Python] Allow Generators to be passed where List is expected by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12602](https://redirect.github.com/duckdb/duckdb/pull/12602) - VectorOperations::Copy - fast path when copying an aligned flat validity mask into a flat vector by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12618](https://redirect.github.com/duckdb/duckdb/pull/12618) - Move android CI to only run during nightly CI triggers by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12622](https://redirect.github.com/duckdb/duckdb/pull/12622) - Add initial support for GeoParquet + Bump spatial by [@Maxxen](https://redirect.github.com/Maxxen) in [https://github.com/duckdb/duckdb/pull/12503](https://redirect.github.com/duckdb/duckdb/pull/12503) - Issue [#12600](https://redirect.github.com/duckdb/duckdb/issues/12600): Streaming Positive LAG by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12609](https://redirect.github.com/duckdb/duckdb/pull/12609) - Feature [#1272](https://redirect.github.com/duckdb/duckdb/issues/1272): Window Group Preparation by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12628](https://redirect.github.com/duckdb/duckdb/pull/12628) - Minor window improvements by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12617](https://redirect.github.com/duckdb/duckdb/pull/12617) - Merge feature into main by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12633](https://redirect.github.com/duckdb/duckdb/pull/12633) - Refactor `quantile` aggregate - clean up code & support `quantile_disc`/`median` for all types by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12630](https://redirect.github.com/duckdb/duckdb/pull/12630) - Feature 1272: Window Payload Preallocation by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12629](https://redirect.github.com/duckdb/duckdb/pull/12629) - \[ART] Configurable index scan threshold by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12635](https://redirect.github.com/duckdb/duckdb/pull/12635) - Subtract start offset for when fetching array child segment by [@Maxxen](https://redirect.github.com/Maxxen) in [https://github.com/duckdb/duckdb/pull/12639](https://redirect.github.com/duckdb/duckdb/pull/12639) - Remove custom logic to detect main vs feature by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12643](https://redirect.github.com/duckdb/duckdb/pull/12643) - Do not quote fields with space in the CSV output mode by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12644](https://redirect.github.com/duckdb/duckdb/pull/12644) - Use lowercase in 'html' output mode by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12612](https://redirect.github.com/duckdb/duckdb/pull/12612) - Internal [#2361](https://redirect.github.com/duckdb/duckdb/issues/2361): Window ROWS Overflow by [@hawkfish](https://redirect.github.com/hawkfish) in [https://github.com/duckdb/duckdb/pull/12652](https://redirect.github.com/duckdb/duckdb/pull/12652) - Quantile: Fix variable used only in D_ASSERT by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12642](https://redirect.github.com/duckdb/duckdb/pull/12642) - Skip pytorch test, it fails spuriously in CI by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12645](https://redirect.github.com/duckdb/duckdb/pull/12645) - Add `histogram_exact` function that adds values to bins only if they match exactly, and add `other` column that contains values that do not fit in any bin by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12650](https://redirect.github.com/duckdb/duckdb/pull/12650) - Add operator hook for sink progress by [@Maxxen](https://redirect.github.com/Maxxen) in [https://github.com/duckdb/duckdb/pull/12637](https://redirect.github.com/duckdb/duckdb/pull/12637) - Regression workflow on newly introduced benchmarks: remove for now by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12659](https://redirect.github.com/duckdb/duckdb/pull/12659) - Fix [#12646](https://redirect.github.com/duckdb/duckdb/issues/12646) - allow SQL value functions in HAVING by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12654](https://redirect.github.com/duckdb/duckdb/pull/12654) - Add != operators on string_t and interval_t by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12658](https://redirect.github.com/duckdb/duckdb/pull/12658) - fix: improve C scalar functions API by [@rustyconover](https://redirect.github.com/rustyconover) in [https://github.com/duckdb/duckdb/pull/12663](https://redirect.github.com/duckdb/duckdb/pull/12663) - Add `approx_top_k` aggregate based on the (Filtered) Space-Saving algorithm, and use it in histogram by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12653](https://redirect.github.com/duckdb/duckdb/pull/12653) - Fix std::sort requirements, from greater_equal to greater by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12669](https://redirect.github.com/duckdb/duckdb/pull/12669) - fix(parquet): two-complement zeroes check on FIXED_BYTE_ARRAY encoded DECIMAL ([#12621](https://redirect.github.com/duckdb/duckdb/issues/12621)) by [@fedefrancescon](https://redirect.github.com/fedefrancescon) in [https://github.com/duckdb/duckdb/pull/12655](https://redirect.github.com/duckdb/duckdb/pull/12655) - \[CSV Reader] Reorder of Columns for CSV Scans on multiple files. by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12288](https://redirect.github.com/duckdb/duckdb/pull/12288) - \[CSV] \[Bug-Fix] Fix for issue related with single-threaded execution and null padding. by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12679](https://redirect.github.com/duckdb/duckdb/pull/12679) - \[Block Size] String block limit and a few other places by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12671](https://redirect.github.com/duckdb/duckdb/pull/12671) - Rework arena allocator allocation policy - and increase pivot threshold by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12690](https://redirect.github.com/duckdb/duckdb/pull/12690) - Julia - Fix Base.isopen(db::DB) in [https://github.com/duckdb/duckdb/pull/12700](https://redirect.github.com/duckdb/duckdb/pull/12700) - \[CLI] Limit history size to 100MB, and avoid writing invalid UTF8 to the CLI history by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12677](https://redirect.github.com/duckdb/duckdb/pull/12677) - Add configurable thresholds for using nested loop join and merge join by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12689](https://redirect.github.com/duckdb/duckdb/pull/12689) - Prevent unnecessary usage of `std::string` in `list` aggregate - and use more efficient `memcpy` for batched copy by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12694](https://redirect.github.com/duckdb/duckdb/pull/12694) - Dont load spatial unless geoparquet metata is present by [@Maxxen](https://redirect.github.com/Maxxen) in [https://github.com/duckdb/duckdb/pull/12692](https://redirect.github.com/duckdb/duckdb/pull/12692) - Serialization: add CustomData and better support for integrating with extensions by [@jeewonhh](https://redirect.github.com/jeewonhh) in [https://github.com/duckdb/duckdb/pull/12681](https://redirect.github.com/duckdb/duckdb/pull/12681) - Removing ODBC driver by [@hannes](https://redirect.github.com/hannes) in [https://github.com/duckdb/duckdb/pull/12706](https://redirect.github.com/duckdb/duckdb/pull/12706) - Support thousand separator for floating point numbers by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12717](https://redirect.github.com/duckdb/duckdb/pull/12717) - \[Python] Use non-owning references to hold created cursors by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12711](https://redirect.github.com/duckdb/duckdb/pull/12711) - LIST(VARCHAR) - reduce memory usage by avoiding allocation of nullmask for string data, and allocate larger initial batches by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12705](https://redirect.github.com/duckdb/duckdb/pull/12705) - \[CSV] Bug fix for race condition in single-threaded multifile reader + properly print paths on union_by_name errors. by [@pdet](https://redirect.github.com/pdet) in [https://github.com/duckdb/duckdb/pull/12697](https://redirect.github.com/duckdb/duckdb/pull/12697) - Issue template: Add ODBC and Node (neo) clients by [@szarnyasg](https://redirect.github.com/szarnyasg) in [https://github.com/duckdb/duckdb/pull/12714](https://redirect.github.com/duckdb/duckdb/pull/12714) - Shell: add .sql suffix to temporary file created with \e by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12723](https://redirect.github.com/duckdb/duckdb/pull/12723) - Partitioned write - keep only up until 100 files open, when this limit is exceeded close the file and create a new file if more data for this partition appears by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12708](https://redirect.github.com/duckdb/duckdb/pull/12708) - Change setting types to fix warnings by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12724](https://redirect.github.com/duckdb/duckdb/pull/12724) - Avoid unnecessarily copying child expression when binding COLLATE statements by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12725](https://redirect.github.com/duckdb/duckdb/pull/12725) - Support for variadic arguments in scalar UDFs in the C API by [@taniabogatsch](https://redirect.github.com/taniabogatsch) in [https://github.com/duckdb/duckdb/pull/12678](https://redirect.github.com/duckdb/duckdb/pull/12678) - \[Relation API] Dont push DISTINCT modifier for EXCEPT/INTERSECT ALL by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12599](https://redirect.github.com/duckdb/duckdb/pull/12599) - Builds for Windows on ARM64 by [@hannes](https://redirect.github.com/hannes) in [https://github.com/duckdb/duckdb/pull/12586](https://redirect.github.com/duckdb/duckdb/pull/12586) - Rework `union_by_name` so that files are no longer kept open by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12730](https://redirect.github.com/duckdb/duckdb/pull/12730) - Fix [#12729](https://redirect.github.com/duckdb/duckdb/issues/12729): early-out when checking for perfect hash joins when running on empty tables by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12731](https://redirect.github.com/duckdb/duckdb/pull/12731) - CLI: Replace \n with \r\n again in history again by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12735](https://redirect.github.com/duckdb/duckdb/pull/12735) - Fix [#11228](https://redirect.github.com/duckdb/duckdb/issues/11228) - add support for unsigned integers in printf/format by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12736](https://redirect.github.com/duckdb/duckdb/pull/12736) - Various CI fixes by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12737](https://redirect.github.com/duckdb/duckdb/pull/12737) - Add repeat(LIST\[], INT) that allows repetition of lists similar to how this is allowed in Python by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12738](https://redirect.github.com/duckdb/duckdb/pull/12738) - \[Python] Add missing options to `read_json` method by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12732](https://redirect.github.com/duckdb/duckdb/pull/12732) - Add support for fetching cardinality estimation and stats through a multifilelist by [@samansmink](https://redirect.github.com/samansmink) in [https://github.com/duckdb/duckdb/pull/12740](https://redirect.github.com/duckdb/duckdb/pull/12740) - Fixes warnings detected by cppcheck by [@carlopi](https://redirect.github.com/carlopi) in [https://github.com/duckdb/duckdb/pull/12745](https://redirect.github.com/duckdb/duckdb/pull/12745) - \[Arrow] Add `ArrowQueryResult` by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12496](https://redirect.github.com/duckdb/duckdb/pull/12496) - \[Dev] StreamQueryResult internals cleanup by [@Tishj](https://redirect.github.com/Tishj) in [https://github.com/duckdb/duckdb/pull/12636](https://redirect.github.com/duckdb/duckdb/pull/12636) - ALP/ALPRD: correctly skip when we are skipping fewer values than in a vector by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12753](https://redirect.github.com/duckdb/duckdb/pull/12753) - Maintain prepared statement parameter types explicitly instead of converting into literals by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12759](https://redirect.github.com/duckdb/duckdb/pull/12759) - CLI .changes: use sqlite3\_changes64 and sqlite3\_totalchanges64 to prevent overflows by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12761](https://redirect.github.com/duckdb/duckdb/pull/12761) - Fix [#12569](https://redirect.github.com/duckdb/duckdb/issues/12569): avoid truncating zeros that matter in format function by [@Mytherin](https://redirect.github.com/Mytherin) in [https://github.com/duckdb/duckdb/pull/12762](https://redirect.github.com/duckdb/duckdb/pull/12762) - Fix [#12418](https://redirect.github.com/duckdb/duckdb/issues/12418): Remove .lint command in SQLite shell by [@Mytherin](https://redire </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/d-issy/dotfiles).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Tishj added 2 commits May 21, 2024 14:22

make STANDARD_VECTOR_SIZE and BLOCK_ALLOC_SIZE configurable through t…

347f5aa

…he Makefile

remove erroneous comment

4e2cfa2

Tishj requested a review from taniabogatsch May 21, 2024 12:27

taniabogatsch reviewed May 21, 2024

View reviewed changes

this should make nightly run for this PR

8e2f92e

Tishj marked this pull request as draft May 21, 2024 15:06

Tishj marked this pull request as ready for review May 21, 2024 15:16

carlopi suggested changes May 21, 2024

View reviewed changes

.github/workflows/NightlyTests.yml Outdated Show resolved Hide resolved

Tishj added 3 commits May 21, 2024 18:17

undo nightly test change

89cdea8

set the constants using the Makefile in 'test_block_sizes.py'

2b29c13

use the STANDARD_VECTOR_SIZE in 'test_vector_sizes.py'

8de262c

Tishj marked this pull request as draft May 21, 2024 18:24

Tishj marked this pull request as ready for review May 21, 2024 18:24

duckdb-draftbot marked this pull request as draft May 21, 2024 18:24

Tishj marked this pull request as ready for review May 21, 2024 19:33

Merge remote-tracking branch 'upstream/main' into makefile_configurab…

29c18af

…le_constants

duckdb-draftbot marked this pull request as draft May 22, 2024 08:35

carlopi approved these changes May 22, 2024

View reviewed changes

taniabogatsch approved these changes May 22, 2024

View reviewed changes

Tishj changed the base branch from main to feature May 24, 2024 12:42

Merge remote-tracking branch 'upstream/feature' into makefile_configu…

12df14d

…rable_constants

Tishj marked this pull request as ready for review May 24, 2024 13:20

Merge remote-tracking branch 'upstream/feature' into makefile_configu…

6f09b68

…rable_constants

duckdb-draftbot marked this pull request as draft May 29, 2024 10:47

add checks for the DUCKDB_BLOCK_ALLOC_SIZE define

eafe424

Tishj marked this pull request as ready for review May 29, 2024 11:35

Merge remote-tracking branch 'upstream/feature' into makefile_configu…

c2c1828

…rable_constants

duckdb-draftbot marked this pull request as draft May 30, 2024 15:33

Tishj marked this pull request as ready for review May 30, 2024 15:39

Tishj added the Ready For Review label May 30, 2024

Mytherin merged commit de8c9f3 into duckdb:feature May 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile #12164

[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile #12164

Uh oh!

Tishj commented May 21, 2024 •

edited

Loading

Uh oh!

Tishj commented May 21, 2024

Uh oh!

taniabogatsch commented May 21, 2024

Uh oh!

taniabogatsch left a comment

Uh oh!

Tishj commented May 21, 2024 •

edited

Loading

Uh oh!

taniabogatsch commented May 21, 2024

Uh oh!

carlopi left a comment

Uh oh!

Uh oh!

Tishj commented May 21, 2024

Uh oh!

Tishj commented May 21, 2024

Uh oh!

Tishj commented May 22, 2024

Uh oh!

taniabogatsch commented May 27, 2024

Uh oh!

Mytherin commented May 31, 2024

Uh oh!

Uh oh!

[Dev] STANDARD_VECTOR_SIZE and BLOCK_ALLOC_SIZE can now be set through the Makefile #12164

[Dev] STANDARD_VECTOR_SIZE and BLOCK_ALLOC_SIZE can now be set through the Makefile #12164

Uh oh!

Conversation

Tishj commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tishj commented May 21, 2024

Uh oh!

taniabogatsch commented May 21, 2024

Uh oh!

taniabogatsch left a comment

Choose a reason for hiding this comment

Uh oh!

Tishj commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taniabogatsch commented May 21, 2024

Uh oh!

carlopi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tishj commented May 21, 2024

Uh oh!

Tishj commented May 21, 2024

Uh oh!

Tishj commented May 22, 2024

Uh oh!

taniabogatsch commented May 27, 2024

Uh oh!

Mytherin commented May 31, 2024

Uh oh!

Uh oh!

[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile #12164

[Dev] `STANDARD_VECTOR_SIZE` and `BLOCK_ALLOC_SIZE` can now be set through the Makefile #12164

Tishj commented May 21, 2024 •

edited

Loading

Tishj commented May 21, 2024 •

edited

Loading