Thanks! LGTM

I am still able to reproduce issues 41 and 44

Hmm @PedroTadim can you give a few more insights into your system, version, etc. ? I added the respective test cases, and the CI seems to pass, although they might sneakily bypassed through the one failing CI run (#6189) ...

Hmm @PedroTadim can you give a few more insights into your system, version, etc. ? I added the respective test cases, and the CI seems to pass, although they might sneakily bypassed through the one failing CI run (#6189) ...

For 41, I trigger it in the shell with a persisted database. The program throws an uncaught exception. Try to reduce the memory limit further down if no success. For 44, maybe the same trick will do it.

I can reproduce it if I further decrease the memory limit, but I think it is expected behavior. Mainly because the failing INSERT statement seems unrelated to whether there is an index. Looking at the queries below, DuckDB requests memory for a header and some blocks for the table, and then DuckDB tries to obtain memory for, I think, the hash table for the DISTINCT. The issue is also not reproducible when increasing the memory, and the exception thrown is accurate (DuckDB runs out of memory). If @Mytherin agrees, then I'd say that number 41 is fine?

# load the DB from disk
load __TEST_DIR__/create_idx.db

statement ok
CREATE TABLE t1 AS (SELECT 1 c1, 'a' c2);

#statement ok
#CREATE INDEX i1 ON t1 (c1);

statement ok
PRAGMA MEMORY_LIMIT='1MB';

statement ok
CHECKPOINT;

statement ok
INSERT INTO t1(c2) (SELECT DISTINCT 'b');

Query unexpectedly failed! (test/tmp.test:19)!
================================================================================
INSERT INTO t1(c2) (SELECT DISTINCT 'b');
================================================================================
Out of Memory Error: could not allocate block of 262144 bytes (786432/1000000 used) 

I don't manage to reproduce number 44, and I also added some lines to (I thought) fix the cause... I think without further insights/reproducible steps I cannot do much here at the moment.

I can reproduce it if I further decrease the memory limit, but I think it is expected behavior. Mainly because the failing INSERT statement seems unrelated to whether there is an index. Looking at the queries below, DuckDB requests memory for a header and some blocks for the table, and then DuckDB tries to obtain memory for, I think, the hash table for the DISTINCT. The issue is also not reproducible when increasing the memory, and the exception thrown is accurate (DuckDB runs out of memory). If @Mytherin agrees, then I'd say that number 41 is fine?

# load the DB from disk
load __TEST_DIR__/create_idx.db

statement ok
CREATE TABLE t1 AS (SELECT 1 c1, 'a' c2);

#statement ok
#CREATE INDEX i1 ON t1 (c1);

statement ok
PRAGMA MEMORY_LIMIT='1MB';

statement ok
CHECKPOINT;

statement ok
INSERT INTO t1(c2) (SELECT DISTINCT 'b');

Query unexpectedly failed! (test/tmp.test:19)!
================================================================================
INSERT INTO t1(c2) (SELECT DISTINCT 'b');
================================================================================
Out of Memory Error: could not allocate block of 262144 bytes (786432/1000000 used) 

The out-of-memory is expected, it is just the shell ending abruptly being the problem.

Oh I see! That's a problem, yes. Will have a look :)

For 44. I see the error is thrown when the DEBUG macro is enabled.

For 44. I just reproduced it again with:

CREATE TABLE t0 (c1 INT);
CREATE INDEX i0 ON t0 (c1);
INSERT INTO t0(c1) VALUES (1),(1),(1);
TRUNCATE t0;

What system are you on? Which duckdb version do you use? I just build the current master with DEBUG.

tania@laptop duckdb % build/debug/duckdb
v0.6.1-dev2764 f7827396d7
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D .open persist.duckdb
D CREATE TABLE t0 (c1 INT);
D CREATE INDEX i0 ON t0 (c1);
D INSERT INTO t0(c1) VALUES (1),(1),(1);
D TRUNCATE t0;
D 

I am on Ubuntu 22.04 compiling the latest source on master branch with sanitizer enabled and clang. Here is the cmake command:

cmake -DCMAKE_INSTALL_PREFIX=... -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS="-DDUCKDB_USE_STANDARD_ASSERT -DDUCKDB_CRASH_ON_ASSERT -fsanitize=address,undefined" -DCMAKE_CXX_FLAGS="-DDUCKDB_USE_STANDARD_ASSERT -DDUCKDB_CRASH_ON_ASSERT -fsanitize=address,undefined" -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address,undefined" -DBUILD_ICU_EXTENSION=ON -DBUILD_HTTPFS_EXTENSION=ON -DBUILD_PARQUET_EXTENSION=ON -DBUILD_TPCH_EXTENSION=ON -DBUILD_TPCDS_EXTENSION=ON -DBUILD_BENCHMARKS=ON -DBUILD_SQLSMITH=ON -DBUILD_TPCE=ON -DBUILD_JSON_EXTENSION=ON -DBUILD_SHELL=ON ...