[CSV Reader] Detect SQLNULL types for schema merging, use schema merging in csv relations, add files_to_sniff option. #17467
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks! |
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 21, 2025
[CSV Reader] Detect SQLNULL types for schema merging, use schema merging in csv relations, add files_to_sniff option. (duckdb/duckdb#17467) [Python Dev] Fix failing tests for the Python SQLLogicTester (duckdb/duckdb#17529)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 21, 2025
[CSV Reader] Detect SQLNULL types for schema merging, use schema merging in csv relations, add files_to_sniff option. (duckdb/duckdb#17467) [Python Dev] Fix failing tests for the Python SQLLogicTester (duckdb/duckdb#17529)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 21, 2025
[CSV Reader] Detect SQLNULL types for schema merging, use schema merging in csv relations, add files_to_sniff option. (duckdb/duckdb#17467) [Python Dev] Fix failing tests for the Python SQLLogicTester (duckdb/duckdb#17529)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 23, 2025
[CSV Reader] Detect SQLNULL types for schema merging, use schema merging in csv relations, add files_to_sniff option. (duckdb/duckdb#17467) [Python Dev] Fix failing tests for the Python SQLLogicTester (duckdb/duckdb#17529)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Detect SQLNULL types for schema merging
Previously, we would always upcast a
SQL NULLtype toVARCHAR. We now keep it as aSQL NULLuntil merging the schemas of multiple files, so we can upcast to a type more refined thanVARCHAR. This only happens if we sniffed the entire file. Otherwise, there's no guarantee that a given column is alwaysNULL.Use schema merging in csv relations
When using CSV relations, the
auto_detectcode didn’t use the schema merging algorithm for multiple files, which could cause inconsistent behavior between queries and relations. This PR changes it so both now use the same code.files_to_sniffoptionWhen sniffing and merging multiple schemas, we previously had a hard-coded limit of 10 files. This is now configurable: users can set the limit themselves. Setting it to -1 will detect and merge the schemas of all files used by the scanner. By default, the value remains 10.
Fix: #17452
Fix: #17451