[Python] Pandas Analyzer no longer trips up when the pandas_analyze_sample would only let it find nulls.
#9811
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Mytherin
reviewed
Nov 27, 2023
Mytherin
reviewed
Nov 27, 2023
…and first object is none
|
Thanks! |
|
Question, would it also be possible to add an |
|
+1 for having a way to tell duckdb to use the schema of an existing table, instead of doing inference on the data frame content. I ran into this issue when importing data into tables I already set up with schema. To mitigate, I ended up setting the sample window to a very large value to practically guarantee seeing a non-null value. |
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
Dec 14, 2023
Merge pull request duckdb/duckdb#9802 from Giorgi/master Merge pull request duckdb/duckdb#9811 from Tishj/pandas_analyzer_skip_nulls Merge pull request duckdb/duckdb#9827 from Mytherin/cifix Merge pull request duckdb/duckdb#9826 from carlopi/verboseerror
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes #6669
The logic for getting the offset for sampling is:
This let's us scan
sample_sizecolumns from the dataset.A problem with this is that if there are null values recurring in the dataset at this offset, the analyzer would only find nulls and set the type as NULL, when a non-null value is encountered it's cast to the result type NULL and that causes this issue.
To combat this issue we now find the first non-null value starting from that the offset, instead of just taking the first value we find at that offset.