Skip to content

Conversation

Mytherin
Copy link
Collaborator

Follow-up from #12908

Allow dynamically generated join filters to be pushed through UNION, UNNEST and AGGREGATE nodes in the plan.

For UNION, this means we need to potentially push filters into multiple table scans. For example, in this case we push the filter into both scans over lineitem:

CREATE VIEW double_lineitem AS
FROM lineitem
UNION ALL
FROM lineitem;

SELECT *
FROM double_lineitem
WHERE l_orderkey=(SELECT MAX(l_orderkey) FROM lineitem) ORDER BY ALL

We can push filters through aggregates as well, as long as the filters are on grouping columns. However, in this case, the pipeline dependencies are not correctly set up yet to ensure the join upstream is executed before the aggregate - meaning the filter has no effect yet as the aggregate is computed before we know which rows we can filter out (CC @lnkuiper perhaps something you want to look into at some point).

@Mytherin Mytherin merged commit 2a9d6d7 into duckdb:feature Oct 21, 2024
41 checks passed
@Mytherin Mytherin deleted the joinfilterunion branch December 8, 2024 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant