Fix ordering issues in EXPORT DATABASE #10677

Tishj · 2024-02-15T07:05:14Z

Supersedes #8619

This PR fixes https://github.com/duckdblabs/duckdb-internal/issues/120
This PR fixes #8496
This PR fixes #9713

This PR will:

Remove limitations on Alter, no Alter will be blocked by existing dependencies anymore(1)
Add dependencies for Foreign Key constraints.
Contain all the improvements regarding EXPORT DATABASE and Dependency management made by Fix EXPORT/IMPORT DATABASE schema.sql order #8619

We introduce LogicalDependency as a precursor to PhysicalDependency, this merely contains the information needed to look up the catalog entry, so it's safe to serialize/deserialize

Inside SingleFileCheckpointWriter::CreateCheckpoint() there was a duplicate of GetNaiveExportOrder, this now uses the DependencyManager::GetExportOrder method

Views now create a dependency on the table they're referencing (as long as they are in the same catalog)
This changes some existing behavior, but is in line with postgres behavior

postgres=# drop table tbl;
ERROR:  cannot drop table tbl because other objects depend on it
DETAIL:  view vw depends on table tbl
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

1: Indexes are not included in this, this requires some additional work that needs to be done outside of this PR

…te the catalog entries

…inders CatalogEntryRetriever, add tests with created types and dependencies created by them

… dependencies of the generated columns are registered as dependencies of the table

… depends on it

…, don't make any cross-catalog dependencies, detect and properly error when CREATE OR REPLACE statements depend on itself

…t test

…recreated by GetInfo in the CatalogEntry

src/catalog/catalog_entry/duck_schema_entry.cpp

Tishj

Some things I noticed:
(been a while since I actively worked on this)

src/catalog/catalog_entry/duck_schema_entry.cpp

src/catalog/dependency_manager.cpp

src/catalog/duck_catalog.cpp

src/execution/operator/persistent/physical_export.cpp

test/sql/attach/attach_export_import.test

test/sql/catalog/function/test_recursive_macro.test

test/sql/constraints/foreignkey/test_fk_export.test

test/sql/copy_database/copy_database_fk.test

test/sql/storage/catalog/test_view_explicit_aliases.test

…changes

…powered_export

…w we want to handle this in the future, we want to preserve some dependencies, but not all because dropping a column for example could result in dropping dependencies

…powered_export

Tishj · 2024-03-13T14:20:32Z

# Because this is part of a CTAS we don't register this as a dependency..
statement maybe
CREATE OR REPLACE TABLE t2 AS (
	SELECT struct_pack(a := -54124033386577348004002656426531535114)::my_struct FROM t2 LIMIT 70%
);
----
write-write conflict

One thing I discovered while playing around with this, in this case we would want to register the type as a dependency, but not the table used, currently we don't register any of them as a dependency.

…is no way to introduce a dependency into a CTAS

…powered_export

…s before export (#14196) This PR fixes duckdblabs/duckdb-internal#3050 This PR is a revival of #10677 We use GetNaiveExportOrder as we already do, but now we use `DependencyManager::ReorderEntries` to make sure a dependent entry gets exported first if the dependency manager is aware that there is a dependency on the entry.

Tishj added 30 commits August 17, 2023 16:24

use the DependencyManager to determine what the right order is to wri…

800873a

…te the catalog entries

use the Bind to get dependencies for macro functions

4a1d973

add dependency tracking for Views

5f354dc

register dependencies for indexes

028b23b

add test for exporting indexes with dependencies

a0f6820

update test to properly check that the index was succesfully imported

e2fbe57

make BindLogicalType a non-static method, so we can make use of the B…

ebbac57

…inders CatalogEntryRetriever, add tests with created types and dependencies created by them

add a dependency to 'export_generated_columns.test' and make sure the…

2d7076c

… dependencies of the generated columns are registered as dependencies of the table

verify that the macro can not be dropped because the generated column…

ca34bf2

… depends on it

add extra coverage for different types aliased with 'CREATE TYPE'

efc5797

forgot one

d9ebc96

fix up some compilation issue

dd50e43

remove calls to debug functions

1657016

Merge branch 'master' into macro_function_dependencies

edabfb3

dont populate dependencies when binding the query for CREATE TABLE AS…

0f3f02f

…, don't make any cross-catalog dependencies, detect and properly error when CREATE OR REPLACE statements depend on itself

fix broken tests related to views

dbc192a

set the right catalog in bind_export, improve the attach_export_impor…

509c055

…t test

fix failing slow tests

6bf88ef

fix unused variable error

5adeb14

cascade the deletes in python test fixture

e2c5ff7

skip the foreign key export test for now

141dccf

remove semicolon

dd2f99e

performance-move tidy fix

bc82756

remove some code that can never be hit

89d7eb7

the dependencies of the CreateInfo were getting lost, wasn't getting …

5fa7218

…recreated by GetInfo in the CatalogEntry

Merge branch 'main' into macro_function_dependencies

87c6c6c

implement format (de) serialize for DependencyList

187e056

wrap the reference up in an object in the serialize/deserialize code

dc9dced

fix test with recursive view

30c3712

another broken view test

b8a97d3

Tishj requested a review from Mytherin February 15, 2024 11:55

Tishj commented Feb 19, 2024

View reviewed changes

src/catalog/catalog_entry/duck_schema_entry.cpp Outdated Show resolved Hide resolved

Tishj commented Feb 19, 2024

View reviewed changes

Tishj added 5 commits February 19, 2024 12:55

no longer required

788ae5b

unused variable

928d671

reserve once

c623623

add DUCKDB_API, just in case

d9df9f1

use an unordered map of catalog sets instead, less fragile to future …

4835161

…changes

github-actions bot marked this pull request as draft February 19, 2024 12:43

Tishj added 6 commits February 23, 2024 10:30

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

3384c7d

…powered_export

remove the dependencies from AlterObject for now, unused. Not sure ho…

f461019

…w we want to handle this in the future, we want to preserve some dependencies, but not all because dropping a column for example could result in dropping dependencies

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

1730d7a

…powered_export

add test for cross-catalog views

9d4b1ed

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

be12207

…powered_export

add a check that it also errors if names dont match

d481d82

Tishj added 11 commits March 13, 2024 15:30

remove fixme in concurrent catalog usage that can not be done, there …

be71695

…is no way to introduce a dependency into a CTAS

no need to create new methods for this

8260738

no need for this to be a pointer

cdbf3c5

update enums

184c47f

dont generate enum utils for this

f752c10

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

5d7e578

…powered_export

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

e8a46bb

…powered_export

revert change to how catalog sets are stored

cd3af36

remove unnecessary changes

c8491a4

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

aab530d

…powered_export

Merge remote-tracking branch 'upstream/main' into dependency_manager_…

19f3361

…powered_export

Tishj closed this May 30, 2024

Tishj mentioned this pull request Oct 2, 2024

[Export/Import] Use the DependencyManager to (stable) sort the entries before export #14196

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix ordering issues in EXPORT DATABASE #10677

Fix ordering issues in EXPORT DATABASE #10677

Uh oh!

Tishj commented Feb 15, 2024 •

edited

Loading

Uh oh!

Uh oh!

Tishj left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tishj commented Mar 13, 2024

Uh oh!

Uh oh!

Fix ordering issues in EXPORT DATABASE #10677

Fix ordering issues in EXPORT DATABASE #10677

Uh oh!

Conversation

Tishj commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Tishj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tishj commented Mar 13, 2024

Uh oh!

Uh oh!

Tishj commented Feb 15, 2024 •

edited

Loading