These are for debugging purposes.
What would be the best way to keep them around so they can be useful in the future without causing unnecessary friction?

Would it be desirable to expose this (in some form) to SQL?

Idea is that then it can be tested / and being present there for debug purposes.

That's not a bad idea, maybe we could make two table functions that output a column for every catalog entry, containing a VARCHAR[] containing the names of the dependents/dependencies

Maybe take an optional catalog VARCHAR parameter

Edit:

I thought about it a little more, this might be a better structure
Columns:

• Name (of the entry)
• Connections (the list of dependencies/dependents)
• Type ("DEPENDENCIES" or "DEPENDENTS")
• Catalog (name of catalog the entry belongs to)
• Schema

Then we could fit it into a single table function

dependency_manager()

Btw I just found that we have duckdb_dependencies() table function, but it seems kind of lacking

These need to be removed

These might be a little too terse, but I didn't want to duplicate all of the "helper" functions that Catalog has.

In the future it might be better to move all of the catalog entry look up responsibility to CatalogEntryRetriever, that would also make the Catalog class a little lighter.

This is no longer a static method because the binding of USER types perform a CatalogEntry lookup which we want to perform the CatalogEntryRetrievers callback on.

In the places that previously used this as a static method without being inside a Binder or having access to a binder I've created a binder on the spot.
In a lot of these places a binder was already created later, so I don't think any harm was done here.

These methods were made in an attempt to make it easier to work with the dependency_map and dependents_map, because it's logically a little hard to understand what connections they represent.

Sadly we can't get rid of this error-prone method entirely, though I haven't seen any other classes that derive from Catalog, but this could likely be a point of interest for extensions later on

Maybe GetExportOrder should be a virtual method, so it can be overridden by an extension catalog ?

This implementation is potentially O(n^2), not sure we care enough and it's not immediate to me what an alternative implementation would be.

This implementation is potentially O(n^2), not sure we care enough and it's not immediate to me what an alternative implementation would be.

Yea it's pretty bad, I figured EXPORT DATABASE isn't very performance sensitive and it's better to focus on correctness

But it could definitely be improved

If we turn this into a stack instead and push every dependency to the front, this should be slightly more efficient, as we won't keep checking the same object if the dependencies of the object also have dependencies

now:
obj has dependencies:

state of the queue:

dep1
dep2
dep3
obj

dep1 has dependencies, push these, state of the queue:

dep2
dep3
obj
dep1.1
dep1.2
dep1.3
dep1

Once we reach obj again, we will push dep1 and the rest of its dependencies again (dep2, dep3)

With a stack:
obj has dependencies:

dep3
dep2
dep1
obj

dep1 has dependencies, push these:

dep1.3
dep1.2
dep1.1
dep1
obj

We will naturally deal with all dependencies first before checking an object whos dependencies have not been processed yet

to get the order don't you just do a depth first search over the entries added to the backlog? my c++ is rusty but something like:

void DependencyManager::DepthFirstSearch(catalog_entry_set_t &explored, catalog_entry_vector_t &order, the_type &entry)
{
        if (explored.count(entry) > 0) {
                return;
        }
        explored.insert(entry);
        auto deps = dependencies_map.find(entry);
        if (deps != dependencies_map.end() {
                for (auto &dependency : deps->second) {
                        DepthFirstSearch(explored, order, dependency);
                }
        }   
        order.push_back(entry);
}

catalog_entry_vector_t DependencyManager::GetExportOrder() {
        catalog_entry_set_t explored;
        catalog_entry_vector_t export_order;

        for (auto &entry : dependencies_map) {
                DepthFirstSearch(explored, export_order, entry.first);
        }
        return export_order;
}

I wanted to avoid recursion, to not run the risk of a stack overflow.
But maybe I was being too cautious for no good reason here

I think my proposed change to use a stack would mimic your depth first search approach though

i don't quite follow the explanation with the stack / queue described. For a non-recursive DFS there is an implementation on Wikipedia, https://en.wikipedia.org/wiki/Depth-first_search#Pseudocode, that uses a stack of iterators. That would work to give a topological sort -- appending the element to the output vector at the last pop in the pseudo-code.

here's my attempt, i was bored.
https://github.com/robert-manchester/duckdb/blob/7de8d5dd272737d6397a30affebc20ceae32fc87/src/catalog/dependency_manager.cpp#L197

Can we not do the change where we push the ClientContext into the constructor of every single entry, and instead do the resolution of the dependency list (from logical -> physical) during the binding or e.g. in the schema catalog entry?

Big change request, but would it be possible to perhaps (this should likely be an entirely separate PR or change) remove the physical dependencies altogether - from the dependency manager and elsewhere - and instead always work with logical dependencies (e.g. a pair of CatalogType, schema_name, entry_name)? This should both solve the original problem of not being able to serialize dependencies, as well as simplify the bookkeeping around dependencies in general and make it less error prone.