We need a general category of synthesis algorithms for things that have exponential descriptions. This basically means things defined by Operator or Statevector. A start of this is under qiskit.quantum_info.synthesis, but a bunch of the algorithms are scattered under qiskit.extensions.quantum_initializer.

Then I think instead of defining these operators as circuit libraries (they are really not), we just have a way to insert operators into circuits, and rely on these synthesis algorithms to give us the corresponding decomposition. This would include: Diagonal, Statevector, Unitary, Clifford, Isometry, etc.