What should we add?

This is an issues beyond the bug I reported in #10818. As we start to work with the transpiler and use the estimator more the current flow is rather complicated. While most users can just send abstract quantum circuits to the system I want to be able to send logical circuits to the system using the gate set and coupling map I have. Given I know what I want to do I can in general make a better quantum circuit than the default optimizations and/or I can work on a better definition of the defaults.

In my view we should remove layout from being bound to the transpile circuit and have a layout object and meta data returned to the user after they run the transpiler. I think the workflow should be

qc_transpile, layout, metadata = transpile(qc_example, options=)

personally I find the layout object over complex and should just be a list (I think @mtreinish shares my views here) and in the case that I need metadata from this transpliation I think we can put it all in a new object. I don't have strong opinions on it but some things that are interesting is to know how the layout was changed. Today we kinda get this as it is all put into a tranpilelayout object but it is not clear what pass did what (as I tried to explain in the bug #10818). My suggestion is that the meta data should contain

1. a ordered list of passes that are used on the circuit (this I would find helpful as I can then think about which ones I could improve)
2. It would be useful to know which passes changed the layout so someway of correlating that to the pass would be good. As @mtreinish points out all passes don't change the layout so having one for each pass is a waste
3. if approximations where used rather then exact some way of knowing which passes introduced errors would be good.
4. Timing of each pass and total time used by the transpiler.

These are the ones I can think of for now

Finally we need a new function in the compiler and when I am using the estimator I want to be able to run using the layout from the transpiler.

opts_transpile = permute_sparse_pauli_ops(opts, layout)

that way I can send both qc_transpile and opts_transpile to the estimator, while it is simple to write a few lines of code that is not the nicest user experience.

This is very much related to Qiskit/qiskit-ibm-runtime#338 and solution #9988 but it is fundamental for the workflows using the estimator not only the ability to add ancillas if the transpiler enlarges the system but if the layout changes the layout object needs to be used to redefine the observables and even used in the post-processing if I want the data to be changed back to my original circuit order. We need to make layout a variable that people want to work with.