Thanks @wardi,

Of all these approaches, which one is most amenable to being merged into CKAN core?

Whenever we do an implementation, we always strive to make it something that can be merged (e.g. the initial data dictionary was informed by Boston's use cases, but was eventually incorporated into core).

This is not only in an effort to give back to the community in the spirit of open source, but pragmatically speaking, it is also in our "enlightened self-interest" to having something that's maintained and kept current by the project.

If there is no interest to incorporate it into core, which approach would you recommend if we create a separate extension to promote re-use and wider adoption (and co-maintenance) by the community (so, we don't just code it into the site's theme extension).

All four approaches above assume no core changes, but some parts might be made easier by changes to core.

• If core allowed reordering the resource and dataset creation pages then an extension wouldn't need to copy a big chunk of the flask view code from core just to override a few lines.
• If ckan's upload process was separated from resource creation then creating a task that analyzed the columns before loading into the datastore would be easier. IIRC @TkTech started some work in this direction: files are uploaded to a user's account, then attached to resources afterwards. Making file uploads a first class thing in ckan would also simplify attaching files to other things (like groups, orgs, views, datasets themselves ...). This is a large effort sort of change but could be really useful.
• If ckan's dataset + resource create/update process was converted into a single page, (e.g. with javascript callbacks to package_revise for all fields to allow immediate updates, safe concurrent editing, server-side error checking and no save button required) then reordering resource/dataset creation and adding additional pages/forms to the process would be much easier. This is also a large effort sort of change but it's something that would really improve ckan's form UX/UI possibilities.
• for option 4 datapusher/xloader could add a plugin interface to make it easier to insert processing jobs. This would be a pretty easy addition that would also reduce code duplication.

Thanks @wardi ,
We'll go with option 2 + option 4 then ( new flask view for resource uploads first + new background job ), with a new extension and a fork of datapusher.

We did some work to make datapusher use COPY last year (ckan/datapusher#221), making it 10x faster. We'll build on that and replace messytables with qsv, so we don't even need to insert everything as a string with COPY, as we are guaranteed the inferred data types are correct by qsv.

WDYT?

You'll create a new view to handle the file upload and use that to create the resource, then allow new datapusher to process the file? Yes that should work.