👍 Thanks for this!

In the original code it seems like people could define the limit on the request (eg I just want the first 10 rows). If I'm understanding your changes correctly people now will always get all the rows regardless of limit as it is used as the page size. So passing the limit parameter would just affect the internal paging size. Shouldn't we separate limit and page?

You're right. Are those parameters documented? (Do we want to keep them?) The way they were written made it easy to consume all the memory on the web server if a user requests enough rows.

I can add them back but I would like to document them at least.

There should probably be a hard limit either way, with params or without, even if it's a big one (1m ?)
Your offset is not actually used on the following loops if there is an offset passed as param. I fine dropping support for the params to simplify things.
What would probably be more useful for selective dumps is allow an option to download as CSV on datastore_search.

I agree, a datastore_search CSV output option would be more flexible. It doesn't fit the action API protocol though. Ideally we could get any of our (paginated or not) API calls' data returned as a CSV streamed back to the end user.

Let me just add those options back here in case someone is using them.

On another topic: Should I be including a UTF-8 BOM here so that Excel can open these files? It would be a user-visible change. Add an option for that at the same time?

I personally would not add it by default. I'm happy with an option to enable it optionally.

There's no reason for a hard limit on this call. If the client keeps reading the data this code will continue looping to the next 10k records until the end. There's no benefit in making the client send multiple requests.

Fair enough, we'll assume there are no memory leaks there :)

@wardi regarding incompatibility with the existing api, we could support Accept: text/csv with a utility wrapper around certain actions. A quick LazyCSVWriter that doesn't write the headers until the first dict/row is consumed to get the structure? Makes the assumption that every row will be identical.