Skip to content

Conversation

Tishj
Copy link
Contributor

@Tishj Tishj commented Feb 16, 2023

This PR adds a couple methods to the C-API to create and consume a stream query result.

The pending family of functions has a new addition:

duckdb_state duckdb_pending_prepared_streaming(duckdb_prepared_statement prepared_statement,
                                               duckdb_pending_result *out_result);

This creates a duckdb_pending_result, like before, the pending query result can be partially executed with
duckdb_pending_execute_task.
The only difference is that to execute the pending query and turn it into a streaming query result, the following function needs to be called.

duckdb_state duckdb_create_streaming_result(duckdb_pending_result pending_result, duckdb_result *out_result);

Similar to duckdb_execute_pending this creates a result object out of the pending query result.
(also note that this pending query result can not be turned into a regular result)

To interact with the result, most of the materialized-result family of functions can be used, with the exception of the chunk/row related functions.
To get a chunk out of the stream query result, instead you have to use:

duckdb_data_chunk duckdb_stream_fetch_chunk(duckdb_result result);

@Tishj
Copy link
Contributor Author

Tishj commented Feb 16, 2023

Writing the description, I had a thought, we could also support creating a regular (Materialized) result out of the pending query, then we'll just get the result and call Materialize on it.

EDIT:

duckdb_state duckdb_execute_pending(duckdb_pending_result pending_result, duckdb_result *out_result);

Now also supports a streaming result, it just instantly calls Materialize on it.

Maybe we also want to have a duckdb_result_materialize(duckdb_result result) method which can be called on a streaming result to materialize it even after chunks have been taken from it?

@Tishj
Copy link
Contributor Author

Tishj commented Feb 21, 2023

I wonder if these failures are related to my changes or just spurious
https://github.com/duckdb/duckdb/actions/runs/4232582133/jobs/7354070411 (rate-limit related probably)
https://github.com/duckdb/duckdb/actions/runs/4232582132/jobs/7354613914 (???) ubuntu 20.04 so it's not the end-of-life issue

@Mytherin
Copy link
Collaborator

Mytherin commented Feb 21, 2023

The WebAssembly one has been borked since yesterday, @carlopi is on it. The other does not seem related either

@Mytherin Mytherin changed the base branch from master to feature February 21, 2023 15:49
Copy link
Collaborator

@Mytherin Mytherin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good - one minor comment.

@Mytherin Mytherin merged commit 62d90aa into duckdb:feature Feb 22, 2023
@Mytherin
Copy link
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants