Skip to content

Glib json #480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 20, 2021
Merged

Glib json #480

merged 7 commits into from
Mar 20, 2021

Conversation

SeeSpotRun
Copy link
Collaborator

Note this PR was branched off earlier PR #479 to simplify possible merging.

Should address #464 and #463 and make json handling more robust in future.

One side-effect of this implementation is that the json file doesn't get written until right at the end. The previous implementation writes to file as the results come in.

To do similar buffering with json-glib would require manually writing the opening and closing [ and ] to file, as well as the ,'s between array elements, and using a new JsonGenerator for each element. So it can be done but it's less elegant. But if we we run into problems keeping the JsonArray in memory then we might have to go that way anyway.

Thoughts @sahib ?

@sahib
Copy link
Owner

sahib commented Mar 18, 2021

Hey @SeeSpotRun,

One side-effect of this implementation is that the json file doesn't get written until right at the end. The previous implementation writes to file as the results come in.

That's actually a crucial property for the following reasons:

  • JSON files can get very big (I saw several GB in edge cases), keeping it all in memory is not a good thing.
  • Current JSON has to be written on CTRL-C, for big files that will take long.
  • GUI depends on the streaming of JSON to show the results in realtime.

So, yes, I think you should go with JsonGenerator. IIRC you don't need to instance a new one on each element, but can just call json_generator_set_root() on each (or so I hope to avoid the extra allocation).

@SeeSpotRun
Copy link
Collaborator Author

Yeah I kind of realised that as soon as I documented the PR. Oh well, less elegant but not the end of the world.

@SeeSpotRun SeeSpotRun merged commit b1eea2c into sahib:develop Mar 20, 2021
@SeeSpotRun SeeSpotRun deleted the glib-json branch March 28, 2021 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants