Skip to content

[FEATURE] Read files content lazily #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Aug 11, 2023
Merged

[FEATURE] Read files content lazily #14

merged 13 commits into from
Aug 11, 2023

Conversation

nlepage
Copy link
Owner

@nlepage nlepage commented Aug 25, 2022 •
edited
Loading

  • Stop reading all files content in memory
  • Read whole tar file once and record position of each file
  • Create a specific tar.Reader each time a file needs to be read
  • If the given io.Reader isn't an io.ReaderAt, fallback to reading the whole tar file in a buffer
  • Reimplement io.Seeker over tar.Reader to allow usage with http.FS

@nlepage nlepage added the enhancement New feature or request label Aug 25, 2022
@nlepage nlepage self-assigned this Aug 25, 2022
@nlepage nlepage force-pushed the feature/lazy-read branch from 50090ca to 8b97630 Compare August 25, 2022 14:00
@nlepage nlepage mentioned this pull request Aug 25, 2022
Closed
@nlepage nlepage force-pushed the feature/lazy-read branch 8 times, most recently from 3cdc06e to b8b88e1 Compare August 26, 2022 15:13
@joelanford joelanford mentioned this pull request Jul 21, 2023
Closed
@nlepage nlepage force-pushed the feature/lazy-read branch 2 times, most recently from 149705a to 687c368 Compare July 25, 2023 07:22
@nlepage nlepage force-pushed the feature/lazy-read branch 5 times, most recently from 407e63e to 4470b8a Compare July 27, 2023 14:18
joelanford
joelanford reviewed Aug 3, 2023
View reviewed changes
nlepage and others added 8 commits August 9, 2023 22:39
@nlepage
Add benchmarks
e146caf
@nlepage
✨ Read files content lazily
b02b3cd
@joelanford @nlepage
keep track of offset of each entry's data
fcc48bd 
use the offset to avoid iterating tar file when opening each file.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
@joelanford @nlepage
if the reader is not a readerAt, read it into a buffer
073d7f4 
after we've read the non-readerAt reader into a buffer, we now have a
readerAt either way, which simplifies code and no longer requires
separate buffers per file entry.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
@nlepage
Improve ReadFile test
8481a4a
@nlepage
Fix reading sparse files
588ec2b 
Reading file content straight from the underlying reader was too straightforward.

Instead read from a new tar.Reader with an underlying reader pointing straight to the tar entry we want to read.
@nlepage
Use composition in countingReader
da1d1fc
@nlepage
Forward Seek method on ReadCounter when possible
da24e69 
Should improve initial read of the tar file.
@nlepage nlepage force-pushed the feature/lazy-read branch from 30ec627 to 2aa91f4 Compare August 9, 2023 20:42
joelanford
joelanford reviewed Aug 11, 2023
View reviewed changes
@nlepage nlepage marked this pull request as ready for review August 11, 2023 20:18
@nlepage nlepage merged commit c48d39a into main Aug 11, 2023
@nlepage nlepage deleted the feature/lazy-read branch August 11, 2023 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@joelanford joelanford joelanford left review comments

Reviewers whose approvals may not affect merge requirements
Assignees

@nlepage nlepage

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nlepage @joelanford