-
Notifications
You must be signed in to change notification settings - Fork 48
Description
Hi @nietras
This is a really cool library. Full disclosure, I only have a few years of programming experience, so there might be a way to do this I just haven't identified.
As far as I can tell, Sep's internal buffer can't be cleared. This makes it impossible to seek through the file stream to a specific point in the file (whilst maintaining the header information etc).
I wish to be able to create maps of large files. E.g. a Dictionary<TKey, long> where the long is the raw byte offset into the file for the start of a row/record. I'm able to do this with csv helper as it uses a plain streamreader which we can seek through, and discard the buffered values. (this is helpful for indexing large files and doing random/out of order reads without having it all in memory). However the same trick doesn't work on Sep.
As far as I can tell this is due to the array pool mechanism or some other internal buffer but I dont fully understand it. All i can see is that manually making the stream reader and then seeking on it and clearing the buffer data doesn't seem to work.
Would it be difficult to implement this feature? Just the raw ability to seek the underlying stream. I can manage the byte offsets externally.