Hi,
I am using this library to traverse a bunch of directories containing 70TB worth of files of about 2MB each.

I noticed that my software is using more memory than it should and I profiled it to try and understand what's going on.

The library is only used on startup. I made sure it returns and waited 1 hour before using pprof to analyze the heap and this is what I noticed:

 2670.24MB 33.01% 66.04%  2670.24MB 33.01%  0000000000b5786b github.com/karrick/godirwalk.(*Scanner).Scan /home/travis/gopath/pkg/mod/github.com/karrick/godirwalk@v1.16.1/scandir_unix.go:163
         0     0% 66.04%  2670.24MB 33.01%  0000000000b57f45 github.com/karrick/godirwalk.Walk /home/travis/gopath/pkg/mod/github.com/karrick/godirwalk@v1.16.1/walk.go:258
         0     0% 66.04%  2670.24MB 33.01%  0000000000b585ba github.com/karrick/godirwalk.walk /home/travis/gopath/pkg/mod/github.com/karrick/godirwalk@v1.16.1/walk.go:329
         0     0% 66.04%  2670.24MB 33.01%  0000000000b59e2e github.com/lbryio/reflector.go/store/speedwalk.AllFiles.func2 /home/travis/gopath/src/github.com/lbryio/reflector.go/store/speedwalk/speedwalk.go:64

this is unusual because the function returned and the files were further processed later, there is no reason as to why godirwalk would still have to hold memory.

I tried looking into the code and I noticed that scandir_unix.go calls s.done() before returning false in Scan() which subsequently calls s.dh.Close() , however as the done() doc says:

// done is called when directory scanner unable to continue, with either the
// triggering error, or nil when there are simply no more entries to read from
// the directory.

the function is only called when the scanner is unable to continue which leads me to think that if something were to interrupt the walk before it finishes, memory could not be freed up.

this could happen when we return in the for loop here:

	for ds.Scan() {
		deChild, err := ds.Dirent()
		osChildname := filepath.Join(osPathname, deChild.name)
		if err != nil {
			if action := options.ErrorCallback(osChildname, err); action == SkipNode {
				return nil
			}
			return err
		}

which would not close the file that the scanner has opened leaving the scanner noncollectable by the GC.

If this happens enough times, this could potentially add up to the amount of memory I noticed while profiling (?)

The directory i'm walking has 256 subdirectories and about 115k files in each subdirectory.

Let me know what you think.

Regards,

Niko