command/run: handle large tokens in scanner #438
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #435, fixes #436
Sync uses
run
interface to execute commands. Run scans givenio.Reader
and creates a command from each line. If sync finds multiple files to remove, generated command string exceeds the maximum token size of bufio.Scanner and we are gettingbufio.Scanner: token too long
errors. As described in the issues,sync
fails if it tries to remove a large number of files.Official go documentation recommends using
bufio.Reader
for handling large tokens:This PR fixes two bugs:
Currently, scanner errors are ignored, and commands silently fail if we encounter scanner errors. Always print scanner errors before exiting the command.
Use
bufio.Reader
instead ofbufio.Scanner
to handle large tokens.