Skip to content

Conversation

sonmezonur
Copy link
Member

Fixes #435, fixes #436

Sync uses run interface to execute commands. Run scans given io.Reader and creates a command from each line. If sync finds multiple files to remove, generated command string exceeds the maximum token size of bufio.Scanner and we are getting bufio.Scanner: token too long errors. As described in the issues, sync fails if it tries to remove a large number of files.

Official go documentation recommends using bufio.Reader for handling large tokens:

Programs that need more control over error handling or large tokens, or must run sequential scans 
on a reader, should use bufio.Reader instead.

This PR fixes two bugs:

  1. Currently, scanner errors are ignored, and commands silently fail if we encounter scanner errors. Always print scanner errors before exiting the command.

  2. Use bufio.Reader instead of bufio.Scanner to handle large tokens.

@sonmezonur sonmezonur requested a review from a team as a code owner April 21, 2022 22:22
@sonmezonur sonmezonur requested review from igungor and aykutfarsak and removed request for a team April 21, 2022 22:22
@sonmezonur sonmezonur added this to the v2.0.0 milestone Apr 21, 2022
@lozbrown
Copy link

lozbrown commented May 3, 2022

please can this be reviewed/merged and a beta 2 created there is a known bug in the current beta...

@sonmezonur sonmezonur merged commit c978e4a into master May 13, 2022
@sonmezonur sonmezonur deleted the run_handle_large_tokens branch May 13, 2022 12:31
@sonmezonur
Copy link
Member Author

@lozbrown New release is now available. Let us know if you still encounter the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sync: deletes not propagating and comparison different to aws cli sync fails with a large number of files to remove
4 participants