Skip to content

Conversation

yuya-takeyama
Copy link
Owner

Summary

  • Fixed inconsistent handling of S3 URIs with trailing slashes
  • Normalized prefix processing using path.Clean() to ensure consistent path comparisons
  • Extracted prefix trimming logic into a dedicated function for better maintainability

Problem

When syncing files to S3, URIs with trailing slashes (e.g., s3://bucket/prefix/) were not handled consistently with URIs without trailing slashes (e.g., s3://bucket/prefix). This caused incorrect path comparisons during sync operations, as documented in the investigation notes.

Solution

  1. Normalize S3 prefixes: Use path.Clean() to normalize prefixes, removing trailing slashes and handling multiple slashes
  2. Extract helper function: Created trimS3KeyPrefix() function to centralize prefix trimming logic
  3. Comprehensive testing: Added extensive test coverage for edge cases including:
    • Multiple trailing slashes
    • Multiple internal slashes
    • Empty prefixes
    • Real-world examples

Test plan

  • Unit tests pass for parseS3URI with various edge cases
  • Unit tests pass for trimS3KeyPrefix function
  • Integration tests verify correct interaction between components
  • Manual testing with real S3 buckets (if needed)

Related Issues

Fixes the path comparison issues documented in the checksum investigation.

🤖 Generated with Claude Code

@yuya-takeyama yuya-takeyama force-pushed the yuya-takeyama/feat/fix-s3-checksum-comparison branch from 911e25e to 61852f2 Compare August 10, 2025 09:59
This fixes an issue where S3 URIs with trailing slashes were not handled
consistently, causing incorrect path comparisons during sync operations.

Changes:
- Use path.Clean() to normalize S3 prefixes, removing trailing slashes
- Extract prefix trimming logic into dedicated trimS3KeyPrefix function
- Add comprehensive test coverage for various edge cases

This ensures consistent behavior regardless of whether the S3 URI has
trailing slashes, matching the behavior of aws s3 sync.

Fixes issues reported in checksum_issue.md where paths were incorrectly
compared due to inconsistent prefix handling.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
@yuya-takeyama yuya-takeyama force-pushed the yuya-takeyama/feat/fix-s3-checksum-comparison branch from 61852f2 to 997e188 Compare August 10, 2025 09:59
@yuya-takeyama yuya-takeyama merged commit f109c61 into main Aug 10, 2025
1 check passed
@yuya-takeyama yuya-takeyama deleted the yuya-takeyama/feat/fix-s3-checksum-comparison branch August 10, 2025 10:01
@yuya-takeyama-tagpr yuya-takeyama-tagpr bot mentioned this pull request Aug 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant