Feature Proposal: JSON Result Output

Background

When using strict-s3-sync in CI/CD pipelines for SSG (Static Site Generator) and Headless CMS deployments, it's crucial to determine whether any files have actually changed. This information is needed to:

1. Decide whether CDN cache invalidation is necessary
2. Minimize unnecessary cache invalidation costs
3. Reduce load on the Headless CMS
4. Enable automated decision-making in deployment pipelines

Proposal

Add a new CLI option --result-json-file <path> that outputs the sync operation results to a JSON file.

JSON Schema

{
  "create": [
    "assets/new-image.png",
    "pages/new-page.html"
  ],
  "update": [
    "index.html",
    "styles/main.css"
  ],
  "delete": [
    "pages/old-page.html",
    "assets/unused.jpg"
  ],
  "skip": [
    "favicon.ico",
    "robots.txt",
    "sitemap.xml"
  ],
  "errors": [
    {
      "path": "large-file.bin",
      "error": "RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period"
    }
  ]
}

Key Design Decisions

1. Simple Path Arrays: Most categories (create, update, delete, skip) are simple string arrays containing file paths for easy processing.

2. Structured Errors: The errors array contains objects with path and error message for debugging purposes.

3. Minimal Structure: Focused on the essential information needed for CDN invalidation decisions.

Use Case Example

# In CI/CD pipeline
strict-s3-sync ./dist s3://cdn-bucket/site/ --delete --result-json-file sync-result.json

# Check if any changes occurred
CHANGES=$(jq '.create + .update + .delete | length' sync-result.json)

if [ "$CHANGES" -gt 0 ]; then
  # Extract changed paths for CDN invalidation
  jq -r '.create[], .update[], .delete[]' sync-result.json | \
    xargs -I {} aws cloudfront create-invalidation \
      --distribution-id $DIST_ID \
      --paths "/{}"
fi

Alternative: Include Summary

For convenience, we could also include a summary object:

{
  "create": ["file1.html"],
  "update": ["file2.css"],
  "delete": ["file3.js"],
  "skip": ["file4.txt"],
  "errors": [],
  "summary": {
    "hasChanges": true,
    "totalChanges": 3
  }
}

This would simplify the check to:

if jq -e '.summary.hasChanges' sync-result.json > /dev/null; then
  # Invalidate CDN
fi

Implementation Considerations

1. Path Format: Use S3 keys (without s3://bucket/ prefix) for consistency with CloudFront invalidation paths.

2. Atomic Write: Write to a temporary file first, then rename to ensure the JSON file is always complete.

3. Error Handling: If JSON file writing fails, log a warning but don't fail the sync operation.

4. Empty Arrays: Include all arrays even if empty for consistent structure.

Benefits

1. Simplicity: Minimal JSON structure focused on the essential use case
2. Easy Integration: Simple arrays are easy to process in any language
3. CloudFront Ready: Path format matches CloudFront invalidation requirements
4. Cost Effective: Enables precise invalidation of only changed files

Future Considerations

• Option to include checksums for verification
• Option to output JSONL (JSON Lines) for streaming large results
• Option to filter which categories to include in output