Skip to content

Integrate S3 Storage Backend #919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 58 commits into from
Sep 4, 2025
Merged

Integrate S3 Storage Backend #919

merged 58 commits into from
Sep 4, 2025

Conversation

atimin
Copy link
Member

@atimin atimin commented Sep 2, 2025 •
edited
Loading

Closes #906

Please check if the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • CHANGELOG.md has been updated (for bug fixes / features / docs)

What kind of change does this PR introduce?

Feature

What was changed?

What change?

This pull request integrates an S3 storage backend into ReductStore. With this change, users are able to configure ReductStore to use Amazon S3 (or S3-compatible) object storage as a backend for storing and retrieving data. The update includes:

  • Implementation of an S3 storage layer with support for basic bucket and object operations.
  • Configuration options for connecting to S3, including access/secret keys, bucket name, region, endpoint, and other relevant parameters.
  • Integration of the S3 backend into the storage abstraction, allowing seamless switching between local and S3-based storage.
  • Error handling, logging, and reconnection logic for S3 operations.
  • Updates to documentation and configuration guides to reflect S3 support.
  • Unit and integration tests covering the S3 backend functionality.

This change enables ReductStore to scale storage beyond local disks, making it suitable for cloud-native deployments and large-scale data retention.

Related issues

#906

Does this PR introduce a breaking change?

No

Other information:

@atimin atimin linked an issue Sep 2, 2025 that may be closed by this pull request
Feature: S3 Storage Backend #906
Closed
1 task
@atimin atimin marked this pull request as draft September 2, 2025 15:06
@codecov Codecov
Copy link

codecov bot commented Sep 2, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 95.86039% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.41%. Comparing base (c8e4882) to head (0433c44).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
reductstore/src/backend/remote/s3_connector.rs 94.66% 15 Missing ⚠️
reductstore/src/core/file_cache.rs 91.30% 8 Missing ⚠️
reductstore/src/backend/remote.rs 97.04% 5 Missing ⚠️
reductstore/src/backend/remote/local_cache.rs 96.07% 4 Missing ⚠️
reductstore/src/cfg/provision/bucket.rs 42.85% 4 Missing ⚠️
reductstore/src/storage/entry/entry_loader.rs 84.00% 4 Missing ⚠️
reductstore/src/backend.rs 98.03% 3 Missing ⚠️
reductstore/src/backend/fs.rs 92.68% 3 Missing ⚠️
reductstore/src/cfg.rs 89.28% 3 Missing ⚠️
reductstore/src/cfg/remote_storage.rs 95.45% 1 Missing ⚠️
... and 1 more 
@@            Coverage Diff             @@
##             main     #919      +/-   ##
==========================================
+ Coverage   95.35%   95.41%   +0.05%     
==========================================
  Files         167      175       +8     
  Lines       10042    11083    +1041     
==========================================
+ Hits         9576    10575     +999     
- Misses        466      508      +42
Files with missing lines Coverage Δ
reduct_base/src/batch.rs 100.00% <100.00%> (ø)
reductstore/src/api.rs 82.19% <ø> (ø)
reductstore/src/api/entry/common.rs 98.15% <100.00%> (ø)
reductstore/src/api/entry/write_batched.rs 91.93% <100.00%> (ø)
reductstore/src/auth/token_repository.rs 88.88% <100.00%> (+0.58%) ⬆️
reductstore/src/backend/file.rs 100.00% <100.00%> (ø)
reductstore/src/backend/noop.rs 100.00% <100.00%> (ø)
reductstore/src/main.rs 86.59% <ø> (ø)
reductstore/src/replication/replication_sender.rs 98.97% <100.00%> (ø)
reductstore/src/replication/transaction_log.rs 100.00% <100.00%> (ø)
... and 21 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@atimin atimin mentioned this pull request Sep 4, 2025
Merged
@atimin atimin requested a review from Copilot September 4, 2025 12:18
@atimin atimin marked this pull request as ready for review September 4, 2025 12:18
Copilot
Copilot AI reviewed Sep 4, 2025
View reviewed changes
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request integrates S3 storage backend support into ReductStore, enabling cloud-native deployments with remote object storage. The implementation adds a comprehensive storage abstraction layer that allows seamless switching between local filesystem and S3-compatible storage backends.

Key changes include:

  • Addition of a new storage backend abstraction with filesystem and S3 implementations
  • Implementation of a local cache system for S3 backend to optimize performance
  • Configuration support for S3 credentials, endpoints, and cache management
  • Migration from usize to u64 for content sizes to handle larger files

Reviewed Changes

Copilot reviewed 37 out of 38 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
reductstore/src/backend/ New storage backend abstraction with filesystem and S3 implementations
reductstore/src/cfg/remote_storage.rs Configuration parsing for S3 backend settings
reductstore/src/core/file_cache.rs Enhanced file cache to work with storage backends
reductstore/src/storage/ Updated storage layer to use new backend abstraction
reductstore/src/api/ Updated content size types from usize to u64

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@atimin atimin merged commit 51ca9b3 into main Sep 4, 2025
110 of 111 checks passed
@atimin atimin deleted the 906-feature-s3-storage-backend branch September 4, 2025 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature: S3 Storage Backend
1 participant
@atimin