Skip to content

Conversation

mwylde
Copy link
Member

@mwylde mwylde commented Jan 8, 2025

Previously we were allowing delta-rs to create it own object stores with its own authentication handling. In order to get parity with our existing auth strategy for AWS (which relies on the official aws-config library to get consistency with aws-cli and other AWS tools), we were fetching a token at delta-rs creation time and passing that to the library.

This works for static, long-lived credentials, but fails for dynamically-refreshed credentials like those from IRSA. This PR reworks our use of delta-rs such that we use our own object_store structs, which properly handle credential refreshing.

While debugging this issue, I also discovered that because object_store uses its own HTTP client to make S3 requests (rather than using AWS smithy), there was no token caching being performed, leading to a huge volume of requests to Amazon STS and some additional latency for S3 operations. This PR also introduces our own sophisticated token cache for AWS, with support for lazy and asynchronous token refreshes.

@mwylde mwylde enabled auto-merge (squash) January 8, 2025 19:01
@mwylde mwylde merged commit 4014db4 into master Jan 8, 2025
6 checks passed
@mwylde mwylde deleted the irsa_expiration branch January 8, 2025 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant