-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
S3 API: simpler way to start s3 with credentials #7030
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Summary of Changes
Hello @chrislusf, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly simplifies the initial setup and configuration of the S3 API by introducing support for environment variable-based admin credentials. This change reduces reliance on explicit configuration files, making it easier to deploy and manage S3 instances, especially in containerized or automated environments. The new logic ensures seamless integration with existing identity management while providing clear precedence rules and robust testing.
Highlights
- Simplified S3 Admin Setup: Users can now configure S3 admin credentials directly via
WEED_S3_ADMIN_USER
andWEED_S3_ADMIN_PASSWORD
environment variables. This eliminates the need for a separate configuration file for basic setup, making deployment and management more convenient. - Environment Variable Credential Loading Logic: A new mechanism has been implemented to detect and load admin credentials from environment variables during S3 API initialization. If both
WEED_S3_ADMIN_USER
andWEED_S3_ADMIN_PASSWORD
are set, a default admin identity with full permissions is automatically created using these values. - Credential Precedence: The system prioritizes existing identities: if an identity with the same name as
WEED_S3_ADMIN_USER
already exists (e.g., loaded from a configuration file), the environment variable credentials for that specific user will be skipped to prevent unintended overwrites. - Code Cleanup and Testing: Several unused helper functions related to identity and action checks have been removed, improving code clarity. Comprehensive unit tests have been added to validate various scenarios for environment variable credential loading, ensuring the robustness and correctness of the new feature.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command>
or @gemini-code-assist <command>
. Below is a summary of the supported commands.
Feature | Command | Description |
---|---|---|
Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/
folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a convenient way to provide S3 admin credentials via environment variables, which is a great enhancement for usability. The implementation is mostly sound and includes good test coverage for the new logic.
However, I've identified a critical issue in the configuration loading logic that causes configuration from a file to be overwritten, effectively breaking that feature. I've also noted a few medium-severity issues related to code maintainability and logging clarity that should be addressed.
Once these points are resolved, this will be a solid and valuable addition to the project.
weed/s3api/auth_credentials.go
Outdated
s3ApiConfiguration, err := iam.credentialManager.LoadConfiguration(context.Background()) | ||
if err != nil { | ||
glog.Warningf("failed to load configuration for environment variables: %v", err) | ||
// Create empty configuration if loading fails | ||
s3ApiConfiguration = &iam_pb.S3ApiConfiguration{} | ||
} | ||
|
||
// Load environment variable credentials | ||
iam.loadAdminCredentialsFromEnv(s3ApiConfiguration) | ||
|
||
// Apply the updated configuration | ||
if err := iam.loadS3ApiConfiguration(s3ApiConfiguration); err != nil { | ||
glog.Warningf("failed to apply environment variable configuration: %v", err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This block of code introduces a critical bug that breaks existing functionality. It unconditionally re-loads the configuration from the credentialManager
, which overwrites any configuration that might have been loaded from a file (option.Config
) in the preceding lines (172-182).
Problematic Flow:
- If a user provides a config file,
iam.loadS3ApiConfigurationFromFile()
is called, and theiam
instance is populated. - This new block then executes
iam.credentialManager.LoadConfiguration()
, which fetches configuration from the credential store (e.g., filer), effectively discarding the configuration from the file. iam.loadS3ApiConfiguration()
is called again with this new configuration, overwriting theiam
state.
This needs to be refactored to ensure that configuration is loaded only once, then modified with environment variable credentials, and finally applied.
A suggested approach for NewIdentityAccessManagementWithStore
:
func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitStore string) *IdentityAccessManagement {
// ... (iam and credentialManager initialization)
var s3ApiConfiguration *iam_pb.S3ApiConfiguration
var err error
if option.Config != "" {
glog.V(3).Infof("loading static config file %s", option.Config)
content, readErr := os.ReadFile(option.Config)
if readErr != nil {
glog.Fatalf("fail to read config file %s: %v", option.Config, readErr)
}
s3ApiConfiguration = &iam_pb.S3ApiConfiguration{}
if err = filer.ParseS3ConfigurationFromBytes(content, s3ApiConfiguration); err != nil {
glog.Fatalf("unmarshal error from %s: %v", option.Config, err)
}
} else {
glog.V(3).Infof("no static config file specified... loading config from credential manager")
s3ApiConfiguration, err = iam.credentialManager.LoadConfiguration(context.Background())
if err != nil {
glog.Warningf("fail to load config: %v", err)
// Create empty configuration if loading fails
s3ApiConfiguration = &iam_pb.S3ApiConfiguration{}
}
}
// Load environment variable credentials
iam.loadAdminCredentialsFromEnv(s3ApiConfiguration)
// Apply the updated configuration
if err := iam.loadS3ApiConfiguration(s3ApiConfiguration); err != nil {
glog.Warningf("failed to apply initial configuration: %v", err)
}
return iam
}
weed/command/s3.go
Outdated
* The admin access key is from WEED_S3_ADMIN_USER. | ||
* The admin secret key is from WEED_S3_ADMIN_PASSWORD. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The indentation for this list seems to be inconsistent, using a mix of spaces and tabs. This can lead to inconsistent rendering in different terminals and editors.
For better readability and consistency with the surrounding help text, please use tabs for indentation.
* The admin access key is from WEED_S3_ADMIN_USER. | |
* The admin secret key is from WEED_S3_ADMIN_PASSWORD. | |
* The admin access key is from WEED_S3_ADMIN_USER. | |
* The admin secret key is from WEED_S3_ADMIN_PASSWORD. |
weed/s3api/auth_credentials.go
Outdated
adminUser := os.Getenv("WEED_S3_ADMIN_USER") | ||
adminPassword := os.Getenv("WEED_S3_ADMIN_PASSWORD") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The environment variable names "WEED_S3_ADMIN_USER"
and "WEED_S3_ADMIN_PASSWORD"
are used as magic strings here and in other parts of the codebase (tests, command documentation). To improve maintainability and prevent typos, it's a best practice to define them as constants.
Please consider defining these at the package level and using the constants throughout the code.
const (
AdminUserEnv = "WEED_S3_ADMIN_USER"
AdminPasswordEnv = "WEED_S3_ADMIN_PASSWORD"
)
// ...
func (iam *IdentityAccessManagement) loadAdminCredentialsFromEnv(...) {
adminUser := os.Getenv(AdminUserEnv)
adminPassword := os.Getenv(AdminPasswordEnv)
// ...
}
weed/s3api/auth_credentials.go
Outdated
// Always load admin credentials from environment variables after loading other configurations | ||
s3ApiConfiguration, err := iam.credentialManager.LoadConfiguration(context.Background()) | ||
if err != nil { | ||
glog.Warningf("failed to load configuration for environment variables: %v", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This log message is misleading. The error is from iam.credentialManager.LoadConfiguration
, not from loading environment variables. This could make debugging more difficult. The log message should accurately describe the failed operation.
glog.Warningf("failed to load configuration from credential manager: %v", err)
/gemini review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR simplifies S3 API initialization by allowing admin credentials to be provided through AWS environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) instead of requiring a configuration file.
- Adds environment variable credential loading to the IAM initialization process
- Creates admin identity automatically when both AWS environment variables are present
- Ensures environment credentials supplement rather than replace existing configuration
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
weed/s3api/auth_credentials.go | Core implementation for loading AWS credentials from environment variables and creating admin identity |
weed/s3api/auth_credentials_test.go | Comprehensive test coverage for environment variable credential functionality |
weed/command/s3.go | Documentation update explaining the new environment variable usage |
|
||
// Add to existing configuration | ||
iam.m.Lock() | ||
iam.identities = append(iam.identities, envIdentity) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The accessKeyIdent map is being accessed without checking if it's initialized. This could cause a nil pointer panic if the map hasn't been properly initialized during IAM setup.
iam.identities = append(iam.identities, envIdentity) | |
iam.identities = append(iam.identities, envIdentity) | |
if iam.accessKeyIdent == nil { | |
iam.accessKeyIdent = make(map[string]*Identity) | |
} |
Copilot uses AI. Check for mistakes.
|
||
glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name) | ||
} else { | ||
glog.V(0).Infof("Access key %s already exists, skipping environment variable credentials", accessKeyId) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Logging the access key ID in plain text could expose sensitive credential information in log files. Consider logging only a masked or truncated version for security purposes.
Copilot uses AI. Check for mistakes.
} | ||
iam.m.Unlock() | ||
|
||
glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The identity name contains part of the access key ID which could expose sensitive information in logs. Consider using a generic message or ensuring logs are properly secured.
glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name) | |
glog.V(0).Infof("Added admin identity from AWS environment variables.") |
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a convenient way to provide S3 credentials via environment variables, improving usability. The implementation is straightforward, and the addition of tests is appreciated. Key areas for improvement include optimizing the access key check, clarifying a misleading comment, expanding test coverage, and refactoring magic numbers and strings into constants. Addressing these points will enhance the code's robustness and maintainability.
weed/s3api/auth_credentials.go
Outdated
accessKeyExists := false | ||
iam.m.RLock() | ||
for _, identity := range iam.identities { | ||
for _, cred := range identity.Credentials { | ||
if cred.AccessKey == accessKeyId { | ||
accessKeyExists = true | ||
break | ||
} | ||
} | ||
if accessKeyExists { | ||
break | ||
} | ||
} | ||
iam.m.RUnlock() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current method of checking for an existing accessKeyId
involves iterating through all identities and their credentials, which can be inefficient, especially with a large number of identities. Since iam.accessKeyIdent
is a map designed for quick lookups of identities by access key, it should be used here for better performance and code simplicity.
iam.m.RLock()
_, accessKeyExists := iam.accessKeyIdent[accessKeyId]
iam.m.RUnlock()
func TestNewIdentityAccessManagementWithStoreEnvVars(t *testing.T) { | ||
// Save original environment | ||
originalAccessKeyId := os.Getenv("AWS_ACCESS_KEY_ID") | ||
originalSecretAccessKey := os.Getenv("AWS_SECRET_ACCESS_KEY") | ||
|
||
// Clean up after test | ||
defer func() { | ||
if originalAccessKeyId != "" { | ||
os.Setenv("AWS_ACCESS_KEY_ID", originalAccessKeyId) | ||
} else { | ||
os.Unsetenv("AWS_ACCESS_KEY_ID") | ||
} | ||
if originalSecretAccessKey != "" { | ||
os.Setenv("AWS_SECRET_ACCESS_KEY", originalSecretAccessKey) | ||
} else { | ||
os.Unsetenv("AWS_SECRET_ACCESS_KEY") | ||
} | ||
}() | ||
|
||
tests := []struct { | ||
name string | ||
accessKeyId string | ||
secretAccessKey string | ||
expectEnvIdentity bool | ||
expectedName string | ||
}{ | ||
{ | ||
name: "Both env vars set", | ||
accessKeyId: "AKIA1234567890ABCDEF", | ||
secretAccessKey: "secret123456789012345678901234567890abcdef12", | ||
expectEnvIdentity: true, | ||
expectedName: "admin-AKIA1234", | ||
}, | ||
{ | ||
name: "Short access key", | ||
accessKeyId: "SHORT", | ||
secretAccessKey: "secret123456789012345678901234567890abcdef12", | ||
expectEnvIdentity: true, | ||
expectedName: "admin-SHORT", | ||
}, | ||
{ | ||
name: "No env vars set", | ||
accessKeyId: "", | ||
secretAccessKey: "", | ||
expectEnvIdentity: false, | ||
expectedName: "", | ||
}, | ||
} | ||
|
||
for _, tt := range tests { | ||
t.Run(tt.name, func(t *testing.T) { | ||
// Set up environment variables | ||
if tt.accessKeyId != "" { | ||
os.Setenv("AWS_ACCESS_KEY_ID", tt.accessKeyId) | ||
} else { | ||
os.Unsetenv("AWS_ACCESS_KEY_ID") | ||
} | ||
if tt.secretAccessKey != "" { | ||
os.Setenv("AWS_SECRET_ACCESS_KEY", tt.secretAccessKey) | ||
} else { | ||
os.Unsetenv("AWS_SECRET_ACCESS_KEY") | ||
} | ||
|
||
// Create IAM instance with memory store for testing | ||
option := &S3ApiServerOption{ | ||
Config: "", // No config file, should use environment variables | ||
} | ||
iam := NewIdentityAccessManagementWithStore(option, string(credential.StoreTypeMemory)) | ||
|
||
if tt.expectEnvIdentity { | ||
// Check that environment variable identity was created | ||
found := false | ||
for _, identity := range iam.identities { | ||
if identity.Name == tt.expectedName { | ||
found = true | ||
assert.Len(t, identity.Credentials, 1, "Should have one credential") | ||
assert.Equal(t, tt.accessKeyId, identity.Credentials[0].AccessKey, "Access key should match environment variable") | ||
assert.Equal(t, tt.secretAccessKey, identity.Credentials[0].SecretKey, "Secret key should match environment variable") | ||
assert.Contains(t, identity.Actions, Action(ACTION_ADMIN), "Should have admin action") | ||
break | ||
} | ||
} | ||
assert.True(t, found, "Should find identity created from environment variables") | ||
} else { | ||
// When no env vars, should have no identities (since no config file) | ||
assert.Len(t, iam.identities, 0, "Should have no identities when no env vars and no config file") | ||
} | ||
}) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test TestNewIdentityAccessManagementWithStoreEnvVars
is a great start, but it only covers cases where no configuration file is loaded. To ensure the logic correctly handles interactions with existing configurations, consider adding test cases for these scenarios:
- Existing Access Key: A configuration file is loaded, and the
accessKeyId
from the environment variables already exists in it. Assert that the identity from the environment variables is not added. - New Access Key: A configuration file is loaded, and the
accessKeyId
from the environment variables is new. Assert that the new identity is added alongside the existing identities from the configuration file.
if len(accessKeyId) > 8 { | ||
identityNameSuffix = accessKeyId[:8] | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
// Create admin identity with environment variable credentials | ||
envIdentity := &Identity{ | ||
Name: "admin-" + identityNameSuffix, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if !iam.isAuthEnabled { | ||
iam.isAuthEnabled = true | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
What problem are we solving?
starting s3 needs a configuration file, which is not so convenient.
How are we solving the problem?
Simplest way to start and use the object store server:
These should work too:
How is the PR tested?
Manual.
Checks