Skip to content

Conversation

chrislusf
Copy link
Collaborator

@chrislusf chrislusf commented Jul 23, 2025

What problem are we solving?

starting s3 needs a configuration file, which is not so convenient.

How are we solving the problem?

Simplest way to start and use the object store server:

export AWS_ACCESS_KEY_ID=admin_access
export AWS_SECRET_ACCESS_KEY=admin_secret
weed server -s3 -dir=/tmp/dir

These should work too:

AWS_ACCESS_KEY_ID=admin_access AWS_SECRET_ACCESS_KEY=admin_secret weed s3
AWS_ACCESS_KEY_ID=admin_access AWS_SECRET_ACCESS_KEY=admin_secret weed filer -s3

How is the PR tested?

Manual.

Checks

  • I have added unit tests if possible.
  • I will add related wiki document changes and link to this PR after merging.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @chrislusf, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly simplifies the initial setup and configuration of the S3 API by introducing support for environment variable-based admin credentials. This change reduces reliance on explicit configuration files, making it easier to deploy and manage S3 instances, especially in containerized or automated environments. The new logic ensures seamless integration with existing identity management while providing clear precedence rules and robust testing.

Highlights

  • Simplified S3 Admin Setup: Users can now configure S3 admin credentials directly via WEED_S3_ADMIN_USER and WEED_S3_ADMIN_PASSWORD environment variables. This eliminates the need for a separate configuration file for basic setup, making deployment and management more convenient.
  • Environment Variable Credential Loading Logic: A new mechanism has been implemented to detect and load admin credentials from environment variables during S3 API initialization. If both WEED_S3_ADMIN_USER and WEED_S3_ADMIN_PASSWORD are set, a default admin identity with full permissions is automatically created using these values.
  • Credential Precedence: The system prioritizes existing identities: if an identity with the same name as WEED_S3_ADMIN_USER already exists (e.g., loaded from a configuration file), the environment variable credentials for that specific user will be skipped to prevent unintended overwrites.
  • Code Cleanup and Testing: Several unused helper functions related to identity and action checks have been removed, improving code clarity. Comprehensive unit tests have been added to validate various scenarios for environment variable credential loading, ensuring the robustness and correctness of the new feature.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a convenient way to provide S3 admin credentials via environment variables, which is a great enhancement for usability. The implementation is mostly sound and includes good test coverage for the new logic.

However, I've identified a critical issue in the configuration loading logic that causes configuration from a file to be overwritten, effectively breaking that feature. I've also noted a few medium-severity issues related to code maintainability and logging clarity that should be addressed.

Once these points are resolved, this will be a solid and valuable addition to the project.

Comment on lines 185 to 198
s3ApiConfiguration, err := iam.credentialManager.LoadConfiguration(context.Background())
if err != nil {
glog.Warningf("failed to load configuration for environment variables: %v", err)
// Create empty configuration if loading fails
s3ApiConfiguration = &iam_pb.S3ApiConfiguration{}
}

// Load environment variable credentials
iam.loadAdminCredentialsFromEnv(s3ApiConfiguration)

// Apply the updated configuration
if err := iam.loadS3ApiConfiguration(s3ApiConfiguration); err != nil {
glog.Warningf("failed to apply environment variable configuration: %v", err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This block of code introduces a critical bug that breaks existing functionality. It unconditionally re-loads the configuration from the credentialManager, which overwrites any configuration that might have been loaded from a file (option.Config) in the preceding lines (172-182).

Problematic Flow:

  1. If a user provides a config file, iam.loadS3ApiConfigurationFromFile() is called, and the iam instance is populated.
  2. This new block then executes iam.credentialManager.LoadConfiguration(), which fetches configuration from the credential store (e.g., filer), effectively discarding the configuration from the file.
  3. iam.loadS3ApiConfiguration() is called again with this new configuration, overwriting the iam state.

This needs to be refactored to ensure that configuration is loaded only once, then modified with environment variable credentials, and finally applied.

A suggested approach for NewIdentityAccessManagementWithStore:

func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitStore string) *IdentityAccessManagement {
	// ... (iam and credentialManager initialization)

	var s3ApiConfiguration *iam_pb.S3ApiConfiguration
	var err error

	if option.Config != "" {
		glog.V(3).Infof("loading static config file %s", option.Config)
		content, readErr := os.ReadFile(option.Config)
		if readErr != nil {
			glog.Fatalf("fail to read config file %s: %v", option.Config, readErr)
		}
		s3ApiConfiguration = &iam_pb.S3ApiConfiguration{}
		if err = filer.ParseS3ConfigurationFromBytes(content, s3ApiConfiguration); err != nil {
			glog.Fatalf("unmarshal error from %s: %v", option.Config, err)
		}
	} else {
		glog.V(3).Infof("no static config file specified... loading config from credential manager")
		s3ApiConfiguration, err = iam.credentialManager.LoadConfiguration(context.Background())
		if err != nil {
			glog.Warningf("fail to load config: %v", err)
			// Create empty configuration if loading fails
			s3ApiConfiguration = &iam_pb.S3ApiConfiguration{}
		}
	}

	// Load environment variable credentials
	iam.loadAdminCredentialsFromEnv(s3ApiConfiguration)

	// Apply the updated configuration
	if err := iam.loadS3ApiConfiguration(s3ApiConfiguration); err != nil {
		glog.Warningf("failed to apply initial configuration: %v", err)
	}

	return iam
}

Comment on lines 169 to 170
* The admin access key is from WEED_S3_ADMIN_USER.
* The admin secret key is from WEED_S3_ADMIN_PASSWORD.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The indentation for this list seems to be inconsistent, using a mix of spaces and tabs. This can lead to inconsistent rendering in different terminals and editors.

For better readability and consistency with the surrounding help text, please use tabs for indentation.

Suggested change
* The admin access key is from WEED_S3_ADMIN_USER.
* The admin secret key is from WEED_S3_ADMIN_PASSWORD.
* The admin access key is from WEED_S3_ADMIN_USER.
* The admin secret key is from WEED_S3_ADMIN_PASSWORD.

Comment on lines 109 to 110
adminUser := os.Getenv("WEED_S3_ADMIN_USER")
adminPassword := os.Getenv("WEED_S3_ADMIN_PASSWORD")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The environment variable names "WEED_S3_ADMIN_USER" and "WEED_S3_ADMIN_PASSWORD" are used as magic strings here and in other parts of the codebase (tests, command documentation). To improve maintainability and prevent typos, it's a best practice to define them as constants.

Please consider defining these at the package level and using the constants throughout the code.

const (
	AdminUserEnv     = "WEED_S3_ADMIN_USER"
	AdminPasswordEnv = "WEED_S3_ADMIN_PASSWORD"
)

// ...

func (iam *IdentityAccessManagement) loadAdminCredentialsFromEnv(...) {
    adminUser := os.Getenv(AdminUserEnv)
    adminPassword := os.Getenv(AdminPasswordEnv)
    // ...
}

// Always load admin credentials from environment variables after loading other configurations
s3ApiConfiguration, err := iam.credentialManager.LoadConfiguration(context.Background())
if err != nil {
glog.Warningf("failed to load configuration for environment variables: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This log message is misleading. The error is from iam.credentialManager.LoadConfiguration, not from loading environment variables. This could make debugging more difficult. The log message should accurately describe the failed operation.

glog.Warningf("failed to load configuration from credential manager: %v", err)

@chrislusf chrislusf marked this pull request as ready for review July 23, 2025 08:21
@chrislusf
Copy link
Collaborator Author

/gemini review

@chrislusf chrislusf requested a review from Copilot July 23, 2025 08:21
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR simplifies S3 API initialization by allowing admin credentials to be provided through AWS environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) instead of requiring a configuration file.

  • Adds environment variable credential loading to the IAM initialization process
  • Creates admin identity automatically when both AWS environment variables are present
  • Ensures environment credentials supplement rather than replace existing configuration

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
weed/s3api/auth_credentials.go Core implementation for loading AWS credentials from environment variables and creating admin identity
weed/s3api/auth_credentials_test.go Comprehensive test coverage for environment variable credential functionality
weed/command/s3.go Documentation update explaining the new environment variable usage


// Add to existing configuration
iam.m.Lock()
iam.identities = append(iam.identities, envIdentity)
Copy link
Preview

Copilot AI Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The accessKeyIdent map is being accessed without checking if it's initialized. This could cause a nil pointer panic if the map hasn't been properly initialized during IAM setup.

Suggested change
iam.identities = append(iam.identities, envIdentity)
iam.identities = append(iam.identities, envIdentity)
if iam.accessKeyIdent == nil {
iam.accessKeyIdent = make(map[string]*Identity)
}

Copilot uses AI. Check for mistakes.


glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name)
} else {
glog.V(0).Infof("Access key %s already exists, skipping environment variable credentials", accessKeyId)
Copy link
Preview

Copilot AI Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging the access key ID in plain text could expose sensitive credential information in log files. Consider logging only a masked or truncated version for security purposes.

Copilot uses AI. Check for mistakes.

}
iam.m.Unlock()

glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name)
Copy link
Preview

Copilot AI Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The identity name contains part of the access key ID which could expose sensitive information in logs. Consider using a generic message or ensuring logs are properly secured.

Suggested change
glog.V(0).Infof("Added admin identity from AWS environment variables: %s", envIdentity.Name)
glog.V(0).Infof("Added admin identity from AWS environment variables.")

Copilot uses AI. Check for mistakes.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a convenient way to provide S3 credentials via environment variables, improving usability. The implementation is straightforward, and the addition of tests is appreciated. Key areas for improvement include optimizing the access key check, clarifying a misleading comment, expanding test coverage, and refactoring magic numbers and strings into constants. Addressing these points will enhance the code's robustness and maintainability.

Comment on lines 156 to 169
accessKeyExists := false
iam.m.RLock()
for _, identity := range iam.identities {
for _, cred := range identity.Credentials {
if cred.AccessKey == accessKeyId {
accessKeyExists = true
break
}
}
if accessKeyExists {
break
}
}
iam.m.RUnlock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current method of checking for an existing accessKeyId involves iterating through all identities and their credentials, which can be inefficient, especially with a large number of identities. Since iam.accessKeyIdent is a map designed for quick lookups of identities by access key, it should be used here for better performance and code simplicity.

iam.m.RLock()
_, accessKeyExists := iam.accessKeyIdent[accessKeyId]
iam.m.RUnlock()

Comment on lines +270 to +359
func TestNewIdentityAccessManagementWithStoreEnvVars(t *testing.T) {
// Save original environment
originalAccessKeyId := os.Getenv("AWS_ACCESS_KEY_ID")
originalSecretAccessKey := os.Getenv("AWS_SECRET_ACCESS_KEY")

// Clean up after test
defer func() {
if originalAccessKeyId != "" {
os.Setenv("AWS_ACCESS_KEY_ID", originalAccessKeyId)
} else {
os.Unsetenv("AWS_ACCESS_KEY_ID")
}
if originalSecretAccessKey != "" {
os.Setenv("AWS_SECRET_ACCESS_KEY", originalSecretAccessKey)
} else {
os.Unsetenv("AWS_SECRET_ACCESS_KEY")
}
}()

tests := []struct {
name string
accessKeyId string
secretAccessKey string
expectEnvIdentity bool
expectedName string
}{
{
name: "Both env vars set",
accessKeyId: "AKIA1234567890ABCDEF",
secretAccessKey: "secret123456789012345678901234567890abcdef12",
expectEnvIdentity: true,
expectedName: "admin-AKIA1234",
},
{
name: "Short access key",
accessKeyId: "SHORT",
secretAccessKey: "secret123456789012345678901234567890abcdef12",
expectEnvIdentity: true,
expectedName: "admin-SHORT",
},
{
name: "No env vars set",
accessKeyId: "",
secretAccessKey: "",
expectEnvIdentity: false,
expectedName: "",
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Set up environment variables
if tt.accessKeyId != "" {
os.Setenv("AWS_ACCESS_KEY_ID", tt.accessKeyId)
} else {
os.Unsetenv("AWS_ACCESS_KEY_ID")
}
if tt.secretAccessKey != "" {
os.Setenv("AWS_SECRET_ACCESS_KEY", tt.secretAccessKey)
} else {
os.Unsetenv("AWS_SECRET_ACCESS_KEY")
}

// Create IAM instance with memory store for testing
option := &S3ApiServerOption{
Config: "", // No config file, should use environment variables
}
iam := NewIdentityAccessManagementWithStore(option, string(credential.StoreTypeMemory))

if tt.expectEnvIdentity {
// Check that environment variable identity was created
found := false
for _, identity := range iam.identities {
if identity.Name == tt.expectedName {
found = true
assert.Len(t, identity.Credentials, 1, "Should have one credential")
assert.Equal(t, tt.accessKeyId, identity.Credentials[0].AccessKey, "Access key should match environment variable")
assert.Equal(t, tt.secretAccessKey, identity.Credentials[0].SecretKey, "Secret key should match environment variable")
assert.Contains(t, identity.Actions, Action(ACTION_ADMIN), "Should have admin action")
break
}
}
assert.True(t, found, "Should find identity created from environment variables")
} else {
// When no env vars, should have no identities (since no config file)
assert.Len(t, iam.identities, 0, "Should have no identities when no env vars and no config file")
}
})
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test TestNewIdentityAccessManagementWithStoreEnvVars is a great start, but it only covers cases where no configuration file is loaded. To ensure the logic correctly handles interactions with existing configurations, consider adding test cases for these scenarios:

  1. Existing Access Key: A configuration file is loaded, and the accessKeyId from the environment variables already exists in it. Assert that the identity from the environment variables is not added.
  2. New Access Key: A configuration file is loaded, and the accessKeyId from the environment variables is new. Assert that the new identity is added alongside the existing identities from the configuration file.

Comment on lines +174 to +176
if len(accessKeyId) > 8 {
identityNameSuffix = accessKeyId[:8]
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Consider using a constant for the maximum length of the access key ID to avoid "magic number".

const accessKeyIdSuffixLength = 8
identityNameSuffix := accessKeyId
if len(accessKeyId) > accessKeyIdSuffixLength {
	identityNameSuffix = accessKeyId[:accessKeyIdSuffixLength]
}


// Create admin identity with environment variable credentials
envIdentity := &Identity{
Name: "admin-" + identityNameSuffix,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Consider defining the string "admin-" as a constant to avoid a "magic string".

const adminIdentityNamePrefix = "admin-"
Name:    adminIdentityNamePrefix + identityNameSuffix,

Comment on lines +197 to +199
if !iam.isAuthEnabled {
iam.isAuthEnabled = true
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Consider using atomic.StoreBool to ensure that iam.isAuthEnabled is updated atomically.

iam.m.Unlock()
			atomic.StoreBool(&iam.isAuthEnabled, true)
			iam.m.Lock()

chrislusf and others added 3 commits July 23, 2025 01:25
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@chrislusf chrislusf merged commit e3d3c49 into master Jul 23, 2025
21 checks passed
@chrislusf chrislusf deleted the simpler-way-to-start-s3-with-credentials branch July 23, 2025 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant