Ubuntu/Rhel Certificate Renewal Implementation (Part 2 of the certificate renewal command feature) #9781

howard-junec · 2025-06-03T21:50:51Z

Issue #, if available:

Description of changes: 
This PR implements certificate renewal functionality for Ubuntu and RHEL nodes in EKS Anywhere clusters. This is part 2 of the certificate renewal feature, focusing on Linux-based operating systems. Key features include:

Direct certificate management for Ubuntu and RHEL nodes using a unified Linux implementation
SSH-based secure certificate deployment and renewal system
Automatic certificate backup and validation
Support for control plane certificate renewal for both stacked and external etcd configurations on Ubuntu and RHEL nodes
Automatic kubeconfig discovery based on cluster name from input YAML file
Configurable logging verbosity with -v flag to control output detail level
- Default mode shows only essential information
- Verbose mode (-v=2) displays detailed renewal process information
Environment variable support:
- Set EKSA_SSH_KEY_PASSPHRASE_ETCD and EKSA_SSH_KEY_PASSPHRASE_CP to avoid manual password input
- Manual password input when environment variables are not set

Example Config File:

clusterName: my-cluster
os: ubuntu  # ubuntu, rhel
controlPlane:
  nodes:
    - 100.000.0.11
    - 100.000.0.12
  ssh:
    sshKey: /path/to/ssh/private-key
    sshUser: ec2-user
etcd:
  nodes:
    - 100.000.0.14
    - 100.000.0.15
  ssh:
    sshKey: /path/to/ssh/private-key
    sshUser: ec2-user

This PR builds on the configuration structure from part 1, implementing the renewal logic for traditional Linux distributions.

Testing (if applicable):

Documentation added/planned (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

eks-distro-bot · 2025-06-03T21:51:03Z

Hi @howard-junec. Thanks for your PR.

I'm waiting for a aws member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cmd/eksctl-anywhere/cmd/renewcertificates.go

pkg/certificates/config.go

cmd/eksctl-anywhere/cmd/renewcertificates.go

pkg/certificates/os.go

pkg/certificates/renewer.go

pkg/certificates/renewerubuntu.go

pkg/certificates/linuxcommand.go

pkg/certificates/renewer.go

pkg/certificates/config.go

cmd/eksctl-anywhere/cmd/renewcertificates.go

pkg/certificates/config.go

pkg/certificates/os.go

pkg/certificates/renewer.go

pkg/certificates/renewerubuntu.go

rahulbabu95 · 2025-06-18T23:53:43Z

pkg/certificates/renewer.go

+	err = r.kube.Get(ctx, secretName, constants.EksaSystemNamespace, secret)
+	if err != nil {
+		if apierrors.IsNotFound(err) {
+			logger.V(2).Info("Secret not found—skipping creation", "name", secretName)


When will we run into this case?

this is for skipping creating the secret if not found, otherwise, we will update the apiserver etcd client secret

rahulbabu95 · 2025-06-19T00:00:54Z

pkg/certificates/renewerubuntu.go

+		return fmt.Errorf("backup certs: %v", err)
+	}
+	if err := ssh.RunCommand(ctx, node,
+		[]string{"sudo", "etcdadm", "join", "phase", "certificates", "http://eks-a-etcd-dumb-url"}); err != nil {


why are we running this?

this is for run certificates join phase to regenerate the deleted certificates

rahulbabu95 · 2025-06-19T00:13:53Z

pkg/certificates/renewerubuntu.go

+		"cd %[1]s && cp -r pki pki.bak_%[2]s && rm -rf pki/* && cp pki.bak_%[2]s/ca.* pki/",
+		linuxEtcdCertDir, backup,
+	)
+	return []string{"sudo", "sh", "-c", script}


nit: can we just move the first 3 params here to a separate func and make it accept any script and just call it here and all other use cases?

rahulbabu95 · 2025-06-19T00:21:40Z

pkg/certificates/ssh.go

+}
+
+func (r *DefaultSSHRunner) buildCommandString(cmds []string) string {
+	if len(cmds) >= 3 && cmds[0] == "sudo" && cmds[1] == "sh" && cmds[2] == "-c" {


Why do we need this only for this command?

since commands are passed as string slices, but sudo sh -c needs proper quoting to preserve script integrity, and in later, bottlerocket's sudo sheltie commands need special handling to maintain heredoc syntax. This function will formats both cases correctly.

rahulbabu95 · 2025-06-19T00:23:58Z

cmd/eksctl-anywhere/cmd/renewcertificates.go

+
+	kubeClient := deps.UnAuthKubeClient.KubeconfigClient(kubeCfgPath)
+
+	osKey := cfg.OS


can we just call it os?

2ez4szliu · 2025-06-19T04:10:55Z

pkg/certificates/ssh.go

+	// RunCommand runs a command on the remote host
+	RunCommand(ctx context.Context, node string, cmds []string) error
+	// RunCommandWithOutput runs a command on the remote host and returns the output
+	RunCommandWithOutput(ctx context.Context, node string, cmds []string) (string, error)


do we need both RunCommandWithOutput and RunCommand? I think we only need RunCommandWithOutput, just don't assign the output if we don't need it
_, err := ssh.RunCommandWithOutput

I tried replacing RunCommand with RunCommandWithOutput as suggested, but found that it loses detailed command output when using high verbosity levels like -v=9. RunCommand has special logging behavior that streams command output directly to the console at high verbosity levels, which is critical for debugging. Both methods serve different purposes, I think we can keep both.

How are we deciding when to run RunCommand or RunCommandWithOutput?

for nor we only use RunCommandWithOutput when we need to programmatically capture and process the command's output, like captures details steps in "copyEtcdCerts", and all other steps we just use RunCommand.

…tialization, and handle environment variables more robustly

pkg/certificates/os.go

pkg/certificates/renewer.go

pkg/certificates/ssh.go

…lified SSH handling

… instead of all nodes

pkg/certificates/renewer.go

pkg/certificates/renewerubuntu.go

pkg/certificates/ssh.go

pkg/certificates/renewer.go

panktishah26 · 2025-07-01T23:53:36Z

Thanks for all the work!
/lgtm

eks-distro-bot · 2025-07-02T17:10:12Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

eks-distro-pr-bot · 2025-07-29T18:39:15Z

@panktishah26: new pull request created: #9957

In response to this:

/cherry-pick release-0.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

panktishah26 · 2025-07-29T19:24:42Z

/cherry-pick release-0.23

eks-distro-pr-bot · 2025-07-29T19:25:20Z

@panktishah26: new pull request created: #9959

In response to this:

/cherry-pick release-0.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

eks-distro-bot added needs-ok-to-test size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 3, 2025

howard-junec force-pushed the cert-renewal-ubuntu branch from fbe65d4 to 51d856f Compare June 6, 2025 22:14

Refactor certificate renewal for better object-oriented design

6eb3e86

howard-junec force-pushed the cert-renewal-ubuntu branch from b2dad8b to 6eb3e86 Compare June 12, 2025 21:32

panktishah26 reviewed Jun 13, 2025

View reviewed changes

pkg/certificates/renewer.go Show resolved Hide resolved

update per comment

c56b9b5

panktishah26 reviewed Jun 18, 2025

View reviewed changes

eks-distro-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 18, 2025

howard-junec force-pushed the cert-renewal-ubuntu branch from 6fcf2db to c56b9b5 Compare June 18, 2025 20:31

eks-distro-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 18, 2025

refactor: use string arrays for SSH commands per PR feedback

8b1dba7

eks-distro-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 18, 2025

rahulbabu95 reviewed Jun 19, 2025

View reviewed changes

2ez4szliu reviewed Jun 19, 2025

View reviewed changes

howard-junec force-pushed the cert-renewal-ubuntu branch from 2851169 to 160214e Compare June 25, 2025 21:26

Refactor certificate renewal to improve consistency, simplify SSH ini…

c59d637

…tialization, and handle environment variables more robustly

panktishah26 reviewed Jun 25, 2025

View reviewed changes

howard-junec force-pushed the cert-renewal-ubuntu branch from fe2d729 to c59d637 Compare June 26, 2025 21:16

Refactor certificate renewal process for improved robustness and simp…

9e664e6

…lified SSH handling

eks-distro-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 30, 2025

howard-junec force-pushed the cert-renewal-ubuntu branch from a454c69 to 9e664e6 Compare June 30, 2025 18:12

howard-junec force-pushed the cert-renewal-ubuntu branch from c1ae8ea to 9e664e6 Compare June 30, 2025 18:17

eks-distro-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 30, 2025

Refactor code structure and copy certificates from a single etcd node…

1ce4975

… instead of all nodes

panktishah26 reviewed Jul 1, 2025

View reviewed changes

howard-junec added 2 commits July 1, 2025 15:56

update logger content

3a22345

update logger and remove newRenewerForCmd

ff73874

eks-distro-bot assigned panktishah26 Jul 1, 2025

eks-distro-bot added the lgtm label Jul 1, 2025

panktishah26 added the approved label Jul 2, 2025

eks-distro-bot merged commit 7942b13 into aws:main Jul 2, 2025
6 checks passed

eks-distro-pr-bot mentioned this pull request Jul 29, 2025

[release-0.23] Ubuntu/Rhel Certificate Renewal Implementation (Part 2 of the certificate renewal command feature) #9957

Closed

eks-distro-pr-bot mentioned this pull request Jul 29, 2025

[release-0.23] Ubuntu/Rhel Certificate Renewal Implementation (Part 2 of the certificate renewal command feature) #9959

Merged


		kubeClient := deps.UnAuthKubeClient.KubeconfigClient(kubeCfgPath)

		osKey := cfg.OS

Ubuntu/Rhel Certificate Renewal Implementation (Part 2 of the certificate renewal command feature) #9781

Ubuntu/Rhel Certificate Renewal Implementation (Part 2 of the certificate renewal command feature) #9781

Uh oh!

Conversation

howard-junec commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eks-distro-bot commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

howard-junec Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

panktishah26 commented Jul 1, 2025

Uh oh!

eks-distro-bot commented Jul 2, 2025

Uh oh!

Uh oh!

eks-distro-pr-bot commented Jul 29, 2025

howard-junec commented Jun 3, 2025 •

edited

Loading

howard-junec Jun 19, 2025 •

edited

Loading