Summary

Enhance the ArgoCD ApplicationSet resource to embed a rollout strategy for a progressive application resource update after the ApplicationSet spec or Application templates are modified.

Motivation

As cluster operators, we would like to make changes to ApplicationSets which may target multiple environments, pre-defined staging areas, or other configurations, and have these changes rolled out in a declarative, defined manner rather than all at once as ApplicationSets currently behave. A progressive ApplicationSet rollout would prevent mistakes in configuration from having a larger blast radius than intended and give cluster operators a chance to verify and have confidence in their changes.

Proposal

As an initial proposal, two rollout strategies are suggested:

• AllAtOnce (Replace?)
  • The current application set rollout strategy behavior (default)
  • No strategy spec
• RollingUpdate
  • The rolling update strategy deterministically chooses applications to update one by one, proceeding each step only if the previous application syncs completed successfully, and respecting the maxUpdate value.
  • Steps for the rolling update are defined by a list of matchExpression label selectors. Each step must finish updating before the next step advances.
  • If steps are left undefined the application update order is deterministic.

ApplicationSet CRD Updates

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  generators:
  - list:
      elements:
      - cluster: engineering-dev
        url: https://1.2.3.4
      - cluster: engineering-prod
        url: https://2.4.6.8
      - cluster: finance-preprod
        url: https://9.8.7.6/
  strategy:
    type: RollingUpdate
    rollingUpdate:
      steps:
        - matchExpressions:
            - key: env
              operator: In
              values: 
                - dev
          maxUpdate: 0 # if undefined or 0, all applications matched are updated together
        - matchExpressions:
            - key: env
              operator: In
              values: 
                - qa
        - matchExpressions:
            - key: env
              operator: In
              values:
                - us_east_2
                - eu_west_1
                - ap_south_1
          maxUpdate: 1 # maxUpdate supports both integer and percentage string values
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      source:
        repoURL: https://github.com/infra-team/cluster-deployments.git
        targetRevision: HEAD
        path: guestbook/{{cluster}}
      destination:
        server: '{{url}}'
        namespace: guestbook

In the above example, when the guestbook ApplicationSet is created or modified, the Application resources are each updated in the order defined in the rollout strategy. In this case, all Applications whose desired resource state matches the label expression env: dev are updated to match the template, as the maxUpdate is set to zero. The rolling update strategy progresses after the first set of Applications has successfully completed. Progress towards the next step starts only after the current step has completely finished, regardless of the maxUpdate value. The maxUpdate field only throttles the total number of matching Applications updating in the current step. After the first step completes, the ApplicationSet updates Application resources matching the second step's matchExpression together, as the maxUpdate was undefined. Finally, after the third step, the Application resources matching are updated, one by one, as the maxUpdate for the final step is 1.

An Application rollout is considered “complete” when the Application resource has been:

• Synced successfully.
• Moved into a “Progressing” state.
• Moved out of a “Progressing” state and into a “Healthy” state.

ApplicationSet creation with rollout strategy

Application resource creation from an ApplicationSet with a defined strategy looks much like the update process. When a brand new ApplicationSet is first created with a rollout strategy specified, the desired Application resource metadata labels are used to determine when each Application resource is created. Each Application created will be created in the order defined by the steps, if any, and advance to the next step only when a step completes successfully. The same applies if an ApplicationSet is modified to target a different set of destination clusters or namespaces, Applications are created or updated in the order defined by their desired state and the defined step order in the strategy.

ApplicationSet rollout failure

In the event that an ApplicationSet spec or template is modified and a target Application resource fails to “complete” a sync in any of the steps, the ApplicationSet rollout is stalled. The ApplicationSet resource will ensure the status field for “ApplicationSetUpToDate” is False. If the maxUpdate allows it, the ApplicationSet will continue updating Applications in the current step, but otherwise, no further changes will be propagated to Application resources by the ApplicationSet, and no steps will advance until each Application can successfully complete a sync. If the ApplicationSet is modified while still in the midst of an ApplicationSet rollout, stalled or otherwise, then the existing rollout is abandoned, the application resources are left in their present state, and the new rollout begins.

Pausing Application Changes

To implement the “paused” functionality of Applications that are not yet ready to be updated, we have a few options.

• Disable auto-sync.
  • Potentially conflicts with user provided auto-sync settings.
  • Provides the benefit of being able to see the full diff of the ApplicationSet change.
• “Pause” the Application.
  • Not Yet Implemented: Provide a way to pause syncing an application #4808
• Prevent any updates at all to the live Applications via the rolling update strategy defined.
  • Example implementation: Experimental AppSet progressive sync capability wmgroot/applicationset#1

Proof of Concept Implementation

We believe that the proof of concept implementation mentioned in option #3 above also demonstrates the strategy concept is firstmost, possible with minimal changes, and secondly, identifies the most likely code locations to implement the feature. The proof of concept utilizes hard-coded labels and annotations as a method of proving the idea can work, not as a guide for the implementation. A summary of the changes are included below:

• Retrieve a complete list of all current Applications.
• Build a map of which Applications belong in each rollout step.
  • This map includes the decision for whether or not each Application rollout step has been completed.
  • The logic to determine which Applications have already been updated depends on state tracked in the ApplicationSet status field, or in annotations set on each Application resource.
• Remove Applications from the list of desired Application updates.
  • Determine the rollout step for each Application.
  • If the previous rollout step is not yet complete, remove the current Application from the list of Applications to update, preventing the live Application from changing.

A full implementation can be built after the enhancement proposal has been approved and accepted by the community.
Co-written w/@wmgroot.