Describe the user problem that you are trying to solve

Operate does not support to reconfigure the number of replicas for a shard if the configuration changes. It also doesn't support changing the number of shards for a template (so that it applies to indexes created in the future). The docs point out this limitation: https://docs.camunda.io/docs/self-managed/operate-deployment/operate-configuration/#settings-for-shards-and-replicas

Our use case requires that we are able to scale up elasticsearch and also change the number of replicas to increase the availability and usage of the nodes.

Describe the solution you'd like to see implemented

• When I restart Operate with a changed configuration of number_of_replicas
  • The it is applied to both, existing templates and existing indexes on startup (for indexes, also taking the numberOfReplicasForIndices setting into account)
• When I restart Operate with a changed configuration of number_of_shards
  • Then it is applies to existing templates
  • It is not applied to existing indexes (as that would require splitting or shrinking the index, a fairly heavy operation); this limitation is documented

Optimize already has a way to do this.

This then allows us to increase the the elasticsearch pods and also the data replication.

Describe alternatives you've considered

Additional Context

• Context here in #tmp-ha-es -> https://camunda.slack.com/archives/C05P0D3AER2/p1694424927422869
• Blocking migration of existing clusters in PDP ticket https://github.com/camunda/product-hub/issues/667
• https://jira.camunda.com/browse/SUPPORT-23300

Definition of Ready - Checklist

• The issue is not duplicated - the is no other issue about the same user problem in Operate/Tasklist/PDP repository
• The issue has a meaningful title, description and clear user problem