Rolling deployment for services #2821
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Support rolling deployment when some service configuration properties are changed. During the deployment, `dstack` updates service replicas one by one. It first starts a new version of the replica, then waits until it is running, then stops the old version of the replica. As part of the implementation, introduce a new run and job property - `deployment_num`. When a new configuration is applied, the `deployment_num` of the run is incremented. Then, `dstack` gradually updates the jobs so that their `deployment_num` matches that of the run. Some jobs are updated in-place, if the new configuration does not affect their spec, others are redeployed as described above. ```shell > dstack apply Active run test-service already exists. Detected configuration changes that can be updated in-place: ['image', 'env', 'commands'] Update the run? [y/n]: y ⠋ Launching test-service... NAME BACKEND RESOURCES PRICE STATUS SUBMITTED test-service deployment=1 running 11 mins ago replica=0 job=0 deployment=0 aws (us-west-2) cpu=2 mem=1GB disk=100GB (spot) $0.0026 terminating 11 mins ago replica=1 job=0 deployment=1 aws (us-west-2) cpu=2 mem=1GB disk=100GB (spot) $0.0026 running 1 min ago ```
|
I'm yet to test on Postgres and possibly add some unit tests. The rest should be ready for the initial release |
un-def
approved these changes
Jun 19, 2025
haydnli-shopify
pushed a commit
to haydnli-shopify/dstack
that referenced
this pull request
Jun 23, 2025
Support rolling deployment when some service configuration properties are changed. During the deployment, `dstack` updates service replicas one by one. It first starts a new version of the replica, then waits until it is running, then stops the old version of the replica. As part of the implementation, introduce a new run and job property - `deployment_num`. When a new configuration is applied, the `deployment_num` of the run is incremented. Then, `dstack` gradually updates the jobs so that their `deployment_num` matches that of the run. Some jobs are updated in-place, if the new configuration does not affect their spec, others are redeployed as described above. ```shell > dstack apply Active run test-service already exists. Detected configuration changes that can be updated in-place: ['image', 'env', 'commands'] Update the run? [y/n]: y ⠋ Launching test-service... NAME BACKEND RESOURCES PRICE STATUS SUBMITTED test-service deployment=1 running 11 mins ago replica=0 job=0 deployment=0 aws (us-west-2) cpu=2 mem=1GB disk=100GB (spot) $0.0026 terminating 11 mins ago replica=1 job=0 deployment=1 aws (us-west-2) cpu=2 mem=1GB disk=100GB (spot) $0.0026 running 1 min ago ```
r4victor
reviewed
Jun 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support rolling deployment when some service
configuration properties are changed. During the
deployment,
dstackupdates service replicas oneby one. It first starts a new version of the
replica, then waits until it is running, then
stops the old version of the replica.
As part of the implementation, introduce a new run
and job property -
deployment_num. When a newconfiguration is applied, the
deployment_numofthe run is incremented. Then,
dstackgraduallyupdates the jobs so that their
deployment_nummatches that of the run. Some jobs are updated
in-place, if the new configuration does not affect
their spec, others are redeployed as described
above.
Initial and main part of #2180