[KubeRay][Docs] Simplify KubeRay upgrade steps and always upgrade CRDs first #44826

andrewsykim · 2024-04-18T02:08:31Z

Why are these changes needed?

Simplifes the KubeRay upgrade steps and strongly recommends upgrading CRDs first. Also removes recommendation for deleting CRDs as this will result in RayCluster, RayJob and RayService resources being deleted.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

andrewsykim · 2024-04-18T02:18:26Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+# upgrade the CRD to v1.1.0
+# NOTE: here we use kubectl because Helm does not support lifecycle management of CRDs
+# See the Helm documentation for more details: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations
+$ kubectl apply --server-side --force-conflicts -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.1.0"


The --force-conflict is needed here to resolve server-side conflicts with helm:

$ kubectl apply --server-side -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.1.0" Apply failed with 2 conflicts: conflicts with "helm" using apiextensions.k8s.io/v1: - .metadata.annotations.controller-gen.kubebuilder.io/version - .spec.versions Please review the fields above--they currently have other managers. Here are the ways you can resolve this warning: * If you intend to manage all of these fields, please re-run the apply command with the `--force-conflicts` flag. * If you do not intend to manage all of the fields, please edit your manifest to remove references to the fields that should keep their current managers. * You may co-own fields by updating your manifest to match the existing value; in this case, you'll become the manager if the other manager(s) stop managing the field (remove it from their configuration). See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

I think it's fine to force resolve these conflicts because Helm is not managing CRDs after initial install anyways.

andrewsykim · 2024-04-18T02:21:55Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+2. Upgrade the kuberay-operator image to the new version
+3. Verify successful upgrade
+
+Here's an example when upgrading KubeRay from v1.0.0 to v1.1.0:


Should we still provide step-by-step example for the v0.6.0 -> v1.0.0 upgrade?

Added an example for v0.6.0 -> v1.0.0 for consistency with the old docs

kevin85421

I will test this manually.

Yicheng-Lu-llll · 2024-04-23T10:45:59Z

LGTM! I manually tested it, and everything works as expected.

kevin85421 · 2024-04-23T16:47:25Z

@andrewsykim I have already requested our doc team to review this PR.

angelinalg

Just some nits.

angelinalg · 2024-04-23T16:50:40Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

-    # Install RayCluster v1.0.0 which uses CRD v1.
-    helm install raycluster kuberay/ray-cluster --version 1.0.0
-    ```
+Due to reliability and security implication of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.


Suggested change

Due to reliability and security implication of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.

Due to reliability and security implications of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.

angelinalg · 2024-04-23T16:50:56Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

-    ```
+Due to reliability and security implication of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.
+
+To upgrade the KubeRay version, it's highly recommended to follow these steps in order:


Suggested change

To upgrade the KubeRay version, it's highly recommended to follow these steps in order:

To upgrade the KubeRay version, follow these steps in order:

angelinalg · 2024-04-23T16:51:04Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+
+To upgrade the KubeRay version, it's highly recommended to follow these steps in order:
+1. Upgrade the CRD manifest, containing new fields added to the v1 CRDs.
+2. Upgrade the kuberay-operator image to the new version


Suggested change

2. Upgrade the kuberay-operator image to the new version

2. Upgrade the kuberay-operator image to the new version.

angelinalg · 2024-04-23T16:51:26Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+To upgrade the KubeRay version, it's highly recommended to follow these steps in order:
+1. Upgrade the CRD manifest, containing new fields added to the v1 CRDs.
+2. Upgrade the kuberay-operator image to the new version
+3. Verify successful upgrade


Suggested change

3. Verify successful upgrade

3. Verify the success of the upgrade.

angelinalg · 2024-04-23T16:51:50Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+2. Upgrade the kuberay-operator image to the new version
+3. Verify successful upgrade
+
+Here's an example when upgrading KubeRay from v1.0.0 to v1.1.0:


Suggested change

Here's an example when upgrading KubeRay from v1.0.0 to v1.1.0:

The following is an example of upgrading KubeRay from v1.0.0 to v1.1.0:

angelinalg · 2024-04-23T16:53:12Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+$ helm install raycluster kuberay/ray-cluster --version 1.1.0
+```
+
+Here's an example when upgrading KubeRay from v0.6.0 to v1.0.0:


Suggested change

Here's an example when upgrading KubeRay from v0.6.0 to v1.0.0:

The following is an example of upgrading KubeRay from v0.6.0 to v1.0.0:

angelinalg · 2024-04-23T16:53:35Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+
+Here's an example when upgrading KubeRay from v0.6.0 to v1.0.0:
+```
+# upgrade the CRD to v1.0.0


Suggested change

# upgrade the CRD to v1.0.0

# Upgrade the CRD to v1.0.0.

angelinalg · 2024-04-23T16:54:00Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+Here's an example when upgrading KubeRay from v0.6.0 to v1.0.0:
+```
+# upgrade the CRD to v1.0.0
+# NOTE: here we use kubectl because Helm does not support lifecycle management of CRDs


Suggested change

# NOTE: here we use kubectl because Helm does not support lifecycle management of CRDs

# Note: This example uses kubectl because Helm doesn't support lifecycle management of CRDs.

angelinalg · 2024-04-23T16:54:24Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+# See the Helm documentation for more details: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations
+$ kubectl apply --server-side --force-conflicts -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.0.0"
+
+# Upgrade kuberay-operator to v1.0.0. This step does not upgrade the CRDs.


Suggested change

# Upgrade kuberay-operator to v1.0.0. This step does not upgrade the CRDs.

# Upgrade kuberay-operator to v1.0.0. This step doesn't upgrade the CRDs.

angelinalg · 2024-04-23T16:54:47Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+# Upgrade kuberay-operator to v1.0.0. This step does not upgrade the CRDs.
+$ helm upgrade kuberay-operator kuberay/kuberay-operator --version v1.0.0
+
+# Install a RayCluster using the v1.0.0 helm chart to verify successful upgrade


Suggested change

# Install a RayCluster using the v1.0.0 helm chart to verify successful upgrade

# Install a RayCluster using the v1.0.0 helm chart to verify the success of the upgrade.

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

andrewsykim · 2024-04-24T04:04:02Z

FYI I updated the doc to use kubectl replace instead of kubectl apply after discussion with Matyas and today's community meeting

smit-kiri · 2024-04-24T13:22:19Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+# Upgrade the CRD to v1.0.0.
+# Note: This example uses kubectl because Helm doesn't support lifecycle management of CRDs.
+# See the Helm documentation for more details: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations
+$ kubectl replace -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.0.0"


FYI this step causes disruptions for any running RayService resources until you apply the RayService config again. See this slack thread for more details: https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1701370178891199

Is that specific to upgrading the CRD first only or are you referring to the entire v0.6.0 to v1.0.0 upgrade? As far as I'm aware the 0.6.0 -> 1.0.0 is known to be disruptive and breaking changes like that are expected but I'm not super familiar with RayService changes from 0.6.0 and 1.0.0 yet to understand what is expected

From your comment here https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1701443938862159?thread_ts=1701370178.891199&cid=C02GFQ82JPM

it looks like upgrading the CRD first didn't cause any issues, but it was the operator upgrade that was problematic. Maybe we need specific guidnace for 0.6.0 -> 1.0.0 based on known upgrade issues?

Ah yes sorry, upgrading the operator causes these issues.
I think 0.6.0 -> 1.0.0 should have specific guidance

cc @andrewsykim do you plan to address this comment in this PR? If you want to do it in a follow-up PR, I am comfortable merging this PR.

If you're planning to add the 0.6.0 -> 1.0.0 guidance in a separate PR, I'd suggest changing the example here to show upgrade from 1.0.0 -> 1.1.0 or something like that otherwise it could be misleading

@smit-kiri are you interested in opening a PR to share your experience after this PR got merged?

The old guidance suggests deleting the CRD, so I think what we have in this PR is still better

I don't have enough experience with 0.6.0 to 1.0.0 upgrade issues right now so it'll have to be a follow up PR if we add one.

@kevin85421 Yeah I can open a PR!

Thanks, @smit-kiri!

kevin85421 · 2024-04-24T17:33:51Z

doc/source/cluster/kubernetes/user-guides/upgrade-guide.md

+# Upgrade the CRD to v1.1.0.
+# Note: This example uses kubectl because Helm doesn't support lifecycle management of CRDs.
+# See the Helm documentation for more details: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations
+$ kubectl replace -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.1.0"


What's the reason for using kubectl replace? Is it to avoid the size limit of kubectl.kubernetes.io/last-applied-configuration?

Yes. Using kubectl apply --server-side also works around the size limit, but I think kubectl replace here is fine because we do not expect multiple writers to the CRD anyways and the manifest in release should be the source of truth overwriting any previous updates by other writers

With kubectl replace we need to be careful not to use the --force command because that will delete the CRD and recreate it. But I think it's fine here as long as we don't recommend it

Using kubectl apply --server-side --force-conflicts -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.0.0" also works well, so I'm fine with putting either one in the docs. kubectl replace seems a bit cleaner

Got it. Thank you for the explanation!

andrewsykim requested review from architkulkarni, maxpumperla, pcmoritz, kevin85421 and a team as code owners April 18, 2024 02:08

andrewsykim force-pushed the kuberay-api-docs branch from f86fe51 to fce2c50 Compare April 18, 2024 02:15

andrewsykim commented Apr 18, 2024

View reviewed changes

andrewsykim mentioned this pull request Apr 18, 2024

[Bug] All worker Pods are deleted if using KubeRay v1.0.0 CRD with KubeRay operator v1.1.0 image ray-project/kuberay#2087

Merged

4 tasks

andrewsykim commented Apr 18, 2024

View reviewed changes

andrewsykim force-pushed the kuberay-api-docs branch from fce2c50 to f2f4f96 Compare April 18, 2024 02:40

richardsliu approved these changes Apr 18, 2024

View reviewed changes

kevin85421 self-assigned this Apr 19, 2024

kevin85421 reviewed Apr 19, 2024

View reviewed changes

Yicheng-Lu-llll approved these changes Apr 23, 2024

View reviewed changes

kevin85421 approved these changes Apr 23, 2024

View reviewed changes

angelinalg approved these changes Apr 23, 2024

View reviewed changes

andrewsykim force-pushed the kuberay-api-docs branch 2 times, most recently from 27d4c96 to 0f6847e Compare April 24, 2024 01:44

Simplify KubeRay upgrade steps and always upgrade CRDs first

1f69ed4

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

andrewsykim force-pushed the kuberay-api-docs branch from 0f6847e to 1f69ed4 Compare April 24, 2024 02:59

smit-kiri reviewed Apr 24, 2024

View reviewed changes

kevin85421 reviewed Apr 24, 2024

View reviewed changes

jjyao merged commit f6ba2e7 into ray-project:master Apr 26, 2024

	Due to reliability and security implication of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.
	Due to reliability and security implications of webhooks, KubeRay doesn't support a conversion webhook to convert v1alpha1 to v1 APIs.

	To upgrade the KubeRay version, it's highly recommended to follow these steps in order:
	To upgrade the KubeRay version, follow these steps in order:

	2. Upgrade the kuberay-operator image to the new version
	2. Upgrade the kuberay-operator image to the new version.

	3. Verify successful upgrade
	3. Verify the success of the upgrade.

	Here's an example when upgrading KubeRay from v1.0.0 to v1.1.0:
	The following is an example of upgrading KubeRay from v1.0.0 to v1.1.0:

	# NOTE: here we use kubectl because Helm does not support lifecycle management of CRDs
	# Note: This example uses kubectl because Helm doesn't support lifecycle management of CRDs.

	# Upgrade kuberay-operator to v1.0.0. This step does not upgrade the CRDs.
	# Upgrade kuberay-operator to v1.0.0. This step doesn't upgrade the CRDs.

	# Install a RayCluster using the v1.0.0 helm chart to verify successful upgrade
	# Install a RayCluster using the v1.0.0 helm chart to verify the success of the upgrade.

[KubeRay][Docs] Simplify KubeRay upgrade steps and always upgrade CRDs first #44826

[KubeRay][Docs] Simplify KubeRay upgrade steps and always upgrade CRDs first #44826

Uh oh!

Conversation

andrewsykim commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin85421 left a comment

Choose a reason for hiding this comment

Uh oh!

Yicheng-Lu-llll commented Apr 23, 2024

Uh oh!

kevin85421 commented Apr 23, 2024

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim commented Apr 24, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsykim Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

andrewsykim commented Apr 18, 2024 •

edited

Loading

andrewsykim Apr 18, 2024 •

edited

Loading

andrewsykim Apr 24, 2024 •

edited

Loading

andrewsykim Apr 24, 2024 •

edited

Loading