-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
RFC - Improving deploying Dask docs #9912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for writing this up Sarah! Some of this seems good to me, some not. I've put some remarks here but mostly I think we leave this and wait and see what other non Coiled affiliated people think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
page is intended as a hub for other relevant Dask projects (i.e. Dask Cloud Provider, Dask Kubernetes, and Dask Gateway) to link back to. This could be a sentence at the top of the docs index for these projects. Something like "Dask Gateway is one of many ways to deploy Dask; for an overview of other options see the Dask documentation
Such a page sounds a lot like https://docs.dask.org/en/stable/deploying.html to me. That already looks like a "hub" page that lists and links to the many options for deploying Dask.
It's a dense list right now, and we could probably do a lot to make it easier to navigate:
- cut down and summarize the current prose into ~1 paragraph
- organize the list of links by where you're deploying dask to (single machine, HPC, k8s, cloud, ad-hoc cluster
- perhaps also organize the list (via tabs at the top?) by who you're deploying dask for: "just me" vs "my team"? For example, dask-gateway likely isn't relevant to a single user; dask-cloudprovider is probably not a good solution for a team. Some options, including the commercial solutions, would be in both.
So basically, I like the idea of having a good "hub" page that makes it easy for users to navigate all the different options for deploying Dask. I just think the title of that page is probably "Deploy Dask Clusters", not "Dask Deployment Best Practices". Since "Deploy Dask Clusters" already exists, maybe we should be improving that page, not making a new one?
Also, in the "For IT" section of the FAQ https://docs.dask.org/en/stable/faq.html#how-would-i-set-up-dask-on-institutional-hardware, it would make a lot of sense to mention commercial services. (Also in the "How do I manage users" section.) |
Co-authored-by: Gabe Joseph <gjoseph92@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good. I think there's more work we can do to the deployment page (I still find the text a bit long, and having the individual vs team split would be nice), but I think we should tackle that incrementally. This is already an improvement to the organization!
Co-authored-by: Gabe Joseph <gjoseph92@gmail.com>
Co-authored-by: Gabe Joseph <gjoseph92@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, thanks for the effort here.
pre-commit run --all-files
RFC PR for making it easier for those looking for Dask deployment options to find available options (including Coiled).
This includes a draft for a new docs page on deployment best practices, based on @mrocklin's PyData NYC 2022 talk.This deployment best practices page is intended as a hub for other relevant Dask projects (i.e. Dask Cloud Provider, Dask Kubernetes, and Dask Gateway) to link back to. This could be a sentence at the top of the docs index for these projects. Something like "Dask Gateway is one of many ways to deploy Dask; for an overview of other options see the Dask documentation on Deploying Best Practices." We can also link to this page from Dask examples that mention deployment options (e.g. https://examples.dask.org/applications/embarrassingly-parallel.html#Start-Dask-Client-for-Dashboard)Edit - "Deployment Considerations" page was added in #9933.
This PR also includes other suggestions of where to link to Coiled, where appropriate.
cc @jacobtomlinson @dchudz @gjoseph92