Skip to content

Conversation

scharlottej13
Copy link
Contributor

@scharlottej13 scharlottej13 commented Feb 2, 2023

  • Closes #xxxx
  • Tests added / passed
  • Passes pre-commit run --all-files

RFC PR for making it easier for those looking for Dask deployment options to find available options (including Coiled). This includes a draft for a new docs page on deployment best practices, based on @mrocklin's PyData NYC 2022 talk.

This deployment best practices page is intended as a hub for other relevant Dask projects (i.e. Dask Cloud Provider, Dask Kubernetes, and Dask Gateway) to link back to. This could be a sentence at the top of the docs index for these projects. Something like "Dask Gateway is one of many ways to deploy Dask; for an overview of other options see the Dask documentation on Deploying Best Practices." We can also link to this page from Dask examples that mention deployment options (e.g. https://examples.dask.org/applications/embarrassingly-parallel.html#Start-Dask-Client-for-Dashboard)

Edit - "Deployment Considerations" page was added in #9933.

This PR also includes other suggestions of where to link to Coiled, where appropriate.

cc @jacobtomlinson @dchudz @gjoseph92

Copy link
Member

@mrocklin mrocklin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing this up Sarah! Some of this seems good to me, some not. I've put some remarks here but mostly I think we leave this and wait and see what other non Coiled affiliated people think.

Copy link
Collaborator

@gjoseph92 gjoseph92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

page is intended as a hub for other relevant Dask projects (i.e. Dask Cloud Provider, Dask Kubernetes, and Dask Gateway) to link back to. This could be a sentence at the top of the docs index for these projects. Something like "Dask Gateway is one of many ways to deploy Dask; for an overview of other options see the Dask documentation

Such a page sounds a lot like https://docs.dask.org/en/stable/deploying.html to me. That already looks like a "hub" page that lists and links to the many options for deploying Dask.

It's a dense list right now, and we could probably do a lot to make it easier to navigate:

  • cut down and summarize the current prose into ~1 paragraph
  • organize the list of links by where you're deploying dask to (single machine, HPC, k8s, cloud, ad-hoc cluster
  • perhaps also organize the list (via tabs at the top?) by who you're deploying dask for: "just me" vs "my team"? For example, dask-gateway likely isn't relevant to a single user; dask-cloudprovider is probably not a good solution for a team. Some options, including the commercial solutions, would be in both.

So basically, I like the idea of having a good "hub" page that makes it easy for users to navigate all the different options for deploying Dask. I just think the title of that page is probably "Deploy Dask Clusters", not "Dask Deployment Best Practices". Since "Deploy Dask Clusters" already exists, maybe we should be improving that page, not making a new one?

@gjoseph92
Copy link
Collaborator

Also, in the TOC, it might be worth adding another page for "Commercial services", "Managed services", "Dask-as-a-Service", something like that.

Screen Shot 2023-02-04 at 12 57 54 PM

@gjoseph92
Copy link
Collaborator

Also, in the "For IT" section of the FAQ https://docs.dask.org/en/stable/faq.html#how-would-i-set-up-dask-on-institutional-hardware, it would make a lot of sense to mention commercial services. (Also in the "How do I manage users" section.)

@github-actions github-actions bot added the documentation Improve or add to documentation label Feb 8, 2023
Copy link
Collaborator

@gjoseph92 gjoseph92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. I think there's more work we can do to the deployment page (I still find the text a bit long, and having the individual vs team split would be nice), but I think we should tackle that incrementally. This is already an improvement to the organization!

@scharlottej13 scharlottej13 marked this pull request as ready for review February 17, 2023 01:43
Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks for the effort here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improve or add to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants