Support registration of external backends #655
Conversation
Codecov Report
@@ Coverage Diff @@
## master #655 +/- ##
==========================================
+ Coverage 95.09% 95.12% +0.03%
==========================================
Files 39 39
Lines 5462 5478 +16
==========================================
+ Hits 5194 5211 +17
+ Misses 268 267 -1
Continue to review full report at Codecov.
|
|
Any feedback on this idea? |
|
I was thinking that maybe we could even move the code of the dask backend into the joblib code base because the backend semi-private API is more subject to change than the dask distributed public API. I would like to find some time in the coming month to automatically scatter heavy data to avoid redundant data transfers without having the use to explicitly use the |
|
That would work for me |
|
This code is fairly lightweight, and it is gives benefit to the user, so I have no major issues with the idea. Two points:
|
Fixes joblib#653 Currently when using external backends with joblib you need to know to `import distributed.joblib`, which is not particularly intuitive. ```python from joblib import parallel_backend with parallel_backend("dask"): pass KeyError: 'dask' ``` Now implementers of downstream parallel backends can register small functions within the Joblib codebase that register their backends appropriately when triggered by users.
6e4a451 to
cc562fb
Compare
|
Those both seem reasonable to fix.
I agree that this would be useful. Short-term I've added a more informative error.
Done. Feedback appreciated to check for tone and and detail. I've also changed the dask example. |
|
LGTM. Merging! |
|
Actually, let me wait for travis |
88ff6af to
65bd019
Compare
65bd019 to
f42b7b5
Compare
|
Tests pass except for codecov. I'm not sure how to interpret the results there. |
|
OK, merging. Thank you. |
* tag '0.12': (116 commits) Release 0.12 typo typo typo ENH add initializer limiting n_threads for C-libs (joblib#701) DOC better parallel docstring (joblib#704) [MRG] Nested parallel call thread bomb mitigation (joblib#700) MTN vendor loky2.1.3 (joblib#699) Make it possible to configure the reusable executor workers timeout (joblib#698) MAINT increase timeouts to make test more robust on travis DOC: use the .joblib extension instead of .pkl (joblib#697) [MRG] Fix exception handling in nested parallel calls (joblib#696) Fix skip test lz4 not installed (joblib#695) [MRG] numpy_pickle: several enhancements (joblib#626) Introduce Parallel.__call__ backend callbacks (joblib#689) Add distributed on readthedocs (joblib#686) Support registration of external backends (joblib#655) [MRG] Add a dask.distributed example (joblib#613) ENH use cloudpickle to pickle interactively defined callable (joblib#677) CI freeze the version of sklearn0.19.1 and scipy1.0.1 (joblib#685) ...
Fixes #653
Currently when using external backends with joblib you need to know to
import distributed.joblib, which is not particularly intuitive.Now implementers of downstream parallel backends can register small
functions within the Joblib codebase that register their backends
appropriately when triggered by users.
FWIW I suspect that not everyone will agree with this change. I will not be surprised if it does not get merged. It was simple enough to implement though that I thought I'd throw it up here just to make conversation about it easier.