[single controller] feat: mitigate pickle cost #1862
Conversation
|
Changes LGTM. Tested with 40 nodes setup (TP=8, PP=10, CP=2, DP=2) and after this fix, the training time reduced by 50% :) |
|
@vermouth1992 will this feature reduce the time of VLMs fsdp dispatch? |
|
No, it won't reduce fsdp dispatch as there is no redundancy in fsdp dispatch method. Eventually, we have to use uri based methods for images/videos. |
Quote from OpenAI Codex agent that helped me understand the code logic LOL cc: @eric-haibin-lin |
|
@vermouth1992 sure, I'll refer to this PR and migrate it into verl |
Glad to hear that, I believe this can fix the performance issue from multi-modal datasets as described here: #1418 |
### Checklist Before Starting - [x] Search for similar PR(s). ### What does this PR do? ray put all the args in advance to avoid duplicate serialization cost for megatron dispatch. ### High-Level Design > Demonstrate the high-level design if this PR is complex. ### Specific Changes > List the specific changes. ### API > Demonstrate how the API changes if any. ### Usage Example > Provide usage example(s) for easier usage. ```python # Add code snippet or script demonstrating how to use this ``` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluatuion results, etc. ### Additional Info. - **Issue Number**: Fixes issue # or discussion # if any. - **Training**: [Note which backend this PR will affect: FSDP, Megatron, both, or none] - **Inference**: [Note which backend this PR will affect: vLLM, SGLang, both, or none] ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide). - [ ] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting). - [ ] Add `[BREAKING]` to the PR title if it breaks any API. - [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs). - [ ] New CI unit test(s) are added to cover the code path. - [ ] Rely on existing unit tests on CI that covers the code path.
Checklist Before Starting
What does this PR do?
ray put all the args in advance to avoid duplicate serialization cost for megatron dispatch.
High-Level Design
Specific Changes
API
Usage Example
# Add code snippet or script demonstrating how to use thisTest
Additional Info.
Checklist Before Submitting
[BREAKING]to the PR title if it breaks any API.