[Mcore] context parallel #970
Conversation
|
thanks a ton for quick support! May I know whether you have done some benchmarking or testing of training efficiency upon the context parallel? |
I tried with 1 node with 8 H100, comparing tp4dp2cp1 with tp4dp1cp2. cp2(gray line) is slower than cp1 in this test. The result is reasonable because it is not a memory limited situation, with less data parallel and more communication. CP would be useful when the sequence length is larger, but so far, I have only focused on whether the functions have been implemented, and I haven't had the time to pay attention to further performance testing. |
| #TODO: support ep | ||
| return os.path.join(checkpoint_path, f"optim", f"distrib_optim_pp{pp_rank}_tp{tp_rank}.pt") | ||
| return os.path.join(checkpoint_path, f"optim", f"distrib_optim_pp{pp_rank}_tp{tp_rank}_cp{cp_rank}_dp{dp_rank}.pt") |
There was a problem hiding this comment.
@ETOgaosion due to the optimizer states are distributed across all gpus, the dp rank also should be saved separately.
There was a problem hiding this comment.
Get~ Later will sync to the doc.
support context parallel for mcore backend. Changes on: * configs * model loader * checkpint * single control dispatcher * forward preprocess and postprocess --------- Co-authored-by: gaoziyuan <gaoziyuan.955@bytedance.com>
support context parallel for mcore backend. Changes on: * configs * model loader * checkpint * single control dispatcher * forward preprocess and postprocess --------- Co-authored-by: gaoziyuan <gaoziyuan.955@bytedance.com>
support context parallel for mcore backend. Changes on: * configs * model loader * checkpint * single control dispatcher * forward preprocess and postprocess --------- Co-authored-by: gaoziyuan <gaoziyuan.955@bytedance.com>
support context parallel for mcore backend.
Changes on: