Summary:
Pipeline parallelism issues all send/recv on a single communicator.
NCCL-group coalescing (batch_isend_irecv -> one ncclGroupStart/End) makes a single mixed
send+recv batch deadlock-free, but it does not remove ordering hazards across
separate batches.
This adds per-direction PP communicators. When enabled, forward (
r->r+1) and backward (r->r-1) P2P are carried on two separate
communicators derived from the stage's group via split_group.
Test Plan:
test_pp_per_direction_pipeline_stage.pytest_pp_torchcomms_p2p_deadlock.py
SOCIAL SHARE CARD GENERATOR