Tensor Parallelism vs Pipeline Parallelism for Training

Quick Verdict: Tensor parallelism is a model-split that partitions each layer across GPUs that can exchange activations quickly; pipeline parallelism is a depth-split that places consecutive layers on different stages and pays a bubble if those stages idle. Use tensor parallel inside a high-bandwidth domain. Use pipeline parallel when the model no longer fits that domain and you can keep stages busy with microbatches.

The wrong split wastes interconnect or leaves GPUs waiting. Do not pick a strategy from a logo on a framework slide. Pick it from memory per layer, collective size, and whether your fabric is a node or a cluster.

Tensor parallelism vs pipeline parallelism decision table

Frameworks often combine both with data parallel replicas. That is 3D parallelism, not a reason to skip the pairwise choice. Get the two-way split right first.

When tensor parallelism wins for training

Tensor parallel wins when a single layer is too wide for one GPU and the shards must talk on every step. Attention heads and large feed-forward blocks are the usual cut. The tax is communication frequency. If those collectives leave the NVLink domain and ride a congested Ethernet hop, step time becomes a network problem dressed as a model problem.

Keep tensor-parallel degree inside the domain your vendor can keep lossless and low-jitter. On many NVIDIA HGX-class nodes that domain is the NVLink island. Crossing nodes with a large tensor-parallel size is possible. It is also how teams discover that NCCL timeouts were a fabric choice, not a PyTorch bug.

AI networking is the constraint most buyers under-price here. OneSource Cloud treats the compute fabric as part of cluster design so a tensor-parallel job is not dropped onto a campus network and asked to behave like an NVLink domain.

When pipeline parallelism is the better split

Pipeline parallel wins when depth, not width, is the leftover problem. After you have used tensor parallel to the edge of the fast domain, remaining layers still need a home. Staging them reduces per-GPU parameter load. The cost is the bubble: early stages wait for the pipeline to fill, and late stages wait at the end of the batch.

Microbatch count, schedule (GPipe-style flush versus 1F1B), and imbalance between stages decide whether the bubble is a footnote or the run. A vision-language tower that is heavier than the language stages will idle the cheap GPUs behind it. Rebalance layers before you buy another node “for pipeline.”

Pipeline also changes failure domains. A stage that OOMs or loses a peer wastes the in-flight microbatches. Checkpoint placement and restart policy belong in the same design review as the parallel plan. AI storage architecture matters once those checkpoints are large enough that restart time exceeds the bubble you just optimized.

How to choose without a framework bake-off theater

Run a short, boring sequence on a representative slice of the model.

1. Measure one-layer memory and the collective time for your candidate tensor-parallel sizes.
2. Raise tensor-parallel degree only while collectives stay inside the fast domain and step time still falls.
3. Only then add pipeline stages, and sweep microbatch count until bubble time stops falling.
4. Add data-parallel replicas last, after the model-parallel unit is stable.
5. Repeat after a precision or sequence-length change; both move the winning split.

Dedicated private AI infrastructure does not pick the algorithm for you. It keeps the topology you sized from being shared with an unrelated job that jitters collectives. If several research groups will change parallel plans weekly, OnePlus, OneSource Cloud’s AI orchestration platform, is the control plane that records which job used which split so the next incident is not folklore.

FAQ

What is tensor parallelism versus pipeline parallelism?

Tensor parallelism shards work inside a layer across GPUs and communicates on the critical path. Pipeline parallelism assigns different layers to different stages and sends activations forward and gradients back. Tensor parallel is a width strategy. Pipeline parallel is a depth strategy. Most large training stacks use both, plus data parallel replicas.

Do I need pipeline parallelism for a 7B fine-tune?

Often no. A 7B model on modern 80 GB-class GPUs can stay on data parallel, or a small tensor-parallel size, unless sequence length or optimizer state explodes memory. Pipeline becomes interesting when the model, context, and optimizer no longer fit the fast domain even after sharding layers.

Does tensor parallelism always require InfiniBand?

No. Intra-node tensor parallel usually rides NVLink or an equivalent GPU fabric. Inter-node tensor parallel needs a cluster fabric that can hold collective tail latency. InfiniBand is a common answer at training scale. A well-tuned Ethernet fabric can work for smaller jobs. Measure NCCL on your topology instead of assuming a logo.

How do microbatches affect pipeline cost?

More microbatches can hide the bubble by keeping stages busy. They also increase activation memory and change the effective batch that reaches the optimizer. If you raise microbatches until you OOM, you did not cheapen the pipeline. You moved the failure from idle time to memory. Sweep both metrics together.

Should inference use the same parallel plan as training?

Not automatically. Inference cares about TTFT, decode, and KV-cache placement. Training cares about step time and optimizer state. A tensor-parallel inference graph may match training shards for weight loading. Pipeline-parallel inference has a different latency profile. Re-validate serving on the plan you will ship, not the plan that trained the checkpoint.

Summary

Tensor vs pipeline parallelism is a communication-and-bubble decision, not a prestige setting. Keep tensor parallel inside the fast GPU domain, add pipeline when depth remains, and prove it with collectives and microbatch sweeps. If you need a dedicated topology to make that plan hold, review OneSource Cloud private AI infrastructure as the environment, not as a substitute for the split itself.