Hugging Face's logo
cesarpetrescu's picture
2 38

cesarpetrescu

cesar5514
Ameeeee's profile picture damianpumar's profile picture
·
https://photonspark.com

AI & ML interests

None yet

Recent Activity

liked a Space 21 days ago
a-ghorbani/ai-phone-leaderboard
liked a model 25 days ago
microsoft/Fara-7B
reacted to flozi00's post with 🔥 26 days ago
We recently discussed how Tensor Parallelism slices matrices to reduce latency within a single node. But what happens when you need to scale beyond that, where the bandwidth drops? That is where Pipeline Parallelism (PP) takes over. Instead of slicing the operation, PP slices the model depth. It turns your GPU cluster into an assembly line: GPU 0 handles layers 1-12, GPU 1 handles 13-24, and so on. The hardware challenge here isn't the interconnect speed—it is the "Pipeline Bubble." In a naive setup, expensive H100s sit idle for most of the cycle waiting for data to flow through the chain. My latest guide breaks down the scheduling strategies used to minimize this idle silicon time. In this deep dive, we cover: The Hardware Mechanics: Vertical Slicing Unlike TP which requires "chatty" All-Reduce operations, PP relies on lightweight Point-to-Point (Send/Recv) communication. This makes it the only viable strategy for crossing node boundaries over Ethernet or InfiniBand. Fighting the Bubble: 1F1B vs. GPipe We analyze the scheduling algorithms that keep the GPUs fed: GPipe: The "flush and fill" approach. Simple, but memory-intensive. 1F1B (One-Forward-One-Backward): The industry standard. By interleaving forward and backward passes, we aggressively free up memory and reduce the bubble size. The Math of Efficiency The "Bubble" is a mathematical inevitability. We look at the efficiency formula M+N−1 M ​ to understand why you need massive global batch sizes to make PP worth the effort. The article includes a conceptual PyTorch implementation of the 1F1B state machine to illustrate exactly how the data is handed off between stages. Read the full breakdown here: https://flozi.net/en/guides/ai/scaling/pipeline_parallel
View all activity

Organizations

None yet