Characterizing the Impact of Congestion in Modern HPC Interconnects

This research paper presentation by Lorenzo Perulli, a PhD student at Sapienza University of Rome working with collaborators from ENEA, Huawei, and Cineca, characterizes how network congestion degrades application performance on modern production HPC interconnects. The motivation is that supercomputers are shared environments where many users run HPC simulations and AI training workloads relying on collective operations, while network capability has not kept pace with compute growth, making congestion and link saturation significant sources of performance variability. The study implements two congestion patterns using collectives: steady congestion, generated by a continuously executing collective that runs until the victim application finishes, and bursty congestion, where collectives are interleaved with pauses. All-to-all traffic generates intermediate congestion and in-cast traffic generates edge congestion. Experiments span five production fabrics, including InfiniBand EDR, HDR, and NDR generations, Slingshot, and an Ethernet-based fabric, scaling to 256 nodes on the production systems Leonardo, Cresco8, and LUMI. Results from 64-node runs show that in-cast congestion causes the most severe degradation, hitting Cresco8 and Leonardo hard, while LUMI tolerates it better, likely due to the synergy between its fabric and topology. The presenter concludes that despite generational progress, congestion remains an open problem: in-cast produces strong edge congestion and short traffic bursts expose limitations, so next-generation networks need better congestion control and load balancing rather than merely faster links, with Ultra Ethernet a hoped-for step in that direction.