Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity
[Sztuczna inteligencja] Autorzy: Jean Lucien Randrianantenaina, Hung Nguyen, Nasib Ullah i 4 więcej
Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pru…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja