FlashVector: Agent for Hierarchical Model Serving Stack Optimization
[Sztuczna inteligencja] Autorzy: Qi Wu, Lohan Lemire, Kai Meng i 8 więcej
Model serving is one of the largest cost drivers in production recommender systems. Maximizing its throughput requires navigating a deeply layered hierarchy: GPU kernels, the ML framework computation graph, the model server, and on-demand feature processing -- each demanding specialized domain expertise. Such cross-layer expertise is inherently difficult to acquire, and does not scale with a workload that continuously grows and evolves, leaving significant cost efficiency gains unrealized. While recent AI agents have demonstrated human expert level efficiency in standalone GPU kernel optimization, automated tuning and optimization for the rest of the serving stack remain largely unexplored. We present FlashVector, an agentic system that optimizes performance across all layers of the model serving stack. The key contribution is an extensible framework to generalize the single kernel opti…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja