A new architecture called Prefill-as-a-Service (PrfaaS) is challenging a long-standing assumption in AI infrastructure: that the two phases of large language model (LLM) inference—prefill and decode—must stay in the same data center. By separating them across different geographic locations, the approach dramatically increases throughput while reducing costs.
Traditional LLM serving couples prefill and decode because transferring the key-value cache (KVCache) needed for generation requires enormous bandwidth. However, modern hybrid-attention models compress this data into a fraction of its original size, making transfers over standard commodity Ethernet feasible.
PrfaaS uses selective offloading: only computationally heavy, long-context requests are sent to specialized remote clusters for prefill. A dual-timescale scheduler then balances hardware workloads and adapts to fluctuating network conditions in real time.
Experimental results show that cross-datacenter prefill substantially boosts total throughput without sacrificing latency. The work, detailed in a recent arXiv paper, points toward a future where AI compute is no longer constrained by the walls of a single facility.