A new architecture called Prefill-as-a-Service (PrfaaS) promises to overcome one of AI's most stubborn scaling challenges: the bandwidth wall that keeps large language models tied to single datacenters. By splitting the prefill and decode phases across different locations, researchers have demonstrated that cross-datacenter inference is feasible using standard commodity Ethernet.
Traditional LLM serving keeps both phases co-located because transferring the Key-Value Cache (KVCache) requires enormous bandwidth. But modern hybrid-attention models drastically shrink the KVCache size, making transfers economical. PrfaaS selectively offloads only the most compute-intensive long-context requests to specialized remote clusters, while a dual-timescale scheduler balances hardware loads and adapts to real-time network conditions.
Experimental results confirm that the approach maintains low latency and high throughput, effectively enabling a 'geography of compute' where AI workloads can be distributed across the globe without performance loss.