Tag
#vLLM
2 articles tagged vLLM.
Continuous batching, revisited
Inference & ServingThree years after the original paper, what does state-of-the-art serving actually look like? A field report from a team running 12B tokens a day.
Notes on KV cache paging at scale
Inference & ServingPagedAttention is a good idea poorly understood. A primer, plus the second-order effects you only see at 10,000 concurrent requests.