Serve models to many users fast and cheaply: batching, caching and the right hardware.
- RequestsMany users at once.
- BatchingGroup requests on the GPU (continuous batching).
- KV cacheReuse attention work between tokens (paged attention).
- MeasureTrack time to first token, tokens/sec and cost.