High-throughput serving engine for open-weight language models
vLLM is an open-source library for serving large language models efficiently. Its core technique, PagedAttention, manages the key-value cache in blocks so memory is not wasted and many requests can be batched together, which raises throughput well above naive implementations. You start a server with a model name and get an OpenAI-compatible HTTP endpoint. Teams use it to self-host models behind their own applications on their own GPUs. The differentiator is throughput per GPU: it is the usual choice when serving open models at volume.
Last updated: 2026-09-20. This site only provides an index; for exact features, pricing, and licensing, see the official website.
self-hosted serving, high-throughput inference, batch generation, private model endpoints
If you're comparing similar products, check the alternatives below, or browse all tools in the AI Models & Platforms category.
It stores the model's key-value cache in small blocks that can be shared and reused instead of one large contiguous buffer, which cuts wasted memory and lets far more requests run at once.
Not for small and mid-size models. Models that do not fit in one GPU's memory can be split across several with tensor parallelism, which vLLM supports through a configuration flag.
vLLM targets GPU servers and high concurrent throughput, while llama.cpp targets broad hardware including CPUs and small devices. Choose based on where and how heavily you need to serve.
Developer console and API keys for the Claude model family
Node-based local interface for running image and video diffusion models
Browser playground for prompting and prototyping with Gemini models
Hosted Jupyter notebooks with optional free GPU and TPU runtimes