Skip to content

vLLM

UC Berkeley

一个高性能的 LLM 推理和服务引擎，提供极致的推理速度。

Official Site
01

概述

vLLM 是由 UC Berkeley 开发的高性能 LLM 推理引擎。其核心创新是 PagedAttention，一种受操作系统虚拟内存启发的注意力机制，显著减少 KV cache 内存浪费。vLLM 的吞吐量比 Hugging Face Transformers 高 14-24 倍，比 Text Generation Inference（TGI）高 2-4 倍。它支持连续批处理、张量并行、流式输出等。vLLM 是生产环境部署 LLM 的首选方案。

02

安装

vLLM 需要 NVIDIA GPU（CUDA 11.8+）和 Python 3.8+。推荐通过 pip 安装。也支持 Docker 部署。安装过程中依赖会自动处理。

bash
# pip install
pip install vllm

# Or install a specific version
pip install vllm==0.6.0

# Docker
docker pull vllm/vllm-openai:latest

# Install from source
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .
03

快速开始

vLLM 提供简单的 API 来启动服务。一条命令即可启动 OpenAI 兼容的 API 服务器。也支持用于离线推理的 Python API。

bash
# Start the API server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

# Python API offline inference
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(["Hello, please introduce yourself"])
04

模型服务

vLLM 提供高性能的模型服务。支持加载 Hugging Face 模型和本地模型。可以配置 GPU 数量、张量并行、量化等。服务兼容 OpenAI API，可无缝替代 OpenAI。

bash
# Start the service (detailed configuration)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --port 8000

# Test the API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
05

OpenAI API

vLLM 提供 OpenAI 兼容 API，可无缝替代 OpenAI。支持 Chat Completions、Completions、Embeddings 等端点。现有应用只需更改 API 地址即可使用 vLLM。

bash
# OpenAI SDK integration
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm"  # vLLM does not require a real key
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")
06

量化

vLLM 支持多种量化方案，减少模型内存占用并提升推理速度。支持 AWQ、GPTQ、SqueezeLLM、FP8 等量化格式。量化可以显著降低硬件要求，让大模型在更小的 GPU 上运行。

bash
# Use an AWQ quantized model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-13B-AWQ \
  --quantization awq

# Use a GPTQ quantized model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-13B-GPTQ \
  --quantization gptq

# FP8 quantization (Hopper GPU)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization fp8
07

分布式服务

vLLM 支持分布式推理，让大模型在多个 GPU 上运行。它支持 Tensor Parallelism 和 Pipeline Parallelism。这使得 70B、405B 等大模型能在多 GPU 系统上运行。

bash
# Tensor parallelism (single machine, multiple GPUs)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

# Multi-node distributed
# Node 0
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

# Node 1
VLLM_HOST_IP=192.168.1.2 \
NCCL_SOCKET_IFNAME=eth0 \
python -m vllm.entrypoints.openai.api_server ...
08

性能

vLLM 的性能优势来自 PagedAttention、连续批处理和优化的 CUDA 内核。吞吐量比 Transformers 高 14-24 倍。支持高并发请求且延迟低。可通过参数调优进一步提升性能。

bash
# Performance optimization configuration
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --swap-space 16  # GB, CPU swap space
09

集成

vLLM 可集成到各种应用中。它支持 LangChain、LlamaIndex 等框架。通过 OpenAI 兼容 API，可在大多数应用中替代 OpenAI。适合构建高性能 AI 应用。

bash
# LangChain integration
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm",
    model="meta-llama/Llama-3.1-8B-Instruct"
)

# LlamaIndex integration
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
    model="meta-llama/Llama-3.1-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="vllm"
)
10

Configuration

vLLM is configured through CLI flags passed to vllm.entrypoints.openai.api_server and environment variables. The most important flags are --model (model id or path), --tensor-parallel-size (GPUs per node), --gpu-memory-utilization (KV cache budget), --max-model-len (context window), and --quantization (awq/gptq/fp8). For persistent settings, write flags into a shell script or systemd unit. The server speaks the OpenAI API, so clients just need base_url and a dummy api_key.

bash
# Core serving flags
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --quantization fp8 \
  --port 8000

# Environment variables
export VLLM_NO_USAGE_STATS=1        # disable usage collection
export HF_TOKEN=hf_xxx              # for gated models
export VLLM_ATTENTION_BACKEND=FLASHINFER  # pick an attention backend

# Run as a background service (systemd unit)
# /etc/systemd/system/vllm.service
# ExecStart=/usr/bin/python -m vllm.entrypoints.openai.api_server \
#   --model meta-llama/Llama-3.1-8B-Instruct

gpu-memory-utilization is the fraction reserved for the KV cache—leave headroom for the model weights, or startup will OOM.

11

FAQ

Common questions cover GPU/memory requirements, quantization, OpenAI compatibility, distributed serving, and throughput tuning. vLLM needs an NVIDIA GPU with CUDA 11.8+; a 7B model fits on a single 16 GB GPU, while 70B needs 4x80 GB or quantization. The /v1 endpoint is OpenAI-compatible, so most SDKs work after changing base_url. Throughput scales with --max-num-seqs and continuous batching, so raise it for high-concurrency workloads.

bash
Q: What hardware does vLLM need?
A: An NVIDIA GPU with CUDA 11.8+. A 7B/8B model fits on a 16 GB GPU;
   a 70B model needs ~4x80 GB or quantization (AWQ/GPTQ/FP8).

Q: Is the API OpenAI-compatible?
A: Yes—point any OpenAI SDK at http://localhost:8000/v1 with a dummy
   api_key; chat/completions, embeddings, and completions all work.

Q: How do I cut memory usage?
A: Use a quantized model (--quantization awq|gptq|fp8), lower
   --gpu-memory-utilization, shorten --max-model-len, or pick a smaller model.

Q: How do I serve across multiple GPUs?
A: Use --tensor-parallel-size N on one machine, or combine
   --tensor-parallel-size with --pipeline-parallel-size for multi-node.

Q: Why is throughput lower than expected?
A: Raise --max-num-seqs and --max-num-batched-tokens, enable
   --enable-chunked-prefill, and keep --gpu-memory-utilization high.

Monitor nvidia-smi while serving—if GPU memory is not saturated, you can usually raise --max-num-seqs for higher throughput.

Ready to try vLLM?

Visit the official site for the latest version and full documentation.

Visit vLLM
All AI guides