Skip to content

Ollama

Ollama

一款在本地运行大语言模型的工具，简化模型部署和使用。

Official Site
01

概述

Ollama 是一款开源的本地 LLM 运行工具，让在个人电脑上运行大语言模型变得简单。其核心理念是简化模型部署：一条命令即可下载并运行模型。Ollama 支持 Llama 3、Mistral、Phi-3、Gemma、Qwen 等开源模型。它提供 REST API 和命令行界面，便于集成到应用中。Ollama 自动处理 GPU 检测、内存管理、模型优化等复杂任务。支持 macOS、Linux 和 Windows。

02

安装

Ollama 支持 macOS、Linux 和 Windows。安装过程简单：下载安装包并运行。安装后环境会自动配置，无需额外设置。

bash
# macOS
# Download the installer from https://ollama.com/download

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download the installer from https://ollama.com/download

# Docker
docker pull ollama/ollama
docker run -d -p 11434:11434 ollama/ollama
03

运行模型

运行模型非常简单，使用 ollama run 命令。首次运行会自动下载模型。下载后模型存储在本地，后续运行无需重新下载。支持交互式对话和一次性查询。

bash
# Run a model (downloads on first run)
ollama run llama3.1

# Interactive conversation
>>> Hello, please introduce yourself

# One-shot query
ollama run llama3.1 "Explain quantum computing"

# List installed models
ollama list

# Delete a model
ollama rm llama3.1
04

模型库

Ollama 提供丰富的模型库，包括 Llama、Mistral、Phi、Gemma、Qwen、DeepSeek 等。模型有不同大小（7B、13B、70B 等），可根据硬件选择。还提供嵌入模型、视觉模型等。

bash
# Common models
ollama run llama3.1        # Meta Llama 3.1
ollama run mistral         # Mistral 7B
ollama run phi3            # Microsoft Phi-3
ollama run gemma2          # Google Gemma 2
ollama run qwen2.5         # Alibaba Qwen 2.5
ollama run deepseek-r1     # DeepSeek R1
ollama run llama3.1:70b    # 70B version (requires more memory)
ollama run nomic-embed-text # Embedding model
05

自定义模型

Ollama 支持自定义模型。通过 Modelfile 定义模型，可以导入 GGUF 格式模型、设置参数、自定义系统提示词等。支持从 Hugging Face 导入模型。

bash
# Modelfile example
FROM llama3.1

# Set parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096

# System prompt
SYSTEM """
You are a professional programming assistant skilled in Python and JavaScript.
"""

# Create the model
ollama create my-assistant -f Modelfile

# Run the custom model
ollama run my-assistant
06

API

Ollama 默认在 11434 端口提供 REST API。API 兼容 OpenAI 格式，可轻松集成到应用中。支持生成、聊天、嵌入等端点。提供流式响应。

bash
# Generate endpoint
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Why is the sky blue?"
}'

# Chat endpoint
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {"role": "user", "content": "Hello"}
  ]
}'

# Python SDK
import ollama
response = ollama.chat(model='llama3.1', messages=[
  {'role': 'user', 'content': 'Hello'}
])
07

微调

Ollama 本身不提供微调能力，但可以运行微调后的模型。你可以使用 LoRA、QLoRA 等技术微调模型，然后转换为 GGUF 格式导入 Ollama。这适用于需要特定领域知识的场景。

bash
# Fine-tuning workflow
# 1. Fine-tune the model with transformers
# 2. Convert to GGUF format
python convert.py model.pth --outtype f16

# 3. Quantize (optional)
./quantize model.gguf model-q4.gguf q4_0

# 4. Create a Modelfile
FROM ./model-q4.gguf

# 5. Import into Ollama
ollama create fine-tuned-model -f Modelfile
08

GPU 配置

Ollama 自动检测并使用 GPU（macOS Metal、NVIDIA CUDA、AMD ROCm）。GPU 显著提升推理速度。可以通过环境变量强制使用 CPU 或特定 GPU。支持多 GPU 配置。

bash
# Check GPU usage
ollama ps  # Show the GPUs in use

# Force CPU usage
OLLAMA_NO_GPU=1 ollama serve

# Specify GPU (Linux)
CUDA_VISIBLE_DEVICES=0,1 ollama serve

# macOS automatically uses Metal
# No additional configuration required
09

Docker

Ollama 提供官方 Docker 镜像，适合服务器部署。需要配置 GPU 支持和卷映射。Docker 部署使管理和扩展更便捷。

bash
# Run with Docker
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# GPU support (NVIDIA)
docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
10

集成

Ollama 可集成到各种应用中。它支持 LangChain、LlamaIndex、Open WebUI 等框架。通过 OpenAI 兼容 API，可在大多数应用中替代 OpenAI。适合构建本地 AI 应用。

bash
# LangChain integration
from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="llama3.1")
response = llm.invoke("Hello")

# OpenAI-compatible API
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello"}]
)

# Open WebUI (Docker)
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main
11

Configuration

Ollama is configured through environment variables and the Modelfile. OLLAMA_HOST changes the listen address, OLLAMA_ORIGINS controls CORS, and OLLAMA_NO_GPU/CUDA_VISIBLE_DEVICES manage GPU selection. Models are stored under ~/.ollama/models by default. A Modelfile lets you create custom models by layering a system prompt, parameters, and a base model. The server exposes an OpenAI-compatible API at /v1, so most OpenAI SDKs work without changes.

bash
# Environment variables
export OLLAMA_HOST=0.0.0.0:11434   # listen on all interfaces
export OLLAMA_ORIGINS="*"          # allow CORS from any origin
export OLLAMA_NO_GPU=1             # force CPU-only
export CUDA_VISIBLE_DEVICES=0,1    # pick specific GPUs

# Modelfile - build a custom model
FROM llama3.1
SYSTEM "You are a senior Python engineer. Be concise."
PARAMETER temperature 0.2
PARAMETER num_ctx 8192

# Build and run
ollama create my-assistant -f Modelfile
ollama run my-assistant

# Config file locations
# Linux/macOS: ~/.ollama/
# Windows: C:\Users\<you>\.ollama\

Models can be many GB—point OLLAMA_MODELS at a disk with enough space if your home partition is small.

12

FAQ

Common questions cover hardware requirements, model storage location, custom models, OpenAI API compatibility, and multi-user access. Ollama runs on CPU but is much faster on GPU; memory needs scale with model size (a 7B model needs ~8 GB, 70B needs ~40 GB). Models live under ~/.ollama/models. The /v1 endpoint is OpenAI-compatible, so existing SDKs work. For multi-user access, run Ollama behind a reverse proxy and set OLLAMA_ORIGINS.

bash
Q: Do I need a GPU?
A: No, Ollama runs on CPU, but inference is much faster on a GPU. A 7B
   model needs ~8 GB RAM/VRAM; a 70B model needs ~40 GB.

Q: Where are models stored?
A: Under ~/.ollama/models by default. Change it with the OLLAMA_MODELS
   environment variable if you need more disk space.

Q: Can I create a custom model?
A: Yes—write a Modelfile (FROM base-model + SYSTEM + PARAMETER) and run
   'ollama create my-model -f Modelfile'.

Q: Is the API OpenAI-compatible?
A: Yes, Ollama exposes /v1/chat/completions; point any OpenAI SDK at
   http://localhost:11434/v1 with api_key="ollama".

Q: How do I serve multiple users?
A: Run Ollama on a server, set OLLAMA_HOST=0.0.0.0 and OLLAMA_ORIGINS,
   and put it behind a reverse proxy (nginx/Caddy) for TLS.

Use 'ollama list' to see installed models and 'ollama rm <model>' to free disk space.

Ready to try Ollama?

Visit the official site for the latest version and full documentation.

Visit Ollama
All AI guides