Monitoring
Track 系统 health 使用 metrics, dashboards, and tools 如 Prometheus and Grafana.
你将学到
- 为什么 monitoring 是关键 for DevOps
- Metrics, logs, and traces — three pillars
- Tools: Prometheus, Grafana, and Datadog
- 构建 effective dashboards
概念讲解
Why Monitor?
Monitoring is the eyes and ears of your production system. Without it, you are flying blind — you won't know about problems until users complain, and you'll have no data to diagnose them. Good monitoring tells you what is happening right now, alerts you when something goes wrong, and provides historical data to understand trends and plan capacity. In DevOps, monitoring is not optional — it is a core practice that enables fast, safe deployments.
The Three Pillars
Observability rests on three pillars. Metrics are numeric measurements sampled over time — CPU usage, request latency, error rate. Logs are timestamped records of discrete events — a user logged in, a payment failed, a connection was refused. Traces follow a single request through all the services it touches, showing you where time is spent and where errors occur. Together, they give you a complete picture of system behavior.
Tools
Prometheus is an open-source metrics database that scrapes metrics from your applications. Grafana is a visualization tool that creates dashboards from Prometheus and many other data sources. Datadog is a commercial all-in-one platform. Jaeger and Zipkin handle distributed tracing. The ELK stack (Elasticsearch, Logstash, Kibana) is popular for log management. Most teams use a combination of these tools.
Effective Dashboards
A good dashboard answers a specific question: "是 checkout 工作?" or "是我们 meeting 我们的 latency SLO?" Start with high-level service health, then drill down to details. Use golden signals: latency, traffic, errors, and saturation. Avoid dashboard overload — too many panels make it hard to find what matters. Color-code consistently: green for healthy, red for problems. Make dashboards that a tired on-call engineer can understand at 3 AM.
Synthetic Monitoring
Beyond real-user metrics, synthetic monitoring runs scripted checks against your service from external locations. A synthetic check simulates a user journey — log in, add to cart, checkout — and alerts if any step fails. This catches issues before real users are affected, even during low-traffic periods. Combine synthetic checks with real-user monitoring for comprehensive coverage: synthetics for early detection, real-user data for understanding actual impact and performance across different geographies and device types.
示例代码
# Prometheus scrape configuration
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'myapp'
static_configs:
- targets: ['app:9090']
metrics_path: /metrics
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
# Key metrics to track:
# http_requests_total - request count by route/code
# http_request_duration_seconds - latency histogram
# process_resident_memory_bytes - memory usage
# node_cpu_seconds_total - CPU usage
Prometheus scrapes metrics 从你的 applications at regular intervals. 配置定义哪个目标到 scrape and 如何经常. Grafana 然后 visualizes 这个数据.
动手试一试
- HTTP 状态 Codes
引用 for monitoring HTTP 错误速率.
- Cron Parser
Parse monitoring schedule 表达式.
- Timestamp Converter
Convert metric timestamps for analysis.
常见错误
错误写法
Monitoring 一切但 alerting on 无
正确做法
Collecting metrics 是 useless 如果 nobody looks at 它们. 定义 alerting 规则 for metrics 那个 matter — 错误速率, latency, saturation. dashboard nobody 检查是 wasted effort.
错误写法
Alerting on symptoms 相反的 causes
正确做法
Alert on 用户-facing symptoms (错误速率, latency) 而不是 underlying causes (CPU at 80%). Symptom-based alerts 是更多 stable and directly indicate 用户 impact.
练习 Monitoring
2 个练习
查阅不认识的术语
新手友好的术语表用大白话解释编程术语——变量、函数、DOM、Promise 等等。
动手做项目
通过引导式项目应用所学,含起始代码与解决方案。
解读错误信息
用大白话解释常见错误——什么意思、为什么发生、如何修复。