Skip to content

Monitoring 数据 Lost

错误信息

                错误: Prometheus missing metrics for 最后 30 minutes
              

快速摘要

Prometheus or another monitoring 系统有 gaps in collected metrics, 通常 due 到 scrape failures or storage issues.

为什么发生

  • Prometheus cannot scrape 目标 (application 下, network issues)

  • Prometheus storage 是完整 or 有 retention issues

  • application's /metrics 端点是 不工作 or 是慢

  • Prometheus 它自己是 overloaded and dropping samples

最小示例

✗ 错误代码
# Prometheus cannot scrape my-app
# Target is down at http://my-app:8080/metrics
✓ 修复代码
# Check if my-app is running
kubectl get pods -l app=my-app
# Restart if needed
kubectl rollout restart deployment/my-app
# Verify metrics endpoint
kubectl exec -it <prometheus-pod> -- curl http://my-app:8080/metrics

Prometheus cannot scrape application 因为它 是下. fix restarts application and 验证 metrics 端点.

如何诊断

  • 检查 Prometheus 目标: visit http://prometheus:9090/targets

  • 检查 Prometheus logs for 错误: kubectl logs <prometheus-pod>

  • 验证 application's /metrics 端点是 responding

  • 检查 Prometheus storage usage and retention 设置

如何修复

  • Fix scrape 目标 (重启 application, fix network)

  • Increase Prometheus storage or adjust retention period

  • Fix application's metrics 端点

  • Scale Prometheus or use remote 编写到 offload storage

如何预防

  • Monitor Prometheus health and alert on scrape failures

  • 设置长-term storage (Thanos, Cortex) for metrics retention

  • Use ServiceMonitors 到 automatically manage scrape 目标

相关资源

← 返回语言