Monitoring Data Lost
Error message
Error: Prometheus missing metrics for the last 30 minutes
Quick summary
Prometheus or another monitoring system has gaps in collected metrics, usually due to scrape failures or storage issues.
Why this happens
Prometheus cannot scrape targets (application down, network issues)
The Prometheus storage is full or has retention issues
The application's /metrics endpoint is not working or is slow
Prometheus itself is overloaded and dropping samples
Minimal example
# Prometheus cannot scrape my-app
# Target is down at http://my-app:8080/metrics# Check if my-app is running
kubectl get pods -l app=my-app
# Restart if needed
kubectl rollout restart deployment/my-app
# Verify metrics endpoint
kubectl exec -it <prometheus-pod> -- curl http://my-app:8080/metricsPrometheus cannot scrape the application because it is down. The fix restarts the application and verifies the metrics endpoint.
How to diagnose
Check Prometheus targets: visit
http://prometheus:9090/targetsCheck Prometheus logs for errors:
kubectl logs <prometheus-pod>Verify the application's /metrics endpoint is responding
Check Prometheus storage usage and retention settings
How to fix
Fix the scrape target (restart the application, fix network)
Increase Prometheus storage or adjust retention period
Fix the application's metrics endpoint
Scale Prometheus or use remote write to offload storage
How to prevent
Monitor Prometheus health and alert on scrape failures
Set up long-term storage (Thanos, Cortex) for metrics retention
Use ServiceMonitors to automatically manage scrape targets