Log Aggregation + Alerting

Trigger alerts from logs and metrics before users report problems.

alerts

Detailed Description

Alerts should represent user impact or urgent operational risk.

Too many low-signal alerts train teams to ignore the monitoring system.

Visual Diagram

Alert Rules (examples)
  error_rate > 5%      → PagerDuty alert
  p99_latency > 2000ms → Slack warning
  pod restarts > 3     → immediate page
  disk usage > 85%     → warning email

  SLO / Error Budget
  SLO: 99.9% uptime = 43.8 min/month
  Budget: consumed 40 min this month
  → freeze deployments, focus on stability

Tradeoffs

Pros

Proactive detection and accountability

Cons

Alert fatigue if thresholds are noisy

Examples: PagerDuty, OpsGenie, Grafana Alerts, CloudWatch Alarms