Prometheus and Alertmanager for alerting on OpenTelemetry

OpenTelemetry doesn't do alerting. By design: it collects and ships logs, metrics and traces, and stops there. Alerts are left to an observability backend. For alerts in an OpenTelemetry stack, I still like the classic pairing of Prometheus + Alertmanager.

Compatibility today is very good: the joint OTel-Prometheus SIG has been working on it for years, and the OpenTelemetry collector sends metrics to Prometheus via remote write. Prometheus 3.0 also has a native OTLP receiver (I haven't tested that one yet, though).

The architecture is simple. The collector sends everything to ClickStack for analysis and debugging and sends metrics to Prometheus, which I use only as an alerting rule engine with short retention. Alertmanager receives the alerts and handles grouping, routing, inhibition and silences.

What I like: it's all text. Rules in YAML and PromQL, routing in YAML. One hundred percent infrastructure as code, and that makes it perfect for agents: Claude writes the rule, validates it with promtool check and verifies it against real data. The Prometheus and Alertmanager web UIs are bare-bones, but I hardly ever use them anyway.

Rule libraries like awesome-prometheus-alerts can't be copied as they are: they expect the semantics of node_exporter and kube-state-metrics, not OTel's. But they are a guide to what is worth monitoring, with thresholds and durations. Translating them into OTel-native rules is fairly mechanical work, and an agent does it well.

The limit: Prometheus only sees metrics. To alert on logs, you need to derive synthetic metrics in the collector.