Service
Observability — Metrics, Traces & Logs
Make the three signals work together, so an incident ends with a cause rather than a guess — and so the platform bill stops growing faster than traffic.
Most teams are not short of telemetry. They are short of the specific signal that says whether users are affected, and they are drowning in alerts nobody acts on any more. The three signals usually exist in three disconnected tools, so an incident becomes a tab-switching exercise rather than an investigation.
The stack question, answered from having run them
There is no universally correct backend, and the honest answer depends on your workload shape:
- Prometheus — pull-based, excellent for Kubernetes and service-level metrics, weaker when you need long retention or high-cardinality dimensions
- InfluxDB with Telegraf — push-based, a better fit for host, network and device telemetry, and for the collection points where a scrape target does not make sense
- Datadog — fastest to value and genuinely good, but the pricing model punishes cardinality, custom metrics and log volume in ways that are hard to see until renewal
- ELK — strong for log search and analysis, and the operational cost of running it yourself is real and should be counted honestly against a hosted alternative
Most organisations end up with more than one of these. The work is usually not replacing them but deciding what each is for and stopping the overlap.
What we look at
- Which alerts fired last quarter, and which produced an action
- Whether service level objectives exist, and whether anyone uses them during an incident
- Cardinality and retention — normally where the bill actually comes from
- Whether traces cross service boundaries or stop at the first hop
- Log volume against log value, and what is being indexed that nobody queries
What you get
An alert set where every alert has an owner and a documented response. Dashboards organised around failure modes rather than around whatever metrics happened to be available. Tracing that reaches the slow dependency instead of stopping at the edge. And a retention and cardinality policy set deliberately, so the platform bill reflects a decision rather than an accident.