SRE and reliability

Not every alert matters equally, and impact on users should set the priority. So we track latency and availability against service level objectives (SLOs), and alerting and response workflows follow the same objectives. Error budgets give a shared basis for discussing feature delivery and reliability work.

Scope of service

• SLIs and SLOs derived from critical user journeys
• An error budget policy that ties release and freeze decisions to data
• Alert tuning that cuts noise and surfaces actionable signals
• Capacity planning and cost-aware scaling
• Controlled fault injection (chaos engineering) and GameDay exercises
• Blameless post-mortems with tracked action items
• Model serving indicators: time to first token, output token rate and KV cache pressure

How we set an SLO

1. 1

   User journey

   Together with the product team, we identify critical user flows and the latency and error signals that affect them.

2. 2

   Measurement point and target

   We choose where to measure (client, gateway or service) and set SLO targets around business needs and current baseline performance.

3. 3

   Error budget and alerting

   We tie dashboards and alerts to error budget burn and run agreed release and reliability priorities when budgets are exhausted.

Tooling

Metrics and alerting

PrometheusAlertmanagerGrafana

Logs

LokiOpenSearch

Tracing

OpenTelemetryJaeger

Controlled fault injection

Chaos MeshLitmusChaos

Model serving observability

PrometheusGrafana dashboards

Synthetic and multi-region availability monitoring

UptimeWarden

On-call

PagerDuty