Cloud observability: instrument everything, keep almost none of it

Most observability programmes we are called into have the same two symptoms at once: a monitoring bill that grew faster than traffic, and an on-call engineer who still cannot answer a simple question at 3 a.m. Those are the same problem. Both come from collecting telemetry without deciding, in advance, which questions it exists to answer.

This is the architecture we start from, the trade-offs behind each decision, and what the tools actually cost at list price.

The only three questions that matter

Every dashboard, alert, and retention policy should trace back to one of three questions. If a signal does not help answer one of them, it is costing money for nothing.

Note what is absent. Nobody has ever shortened an outage by looking at a wall of forty dashboards, and the second question is the one most teams cannot answer at all — deploy events live in the CI tool, flags live in the flag service, and neither is overlaid on the graph that just went red. Getting a deploy marker onto your SLI chart is an afternoon of work and it changes the shape of every incident that follows.

Instrument once, store selectively

The single most consequential decision here is to separate instrumentation from storage. Instrumentation is a multi-year commitment spread across every service you own. Storage is a monthly invoice. Teams that conflate the two end up with a vendor’s agent compiled into three hundred services, and no practical way to move.

So: instrument against OpenTelemetry, export to a Collector you run, and let the Collector decide where data goes. Changing backend then means editing one config file rather than redeploying an estate.

# The Collector is the seam. Applications know nothing about the backend.
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  # Bound the damage a traffic spike can do to the Collector itself.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  # Attach the identity every downstream query will want to filter on.
  resource:
    attributes:
      - { key: deployment.environment, value: production, action: upsert }
      - { key: service.version, from_attribute: git.sha, action: insert }

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/primary:
    endpoint: primary-backend:4317
  # Running two exporters for a fortnight is how you evaluate a vendor on your own
  # traffic instead of on their demo data.
  otlp/candidate:
    endpoint: candidate-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/primary, otlp/candidate]

That dual-export line is worth more than it looks. It is how you run a real bake-off — same traffic, same week, two bills — rather than arguing from vendor benchmarks. It is also the cheapest negotiating position available to you at renewal, because it is credible.

Check signal maturity before you commit

OpenTelemetry is not uniformly finished, and the gaps are not where people assume. Traces and metrics are stable in most major SDKs. Logs are the least mature signal — stability varies considerably by language, and several widely-used SDKs are still pre-stable. Profiling is newer still, with only early implementations.

Read the OpenTelemetry status page for your specific languages before you plan a migration around it. The practical consequence: it is entirely reasonable to adopt OTel for traces and metrics now and leave logs on your existing shipper for another cycle. A hybrid is not a failure of nerve, and pretending otherwise costs a quarter.

Semantic conventions, or you have a swamp

Telemetry without agreed attribute names is a data swamp with a query language bolted on. Two services that call the same field user_id and userId cannot be correlated, and nobody discovers this until an incident.

Adopt the OpenTelemetry semantic conventions and enforce them at the Collector, not in code review:

processors:
  transform/normalise:
    error_mode: ignore
    trace_statements:
      # Fix the drift centrally rather than chasing forty repositories.
      - set(attributes["user.id"], attributes["userId"]) where attributes["userId"] != nil
      - delete_key(attributes, "userId")

      # Strip anything that must never reach a vendor. Doing this at the edge means
      # a leak is a config bug you can fix in minutes, not a support ticket asking a
      # third party to purge their indexes.
      - delete_key(attributes, "http.request.header.authorization")
      - delete_key(attributes, "db.statement.parameters")

  attributes/redact:
    actions:
      - { key: user.email, action: hash }

Redacting at the Collector rather than in each service is the difference between a personal-data incident being a config change and being a conversation with your vendor’s support team about deleting their indexes.

Traces first, metrics derived, logs as evidence

The conventional order — logs, then metrics, then maybe traces one day — is backwards, and it is expensive. Traces are the only signal that carries the causal structure of a request. Once you have them, most of what you were paying to log becomes redundant.

1. Traces are primary.One well-instrumented trace answers “where is the time going” directly, with no correlation step and no guessing which service to look at next.
2. Derive metrics from spans.The Collector’s span-metrics connector generates request rate, error rate, and latency histograms from trace data you already send. One instrumentation effort, both signals, and the metrics are guaranteed consistent with the traces because they came from them.
3. Logs are evidence, not the interface. Emit structured logs carrying trace_id so they attach to the span that needs explaining. Searching logs to reconstruct a request is the expensive alternative to having traced it.
4. Profiles, when you have a CPU or allocation question. Narrow, high-value, and only worth wiring once the first three are working.

connectors:
  # Rate, errors, duration for every service — from spans you are already paying to
  # send. No extra instrumentation, and no drift between the graph and the trace.
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s]
    dimensions:
      - { name: http.route }
      - { name: http.response.status_code }
      - { name: deployment.environment }
    # NOT http.target, NOT user.id. See the next section for why.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/normalise, batch]
      exporters: [spanmetrics, otlp/primary]
    metrics/from-spans:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]

Note http.route rather than http.target. The route is the template (/orders/:id); the target is the actual URL (/orders/8814-aa). One is bounded at a few hundred values. The other is unbounded, and choosing it is the most common way to accidentally spend five figures a month.

Cardinality decides the bill

This is the section that determines what you pay, and it is almost always the thing nobody owns. A time series is created for every unique combination of label values. Adding one unbounded label does not increase cost incrementally — it multiplies it.

A single metric with a customer ID label, across 10,000 customers, 20 routes, 5 status codes and 3 environments, is three million series. Teams routinely discover this when their Prometheus pod starts requesting tens of gigabytes of memory. Reported production cases include instances climbing past 55 GB purely from label explosion.

Guardrails that fail loudly

Prometheus will enforce limits at scrape time if you ask it to, and failing a scrape is much better than silently absorbing a cardinality bomb:

scrape_configs:
  - job_name: services
    # Fail the scrape rather than ingest an explosion. A broken scrape pages
    # someone; a cardinality bomb just quietly arrives on the invoice six weeks
    # later, by which point nobody remembers which deploy caused it.
    sample_limit: 5000
    label_limit: 30
    label_name_length_limit: 128
    label_value_length_limit: 256
    target_limit: 2000

Complement that at the Collector, where you can drop dimensions before they are ever billed:

processors:
  # Aggregate away the dimensions nobody queries. Anything that only appears in a
  # dashboard nobody has opened in ninety days is a candidate.
  metricstransform/prune:
    transforms:
      - include: http.server.duration
        action: update
        operations:
          - action: aggregate_labels
            label_set: [http.route, http.response.status_code, service.name]
            aggregation_type: sum

  # Belt and braces: refuse known-dangerous labels outright.
  attributes/drop-unbounded:
    actions:
      - { key: http.target, action: delete }
      - { key: user.id, action: delete }
      - { key: request.id, action: delete }
      - { key: session.id, action: delete }

Those identifiers still belong on spans, where they cost per-byte rather than per-series and are exactly what you need for debugging. The rule is simple and worth writing on a wall: high-cardinality data belongs on traces, never on metrics.

The Collector contrib repository also has a cardinality-guardian processor in development, aimed at catching explosions in flight. Worth tracking, not yet worth depending on.

Scrape interval is a pricing decision

On any platform that bills per sample rather than per series, the scrape interval multiplies directly into the invoice. 100,000 series at a 15-second interval is roughly 17.3 billion samples a month. At 60 seconds it is 4.3 billion — a quarter of the cost, for data that is almost always used to draw a graph nobody reads at sub-minute resolution.

Scrape the handful of series that feed alerts at 15 seconds. Scrape the rest at 60. This one change has paid for entire engagements.

Tail sampling: keep the interesting two percent

Head sampling — deciding at the start of a request — is cheap and throws away exactly the traces you needed, because a request is not yet known to be slow or failing when it begins. Tail sampling decides after the trace completes, which means you can keep every error and every slow request while discarding the overwhelming majority of successful, fast, boring ones.

In practice this removes 90–98% of trace volume with no loss of debugging ability, because the traces you drop are the ones nobody would ever open.

processors:
  tail_sampling:
    # Hold spans this long waiting for the trace to finish. Must exceed your p99
    # request duration or you will make decisions on partial traces.
    decision_wait: 15s
    num_traces: 100000
    expected_new_traces_per_sec: 2000

    policies:
      # Never drop a failure.
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      # Never drop something slow.
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }

      # Never drop the customers with a contractual SLA.
      - name: tier-one-tenants
        type: string_attribute
        string_attribute:
          key: tenant.tier
          values: [enterprise]

      # Keep a small, unbiased baseline so you can still reason about normal.
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

      # A backstop so one pathological tenant cannot dominate the bill.
      - name: ceiling
        type: rate_limiting
        rate_limiting: { spans_per_second: 20000 }

The caveat that breaks naive deployments

Tail sampling requires that every span of a trace reaches the same Collector instance. Put a plain round-robin load balancer in front of a Collector fleet and you will make sampling decisions on fragments of traces, then spend a week wondering why traces are incomplete.

The fix is a two-tier Collector topology: a stateless first tier that routes by trace ID, and a second tier that does the sampling.

# Tier 1: routing only. Scales horizontally, holds no state.
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: { insecure: true }
    resolver:
      # Headless service so every sampler pod is individually addressable.
      k8s:
        service: otel-sampler.observability.svc.cluster.local

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loadbalancing]

# Tier 2 pods run the tail_sampling processor above. Scale this tier on memory,
# since decision_wait x throughput is what it has to hold.

Retention tiers

Almost nobody queries telemetry older than a week, and almost everybody pays to keep it hot for a month. Tier deliberately:

The mistake is treating audit retention as an observability requirement. If a regulator needs seven years of access logs, that is a cheap append-only bucket, not seven years in your APM vendor at indexed rates.

Alert on burn rate, not thresholds

Threshold alerts on CPU, memory, and raw error counts are why on-call rotations burn people out: they fire when a machine is busy rather than when a customer is affected. Define an objective, then alert on how fast you are consuming its error budget.

Google’s SRE workbook gives the parameters for a 99.9% objective, and they are worth using as published rather than invented locally:

The short window is one twelfth of the long one, and both must breach for the alert to fire. That second condition is what makes the alert stop within minutes of the problem ending, instead of holding someone awake for the remainder of a six-hour window.

groups:
  - name: slo-checkout-availability
    rules:
      # Pre-compute the ratio at several windows. Recording rules keep the alert
      # expressions readable, which matters when someone reads them under pressure.
      - record: slo:checkout_error_ratio:5m
        expr: |
          sum(rate(http_server_duration_count{service="checkout",http_response_status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_duration_count{service="checkout"}[5m]))

      - record: slo:checkout_error_ratio:1h
        expr: |
          sum(rate(http_server_duration_count{service="checkout",http_response_status_code=~"5.."}[1h]))
          /
          sum(rate(http_server_duration_count{service="checkout"}[1h]))

      # 14.4 x the 0.1% budget = 1.44% error rate sustained over both windows.
      - alert: CheckoutErrorBudgetBurningFast
        expr: |
          slo:checkout_error_ratio:1h  > (14.4 * 0.001)
          and
          slo:checkout_error_ratio:5m  > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: Checkout is consuming its monthly error budget in under three days
          runbook: https://runbooks.internal/checkout-availability

Three alerts per service, derived from one objective, replace most of a dashboard wall. They also give you a defensible answer to “should we ship on Friday”, which is a conversation worth having with data rather than instinct.

The tool landscape

All list prices below were checked in July 2026 and exclude negotiated discounts, which are substantial above roughly $100k annually. Treat them as a starting point for modelling, not a quote.

Where the money actually goes

Comparing headline prices is less useful than comparing the same workload. Take one terabyte of logs a month at roughly 1 kB an event — about a billion events:

The interesting number is not the cheapest column. It is the eighteen-fold gap between ingesting a terabyte and indexing it on the same platform. That gap is a decision about which log streams anyone actually searches — and it is available to you on day one, before any migration.

Metrics show the same pattern more sharply. 100,000 custom metrics on CloudWatch is roughly $12,000 a month at list, because the first 10,000 bill at $0.30 each. The equivalent series on Grafana Cloud is around $650. Same telemetry, and the difference is entirely a pricing-model mismatch rather than anything technical.

Build or buy

Self-hosting the Grafana stack is genuinely viable and routinely under-costed. The licence is free; the operation is not. Metrics storage at scale needs care, log query performance needs tuning, and someone is on call for the thing that tells you whether you are on call.

Self-host when at least two of these hold:

• Volume is large and stable enough that per-GB pricing dominates the salary of a platform engineer
• Data residency or contractual terms rule out sending telemetry to a third party
• A platform team already exists and has capacity — not one you plan to hire
• Your retention needs are unusual enough that vendor tiers fight you

Otherwise buy, and spend the saved effort on instrumentation quality — which is where the answers come from, and which no vendor can do for you.

What would make us change our minds

• Skip tail sampling below roughly 10 million spans a day. Sample everything at 100%, keep it simple, and revisit when the bill notices.
• Skip traces entirely for a genuine monolith with no network hops in the request path. Profiles and good metrics will serve you better, and we would say so rather than sell you a tracing project.
• Stay on your existing log stack if OTel logs are pre-stable in your primary language. Adopt traces and metrics now, revisit logs next cycle.
• Reconsider self-hosting the moment cardinality growth means your metrics store needs a dedicated owner. That owner costs more than the licence you avoided.

The order we build it in

1. Pick one user-facing journey. Write one SLO for it, with a number someone will defend in a meeting.
2. Stand up the Collector as the seam, exporting to whatever you use today. No application changes yet.
3. Instrument that one journey end to end with OTel traces. Verify a trace crosses every hop, including the queue.
4. Derive RED metrics from those spans with the span-metrics connector. Delete the hand-rolled equivalents.
5. Add the three burn-rate alerts. Delete every threshold alert the new ones make redundant — this step is the one people skip, and it is where on-call quality comes from.
6. Put deploy and feature-flag markers on the SLI dashboard. Cheapest incident improvement available.
7. Add cardinality guardrails and the drop lists before widening to more services, not after.
8. Add tail sampling once volume justifies it, with trace-ID-aware routing from the start.
9. Then widen, service by service, with the guardrails already in place.

The ordering is the point. Teams that instrument everything first and define objectives later end up with a large bill, forty dashboards, and the same unanswerable question at 3 a.m. — which is precisely where this started.