Why Observability Isn't Optional in Worker Pipelines
A worker pipeline you can't see is a worker pipeline you can't debug at 3 AM.
When a job stalls, three questions arrive before the on-call engineer is fully awake: where is it, why did it stop, and what state is it in. Logs alone won't answer all three.
Three pillars matter, in this order:
1. Structured logs with correlation IDs. Every job carries a job_id and stage that propagates through every log line, queue message, and downstream call. One grep, one query, full lifecycle visible.
2. Queue and worker metrics. Track queue depth, worker count, jobs in flight, and per-stage duration histograms. Alert on queue depth growing faster than drain rate — that's the zombie worker signature.
3. Distributed traces across stages. OpenTelemetry spans reveal which stage consumed the latency budget. A trace spanning 200ms with 180ms in one stage points at exactly one line of code.
Skip any one and you reconstruct state by hand for hours. Wire all three and you answer every 3 AM page in seconds.