Engineering teams are taking longer to resume work after incidents, with MTTR over one hour rising from 47% in 2021 to 82% in 2024. During the same period, teams increased the number of observability tools used to eight or nine platforms. The common industry response has been to add more tools, dashboards, and signals, assuming visibility improves speed of fixing. Evidence presented indicates the opposite: too much observability data can create cognitive overload, making root cause analysis harder and increasing MTTR. A real on-call example shows dashboards appeared healthy while packet drops were caused by SNAT port limits exceeded by bursty traffic, resolved by adding nodes after packet capture.
"The amount of time it takes engineering teams to get back to work after an incident is getting worse every year, even though spending on observability tools has reached record highs. This should worry everyone in this field. The Logz.io Observability Pulse followed teams with a mean time to resolution (MTTR) of more than one hour: 47% in 2021, 64% in 2022, 74% in 2023 and 82% in 2024. Four years in a row of going backwards. During the same time, the average number of tools used by a team rose to eight or nine different platforms."
"The answer from the industry has always been the same: More - more tools, more dashboards, more signals. The working assumption is that the problem is visibility; that if engineers could see more of what was going on, they would be able to fix problems faster. This article asserts that the contrary is accurate: Beyond a specific limit, excessive observability data results in cognitive overload that hinders root cause analysis (RCA) and increases MTTR. More signals can, surprisingly, mean less clarity."
"While I was on-call for a big cloud provider, at one of my previous jobs, I had to deal with a problem where a customer was seeing packet drops during a performance test. The war room got going quickly. We looked at all our dashboards, which showed things such as average CPU, per-node CPU, soft IRQ and memory use across the fleet. Everything seemed fine. There was no problem anywhere. We spent a long time in that space, methodically going through the stack, sure that the answer was in there somewhere if we just looked harder."
"However, it wasn't. We finally did a packet capture, which is a simple, old-school way to diagnose a problem, and we found the real problem right away. Bursty traffic had pushed the use of the SNAT port past its limit. There were too many connections happening at the same time. The solution was simple: Add nodes."
Read at DevOps.com
Unable to calculate read time
Collection
[
|
...
]