A maintenance technician troubleshooting an aircraft system and an on-call engineer facing a broken online service are solving the same kind of problem: something is failing, the cause is unknown, and the clock is running. This article diagnoses the common ways such responses go wrong and pairs each with a repair, drawing on the discipline some people call IncidentOps. The aim is precision rather than hype, so each problem is stated first and the fix follows.
For definitions, the working source here is the Atlas reference page for IncidentOps, which describes the term as a loose label for incident management and lists the habits that separate orderly responses from chaotic ones. Everything below that touches on the discipline comes from that entry or from general maintenance practice.
Problem 1: Nobody calls it an incident until it is large
Symptom. Three people have noticed a fault. Each assumes someone else has raised it. By the time a formal response starts, the problem has spread and customers have noticed.
Diagnosis. There is no written threshold for escalation, so declaring an incident feels like admitting a mistake. The Atlas lists this as one of its myths and notes that punishing teams for the number of declared incidents simply teaches them to stay quiet.
Repair. Adopt criteria in writing. The entry cites a search giant whose triggers include needing help from another team, customers seeing the problem, or the fault remaining unsolved after about an hour. In a hangar, the equivalent is a defect-reporting rule: if a finding touches airworthiness or needs another trade, it goes on the record immediately. A declaration that proves unnecessary costs minutes; a late one costs hours.
Problem 2: Everyone fixes, nobody coordinates
Symptom. Four engineers are changing things at once. One undoes a change another just made. Nobody has the full picture.
Diagnosis. Roles have not been assigned. The Atlas says the coordinator should decide and organise but not be a resolver, while only a designated group of subject experts changes production. Outward communication belongs to a separate person.
Repair. Assign the jobs explicitly and treat them as hats rather than ranks. The entry also notes that the person wearing the coordinator hat is the highest-ranking one present on the call, regardless of their usual position, and that the role should rotate. Compare this with a shift handover on the hangar floor, where the licensed engineer signing off the work is not necessarily the most senior person nearby.
Problem 3: The team insists on knowing the cause before acting
Symptom. A fault is isolated to a subsystem, yet the team debates theories for an hour while the system stays down.
Diagnosis. Root-cause analysis and restoration have been confused. The Atlas separates them: stopgaps like a rollback, a drain, a feature switch-off, a failover or load shedding depend on knowing where the fault sits, not why it exists.
Repair. Agree up front that relief comes first and the explanation second. An aircraft on the ground follows the same logic when a technician swaps a suspect line replaceable unit to restore serviceability and sends the removed unit to the shop for deeper investigation. The Atlas shows the price of ignoring this with a long managed-cluster failure, where going back to older images the moment the faulty area was found could have ended the pain well before the actual fix landed.
Problem 4: Updates stop when the work gets hard
Symptom. Customers and managers refresh a status page that says nothing new. Executives start phoning engineers directly.
Diagnosis. No one owns communication, or the cadence is undefined.
Repair. Name a communications lead and fix a rhythm. The entry sketches a timetable: a first post roughly five minutes after the call begins, followed by a scoping note, a refresh every twenty minutes or sooner through the early stretch, and a closing notice that says plainly whether data went missing. It also offers a polite script for an interrupting executive: ask whether they wish to take command. That one question usually ends the interruption or transfers responsibility cleanly.
A good rule from maintenance control applies here too: a customer told “still investigating, next update at the top of the hour” is easier to manage than a customer told nothing.
Problem 5: The review blames a person
Symptom. The written summary names “human error” as the cause and the action list contains a reminder to be more careful.
Diagnosis. The investigation stopped too early. The Atlas leans on a safety researcher’s point that looking back from a known outcome warps the way people rate the operators involved. When a mistake was possible, the question to ask is what made it easy.
Repair. Keep digging until you reach conditions in the system: unclear procedures, a confusing interface, missing checks. Aviation maintenance investigation frameworks reached this view long ago, which is why a modern occurrence report looks for contributing factors in the task, the tools and the environment rather than a single person.
Problem 6: Action items pile up and nothing changes
Symptom. Every review produces twelve actions. A quarter later, most are still open, and several were closed by redefining them.
Diagnosis. The Atlas lists several anti-patterns here: actions that are mostly stopgaps, vague phrasing, equal priority for everything, and closing items by downgrading them. More actions do not mean a better review.
Repair. Keep the list brief, attach a single name and a single deadline to each item, and track how many get closed. One vendor cited in the entry sets a target of four or eight weeks for priority actions depending on the service. Whatever the target, it only works if somebody checks it.
Problem 7: Too many people on the call
Symptom. Forty people join a call. The core responders retreat to private chats to get anything done.
Diagnosis. The crowd was invited by default. The Atlas points to field research where crowds of hundreds pushed the working group into private side channels, and to one long cloud-service failure that had dozens of people in the main chat.
Repair. Run one working channel for responders and a separate updates channel for observers. Keep each supervisor to roughly five people and form sub-teams beyond that. The emergency-management guidance behind this figure comes from the fire service, and it maps neatly onto a hangar: a lead can supervise a handful of technicians effectively, not twenty.
Why the fixes hold up
Each repair above removes a decision from the moment of stress. Written triggers settle the debate about escalation, named hats settle who speaks, and a fixed message rhythm settles when the next note is due. Maintenance organisations rely on the same idea whenever they replace memory with a task card: under pressure, people follow the card.
A short field diagnostic
When you suspect your own process is leaking, scan this list:
- Is there a written trigger, and has anyone used it this month?
- Can every responder name who is coordinating right now?
- Do you hold off changes that are not mitigations until service is back?
- Is there a communication owner and a time-boxed rhythm?
- Do reviews end with a handful of dated, owned actions?
Related reading on this site
Budgeting for failure is a theme beyond engineering. If you want to see how a household or workshop traces a leak back to its source, a piece on tracking where monthly spending goes uses the same inspect-then-attribute logic. For readers weighing a move into infrastructure roles, advice on levelling up an engineering career covers the learning path.
Conclusion
The pattern across these seven problems is consistent. Declare early, assign roles, restore before explaining, communicate on a clock, review without blaming a person, keep actions few and owned, and keep the crowd small. None of it needs expensive tools. It needs agreement, written down, before the next fault appears.

