{"id":1084,"date":"2026-11-14T14:40:00","date_gmt":"2026-11-14T14:40:00","guid":{"rendered":"https:\/\/planespart.com\/blog\/?p=1084"},"modified":"2026-11-15T08:40:00","modified_gmt":"2026-11-15T08:40:00","slug":"diagnosing-leaky-cluster-kubernetessecops-repair-log","status":"publish","type":"post","link":"https:\/\/planespart.com\/blog\/diagnosing-leaky-cluster-kubernetessecops-repair-log\/","title":{"rendered":"Diagnosing a Leaky Cluster: A KubernetesSecOps Repair Log"},"content":{"rendered":"<p>Every maintenance hangar has a story about the fault that was not on the squawk sheet. A pilot reports a faint vibration, the technician finds a worn bracket, and tracing it back reveals that the real culprit was a washer that nobody had ever put on the inspection list. This article tells a similar story, one that is illustrative and not taken from any real company, about an online parts business whose storefront runs on a container cluster. The repair method is what the Ops Atlas calls KubernetesSecOps, the daily upkeep of who can change a cluster, what may run, what may talk and what gets recorded.<\/p>\n<p>Think of the team as a small shop of five people. They sell avionics bits and fittings, and their stock catalogue, order system and customer accounts all live on one cluster. Nothing has gone wrong yet. They simply feel uneasy, the way a mechanic feels before a signed-off aircraft returns with a snag.<\/p>\n<h2>The symptom: nobody can say who touched what<\/h2>\n<p>The first complaint is mundane. Last Tuesday a configuration changed, an order service restarted, and nobody can explain who did it. That is the equivalent of finding a torque stripe broken with no entry in the log. According to <a href=\"https:\/\/www.xopsschool.com\/kubernetessecops\/\">the Ops Atlas discussion of KubernetesSecOps<\/a>, a cluster you run yourself writes no record of API actions unless an audit policy file is supplied, so the missing trail is not carelessness; it is the factory setting.<\/p>\n<p>Diagnosis: no audit policy. Repair: write one, remembering that the first rule that matches wins, so specific entries sit above broad ones. For secrets and configuration maps, keep only who-and-when details, so the log never ends up holding the sensitive contents itself. Re-inspect: edit some throwaway object, then look for that edit in the log with a user name attached.<\/p>\n<h2>The second finding: everyone is a chief engineer<\/h2>\n<p>While reading the audit entries, the team notices that their build robot deploys with the broadest administrator role in the cluster. It was convenient during setup and nobody revisited it. The Atlas lists this exact habit among its anti-patterns: granting the top role to continuous delivery. A compromised build account then becomes a compromised cluster.<\/p>\n<p>In an aircraft shop you would not hand the stores key, the release-to-service stamp and the tool crib to a single apprentice. The repair is to cut the robot&#8217;s role down to the verbs it truly uses in the one namespace it deploys into. The team also takes two human accounts out of the built-in superuser group, which the Atlas singles out because rights checks are simply skipped for its members.<\/p>\n<p>For another angle on this kind of mistake, readers who browse <a href=\"https:\/\/planespart.com\/blog\/control-system-parts-buying-mistakes-to-avoid\/\">the list of control-system buying mistakes<\/a> will notice the pattern: the expensive failures are usually about authority and process, not about the part itself.<\/p>\n<h2>The third finding: a shared hallway<\/h2>\n<p>Next the team asks a question they had never asked: can the catalogue service be reached from the payments service, and can both be reached by any stray program? They test it and the answer is yes to everything. That matches the Atlas&#8217;s description of the starting state: a pod is open to all callers until some network policy picks it out.<\/p>\n<p>The fix is to create a deny-everything policy in each application namespace and then open only the paths that are needed, catalogue to database, storefront to catalogue and so on. One subtlety appears in their notes from the Atlas: connections that were already open before the deny rule landed might survive, depending on the network plugin, so they restart the affected services after rolling out the policy. They also stop assuming that a namespace is a wall. Pods in different namespaces share a kernel, and the team keeps the sensitive services on separate nodes by a scheduling rule rather than by wishful thinking.<\/p>\n<h2>The fourth finding: the plain-text drawer<\/h2>\n<p>Then comes the awkward discovery. Customer-facing credentials are stored as secrets, and the team believed that was the safe place. The Atlas says otherwise by default: those objects sit unencrypted in the datastore unless encryption at rest is configured. The team also learns that base64 is just a way to write bytes, not a lock, and that anyone allowed to list secrets in a namespace can read the lot.<\/p>\n<p>The repair has two parts. They turn on encryption at rest using the key-management integration that the Atlas recommends, noting that the first-generation version was deprecated several releases ago, and that one of the common cipher options is rated weak. Then they review who holds the right to list or watch secrets and prune it. A subtle point follows from the Atlas: anyone who can create a workload in a namespace can in effect read that namespace&#8217;s secrets by mounting them, so workload-writing rights count as secret-reading rights.<\/p>\n<h2>The fifth finding: a component that nobody owns<\/h2>\n<p>The last finding is the most sobering. Their traffic-routing component, the controller that brings outside visitors in, is an open-source project they installed two years ago and never revisited. The Atlas uses a well-known case to show why that matters: a critical flaw disclosed in March 2025 allowed something inside the pod network to abuse the controller&#8217;s admission webhook and take over the cluster, and in its stock configuration the controller was able to read secrets cluster-wide. The project was later archived and is no longer getting fixes.<\/p>\n<p>In hangar terms, this is a supplier discontinuing a part that is fitted to every aircraft you operate. The Atlas repeats the oversight committees&#8217; caution that without active checking, an affected owner may only learn of it after a break-in. The team checks its version, finds it exposed, and plans a migration to a maintained alternative. Until then they disable the vulnerable webhook, which the advisory allows as a stopgap, and limit what the component&#8217;s account can read.<\/p>\n<p>This kind of inventory discipline mirrors what careful buyers do when they study <a href=\"https:\/\/planespart.com\/blog\/complete-guide-to-aircraft-fasteners-and-hardware\/\">how aircraft hardware is specified and tracked<\/a>: know every part you rely on, and know who stands behind it.<\/p>\n<h2>Ranking the repairs<\/h2>\n<p>With five findings on the board, the team does what any good planner does and ranks by consequence and effort. Their first week covers audit logging and cutting down the build robot&#8217;s role, because both are quick and both reduce the damage of any incident. The second week adds default-deny policies in a trial namespace and begins the encryption change. The third handles the unowned component. They schedule a monthly review of exceptions, and a quarterly look at access rights, as the Atlas&#8217;s rhythm suggests.<\/p>\n<p>They also agree on how to roll out any new blocking rule: first observe, then warn with a published date, then enforce. A rule that suddenly stops the night shift from deploying is a rule that will be switched off by Friday, and a switched-off rule protects nobody.<\/p>\n<h2>Mistakes they decided not to make<\/h2>\n<ul>\n<li>They did not turn the whole sample-policy library to blocking on the first day.<\/li>\n<li>They did not set every policy engine to ignore failures, which would silently turn a safety gate into decoration.<\/li>\n<li>They did not plan to delete a suspicious pod automatically, since that would erase evidence and leave stolen tokens usable.<\/li>\n<li>They did not treat a clean permissions scan as proof of health.<\/li>\n<\/ul>\n<h2>What the story shows<\/h2>\n<p>None of these repairs required exotic tools. They required noticing the defaults, asking uncomfortable questions in a particular order, and writing the answers down. A shop that sells aircraft parts understands that trust is built from records, and the same is true on the software side. If your own business depends on a cluster and the squawk sheet is blank, the sensible assumption is that the washer is still missing. For sourcing decisions beyond the cluster, <a href=\"https:\/\/planespart.com\/blog\/sourcing-enterprise-technology-a-strategic-manual-for-application-selection\/\">this manual on choosing enterprise applications<\/a> is a decent companion read.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A maintenance-shop style walk from symptom to fix, showing how KubernetesSecOps habits turn a vague worry about cluster safety into a short list of repairs.<\/p>\n","protected":false},"author":6,"featured_media":1085,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[617],"tags":[],"class_list":["post-1084","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aviation-maintenance"],"_links":{"self":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/1084","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/comments?post=1084"}],"version-history":[{"count":1,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/1084\/revisions"}],"predecessor-version":[{"id":1181,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/1084\/revisions\/1181"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/media\/1085"}],"wp:attachment":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/media?parent=1084"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/categories?post=1084"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/tags?post=1084"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}