A Namespace Relabel Silently Broke a NetworkPolicy Selector, and Cross-Namespace Traffic Started Timing Out

By KP | TZoneLabs | DevOps & Cloud Engineering

A payments-team service started timing out on calls to a shared internal API, intermittently, with nothing in either side’s logs pointing at why. No connection refused, no TLS error, no 5xx. Just a hang until the client’s own timeout gave up. The pods were healthy, the Service had correct endpoints, and nothing in the deploy history for either service lined up with when it started.

This covers how a routine namespace label cleanup silently tripped a NetworkPolicy default-deny rule, why the failure produced no errors on either side, and what we changed so a label change like that can’t cause a silent networking outage again.

What Happened

Every namespace in the cluster runs under a default-deny NetworkPolicy, with explicit allow rules layered on top using namespaceSelector matches against a custom label, network-zone, that groups namespaces into trust zones. A few days earlier, as part of a cleanup pass standardizing namespace labels across the cluster, someone ran a bulk kubectl label namespace --overwrite pass to align legacy labels with a new naming convention. It touched every namespace, including the ones this policy depended on.

Where We Started Looking

Step 1 — Check both services’ application logs

context deadline exceeded

That’s all the calling service logged. Not a connection error, a timeout, meaning the request never got a response of any kind, not even a rejection. The destination service’s logs showed nothing at all for these calls, which meant the request wasn’t reaching it.

Step 2 — Confirm the destination is healthy and reachable in principle

kubectl get pods -n internal-api -o wide
kubectl get endpoints internal-api -n internal-api

Pods Ready, Endpoints populated with the correct pod IPs. Nothing here explained a hang.

Step 3 — Test the connection directly, below the application layer

kubectl exec -n payments deploy/payments-api -- curl -v --max-time 5 http://internal-api.internal-api.svc.cluster.local

*   Trying 10.42.1.88:80...
* Connection timed out after 5000 milliseconds

A timeout at the raw connection level, before any HTTP exchange. DNS resolved the Service correctly, so the request knew exactly where to go. Something between the two pods was dropping the packets rather than the destination refusing them, and a silent drop with no RST is the signature of a NetworkPolicy denial, not an application problem.

Step 4 — Check what the destination namespace’s NetworkPolicy actually requires

kubectl get networkpolicy -n internal-api -o yaml

spec:
  podSelector: {}
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              network-zone: internal

kubectl get namespace payments --show-labels

NAME       LABELS
payments   kubernetes.io/metadata.name=payments,team=payments,env=prod

No network-zone label anywhere on the payments namespace. It had been dropped during the relabeling pass days earlier, replaced by the newer team and env labels, with nobody realizing a NetworkPolicy’s ingress rule depended on the one being removed.

Why This Produced No Errors on Either Side

🔴 Root cause: a namespace relabel removed a label a NetworkPolicy’s namespaceSelector depended on, and the resulting default-deny drop is invisible by default.

The internal-api namespace’s ingress rule only allowed traffic from namespaces carrying network-zone: internal.

A bulk namespace relabel, done to standardize on newer label keys, overwrote every namespace’s labels and dropped the older network-zone label in the process.

Once payments lost that label, it no longer matched the allow rule, and the default-deny policy started silently dropping every packet from it, with no log line, no Kubernetes event, and no alert, since the CNI wasn’t configured to log denied connections.

Both sides saw exactly what a network partition looks like from the inside: the caller times out, and the destination never even sees the attempt.

Nothing about the incident pointed at NetworkPolicy specifically. It looked like a networking blip until someone thought to check what had changed on the namespaces themselves.

The Fix

Immediate: restore the missing label

kubectl label namespace payments network-zone=internal

Traffic recovered immediately once the label was back, confirming the diagnosis before touching anything else.

Real fix: stop keying critical policy on a label anyone can overwrite

# Before — depends on a custom label with no protection against removal
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            network-zone: internal

# After — keyed on the immutable name label Kubernetes sets itself,
# combined with the custom label as an additional, documented requirement
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: internal-api-clients

kubernetes.io/metadata.name is set automatically by Kubernetes and can’t be overwritten away by an unrelated relabeling pass the way a hand-added custom label can. Where a custom trust-zone label is still useful, it now lives in the same GitOps-managed manifest as the namespace definition itself, instead of being something anyone can change with an ad hoc kubectl label command.

What We Put in Place After

1. Turned on connection-denied logging in the CNI

The default-deny policy had been silently dropping traffic with nothing to show for it. We enabled policy denial logging so a spike in denied connections shows up as a metric, not just a downstream team noticing timeouts.

2. Moved namespace labels into version control

Namespace labels that any NetworkPolicy depends on are now defined in the same GitOps-managed manifest as the namespace itself, so a bulk relabel has to go through a diff and a review instead of a one-off kubectl command touching every namespace at once.

3. Added a synthetic cross-namespace connectivity check

A small probe now calls between a few representative namespace pairs on a schedule and alerts on a sustained timeout, so a silent policy drop like this one gets caught in minutes instead of waiting for a service owner to notice.

Key Lessons

Default-deny NetworkPolicies fail silently, not loudly.
A dropped connection looks identical to a network partition from both ends, no error, no log line, just a hang until timeout.
A policy keyed on a mutable custom label is only as durable as whatever process manages that label.
Anything a routine cleanup script can overwrite shouldn’t be load-bearing for security-critical rules.
kubernetes.io/metadata.name is the one namespace label nothing else can quietly remove.
Prefer it over custom labels for policy selectors wherever the grouping it provides is good enough.
Most CNIs don’t log denied connections until you turn it on.
Enable it ahead of time, since after an incident is the wrong moment to discover the visibility doesn’t exist.
Namespace-wide label changes deserve the same review as any other infrastructure change.
An ad hoc kubectl label pass across every namespace has no diff, no review, and no easy rollback.

Summary

Layer	What Happened	Tool to Check
Namespace Labels	Bulk relabel dropped a label a NetworkPolicy depended on	`kubectl get namespace --show-labels`
NetworkPolicy	Default-deny silently dropped traffic from the now-unmatched namespace	`kubectl get networkpolicy -o yaml`
Application Logs	Showed only a generic timeout, no indication of the real cause	Service logs on both sides
Raw Connectivity	Confirmed the drop happened below the application layer	`curl` from inside a pod, direct to the destination IP

The label removal itself was routine and reviewed as a naming cleanup. Nobody involved knew a NetworkPolicy was quietly depending on the label being cleaned up.