I put together a demo of PostgreSQL as a Service where applications authenticate using their Kubernetes identity instead of a database password.

CloudNativePG manages the databases. OpenUnison validates the application’s ServiceAccount token through Kubernetes’ TokenReview API and exchanges it for a short-lived JWT. A sidecar handles refresh, and PostgreSQL 18 uses Percona’s OAuth validator to validate the JWT and map its identity to a database role.

This removes the application database password and its distribution/rotation process. You still manage database roles, permissions, and issuer trust.

A couple of caveats: the demo uses an unmerged JDBC OAuth PR, and TokenReview checks the Pod binding at exchange time. Issued JWTs remain valid until expiration, and expiration doesn’t terminate existing database connections.

Full walkthrough and demo code: PostgreSQL as a Service, Without Passwords

Disclosure: I’m with Tremolo Security, the company behind OpenUnison.

New video: cgroup v2 cpuset and memory limits 🐧

Why does one container get OOM killed while the rest of the machine is fine?

In this part of my cgroup v2 series I cover:

🔹 cpuset: pinning workloads to CPUs and NUMA nodes

🔹 memory.high vs memory.max: a soft ceiling that throttles and reclaims, against a hard ceiling that ends in an OOM kill

🔹 The kernel's allocation flow, step by step

🔹 A hands-on "memory hog" lab with stress-ng

The one-line summary is easy. The details (who does the reclaim, what you see in the counters) are where performance engineering gets interesting.

#Linux #cgroups #PerformanceEngineering #SRE #DevOps #Kubernetes

Cluster Info:

• Cluster note: GKE Dataplane V2 with WireGuard inter-node encryption (UDP 51871)
• Versions: Control plane and all live pools 1.35.8-gke.1796000, channel REGULAR (changed to STABLE during this incident)
• Anetd Image: cni-writer v0.1.40-gke.6, cilium v1.18.7-gke1.35-gke.39

I have also posted on https://discuss.google.dev/t/gke-anetd-mtu-shrinking-leads-to-network-failure-dataplane-v2/408300 but wanted to get more insights from everyone here that I've been following. Please let me know if this is not the right place.

Summary:

My team just recovered from a downtime started from a GKE control plane upgrade last night. Our observation so far points to a suspected bug somewhere in anetd cni-writer. I'm hoping for anyone who may have experienced something similar and can give pointers to the right direction.

• GKE's anetd cni-writer v0.1.40-gke.6 lowers a node's MTU by 80 bytes on every anetd restart.
• Pods created earlier keep the larger MTU, and their full-size packets vanish.
• We recover by recreating pods that predate anetd restart, then to be on the safer side, we replace every node to reset the MTU to baseline.
• Current measures my team is taking: monitor anetd pods for any restart/delete. Every node MTU is fresh at 1460, so one anetd restart drops that node MTU to 1380 while its running pods keep 1460. We observe that mismatch to be the outage.
• Logs from our apps:App -> RabbitMQ: Broker closed AMQP connections for missed heartbeats; unhandled ECONNRESET exits the processApp -> Valkey: Valkey replies dropped; ioredis waited for the kernel retry limit, then failed every queued command

So, the suspected bug, step-by-step:

1. A fresh node starts at MTU 1460: NIC, CNI configuration, host-side pod interfaces, and pod eth0. WireGuard cilium_wg0 is 1380, leaving 80 bytes for encryption.
2. Every anetd pod start reruns its cni-writer init container. Control-plane upgrades roll anetd on every node, including pools with node auto-upgrade off.
3. cni-writer reads cilium_wg0 (1380) as the base MTU and writes it as the CNI pod MTU.
4. Host-side pod interfaces shrink to 1380. Running pods keep eth0 at 1460 and send packets the host side drops, with no ICMP back. Large replies stall; clients time out.
5. Cilium subtracts 80 from the smaller base: WireGuard becomes 1300. The next restart copies 1300, and so on.

Logs from cni-writer anetd pods:

# anetd-9j66s on 3fc1dd1c-6nrh, second anetd start on this node
I1025 19:18:02.916461 cni_writer.go:378] Checking priority interface "cilium_wg0" for MTU.
I1025 19:18:02.916725 cni_writer.go:383] Using MTU 1380 obtained from priority interface cilium_wg0
I1025 19:18:02.916738 cni_writer.go:518] Successfully determined and cached MTU: 1380 (from cilium_wg0)

# anetd-cdmvs on 521c6876-gk4s
I1025 19:04:35.214705 cni_writer.go:383] Using MTU 1220 obtained from priority interface cilium_wg0

# anetd-l8mf6 on 56b8fd8a-mdxw, control-plane upgrade roll
I1025 17:23:25.846768 cni_writer.go:383] Using MTU 1300 obtained from priority interface cilium_wg0

Update, I also ran an LLM to check upstream repos for clues, here goes:

• GoogleCloudPlatform/netd PR #199 (commit ac67942e, 2023-02-27)
  • Prefers cilium_wg0 over the default NIC when writing pod MTU.
  • Matches the policy in our cni-writer logs; legacy shell source, not proven to be our binary.
• GoogleCloudPlatform/netd PR #202 (commit c50a56a0, 2023-02-28)
  • Refactors MTU selection into a loop, keeping WireGuard first.
  • The policy was kept, not reverted.
• cilium/cilium commit e99bee54 (2019-05-27)
  • CNI configuration MTU overrides the agent's configured MTU.
  • Why --mtu=1460 alone does not hold the base.
• cilium/cilium commit 6ae08fdc (2021-05-05)
  • WireGuard MTU is device MTU minus WireguardOverhead, 80 bytes.
  • Normal accounting becomes a loop when the base is copied from WireGuard.
• GoogleCloudPlatform/netd commit 38ff1139 (2025-01-22)
  • Skips legacy CNI writing on unified-CNI nodes, handing it to anetd.
  • Marks the boundary; does not date the defect.
• GoogleCloudPlatform/netd issue #33 (closed 2025-12-04)
  • Asks to move the shell CNI generator to Go; closed as 'CNI writer is there now'.
  • The Go writer exists; no public source or build mapping.

cilium-agent startup:

+ mtu=1460 + exec /usr/bin/cilium-agent --k8s-kubeconfig-path=/var/lib/kubelet/kubeconfig --config-dir=/tmp/cilium/config-map --mtu=1460 {"level":"info","msg":"Overwriting MTU based on CNI configuration","module":"agent.datapath.mtu","mtu":1460} {"level":"info","msg":"Using configured MTU","module":"agent.datapath.mtu","mtu":1460}

Reconstructed loop:

• The writer reuses an encapsulation-adjusted MTU as the unadjusted base. Cilium accepts it and subtracts the overhead again. The observed 80-byte steps match.
• We found no public cni_writer.go for v0.1.40-gke.6 and no build-to-commit mapping. The deployed Cilium revision b8f2f41d06 does not resolve publicly; upstream v1.18.7 is commit 2adae214.
• The public netd MTU test disables the Cilium plugin and mocks NICs, so it never exercises the restart loop.
• No public issue confirms this ratchet or a fixed GKE version.

currently I am a linux admin, having 8 yrs of rich experience. recently I learned a new skill docker n kubernetes and want to switch jobs now. Any refferal please

We're evaluating Envoy Gateway and Traefik as a replacement for ingress-nginx and our existing API gateway on Kubernetes (GKE).
Our backend consists mainly of Go microservices. We already have a Go authentication service that generates JWTs and manages client credentials, so we're considering either native JWT validation with Envoy or ForwardAuth with Traefik.
We're particularly interested in:

● Ease of deployment, configuration, and ongoing maintenance
● Performance and reliability in production
● JWT authentication and authorization
● Kubernetes Gateway API support
● Operational complexity and troubleshooting
We're trying to avoid adding too many infrastructure components to maintain.
Has anyone used both in production? Which would you recommend, and what issues have you encountered?

confused between above 3, can anyone explain with specific examples

Disclosure: I run a DevOps consultancy (MeteorOps). Nothing for sale here; my co-founder wrote the long version, link in a comment.

We set CPU limits for every container, but if there's a latency-sensitive service we avoid setting any limits. When I say "we" it's actually most of the team, but some opt for setting limits on everything.

The obvious follow-up on the no-limits school is that we give the developers dashboards to be aware of the apps' resource utilization, and over time it improves, but it's more of a "culture" fix.

For the CPU-limits-setting engineers in the crowd, what do you do to prevent over-utilization crashes?

Got something working? Figure something out? Make progress that you are excited about? Share here!

What happens if you break an embargo? Like, fines or worse? Or nothing?

“On October 21st at 07:41 UTC, more than six hours before the coordinated release, Chainguard unilaterally broke the embargo and published built images containing the fix. Like other distributors, Chainguard had received the patch in advance so its users could update the moment the vulnerabilities were disclosed. Any attacker could now study the fix in binary form while every other PHP user was still waiting for the release. This disregarded weeks of work by many parties toward a coordinated disclosure.”

Over the years, I have submitted a couple of pull requests to repos under the Kubernetes GitHub organization. None of them were ever reviewed or merged. And I don't mean rejected (for low quality, missing explanation, etc.), just ignored forever without any form of acknowlegement. Most of them even resolved existing issues and feature requests.

Yet, these repos have PRs that get merged multiple times a day. And if you look at the people who create the PRs and who approved them, it's mostly the same groups of people:

• members of the Kubernetes organization
• work at a CNCF-affiliated company (ex. Google, RedHat, Microsoft, Datadog)
• part of CNCF's developer programs (ex. LFX Mentorship, GSoC)

So I was wondering, is Kubernetes actually open to outside contributors? To me, Kubernetes looks like a public collaboration between a handful of companies rather than a community project.

EDIT: to add more context, the PRs were very small and wouldn't take more than 5 minutes to find the context to review. Nothing fancy. Since I am also an open source maintainer, I knew I always have to follow the contributing process for each individual repository (using templates, signing a CLA, etc.). I also remember joining a Slack channel and messaging people to review a PR, made a couple of followups, no reaction.

I'm working as L1 Linux System Admin. I want to learn Docker & Kubernetes in a structured way. Can you please suggest some good courses for Docker & Kubernetes which is still relevant in 2026?

Is there any real reason to put a heavy appliance like NetScaler in front of the API server? It feels like massive overkill, and I'd rather avoid dealing with the network team's ticket queue.

I'm leaning toward kube-vip or just spinning up some HAProxy VMs. What are you all running in production for your master node LB, and why?

A detail I hadn't fully appreciated until I rebuilt Monzo's 2017 outage: a normal Service always resolves (to its ClusterIP) even with no endpoints, but a headless Service (clusterIP: None) resolves straight to its ready Pods, so with zero Pods its name has no records at all.

Put that name in an nginx proxy_pass and nginx won't start: it resolves every upstream once at startup and exits if any lookup fails. Running routers don't notice, because they loaded the config before the route was added. Restart them all at once and everything behind the router goes down:

nginx: [emerg] host not found in upstream "cards-v2" in /etc/nginx/conf.d/default.conf:6

kubectl logs deploy/router --previous is the command that finds it, since the crash-looping container has usually restarted by the time you look.

Guide (free): https://learn.k8sight.in/go/nginx-upstream

Also new: CPU throttling from limits set to the average (Buffer), and Pods Running but not Ready after a dnsPolicy: None change (Salesforce). They're all at https://learn.k8sight.in/go/k8s-guides
Disclosure: I built the platform; the hands-on incidents are paid, the guides aren't. Which outage would you want rebuilt next?

Hello,

So I have deployed this micro services demo in my cluster using flux. I want to collect metrics from services so I have deployed the monitoring stack using kube-Prometheus-stack and want to use service monitors to identity the services but I see most of the services expose grpc port but Prometheus collect over http right so what can I do in this situation? Thanks

Follow-up to yesterday's gVisor intro. This is the stub-memory bug I mentioned there, with the debugging steps.

gVisor's systrap platform runs app code in host "stub" processes and shifts their shared memory by a random, page-aligned gap at startup, so the addresses are hard to guess. In stub_unsafe.go three regions get += gap. The spinning queue, sitting between them in the layout, did not.

With a big gap the queue sits alone in empty space. With a small one, the shifted per-thread stacks land on the queue's old address. The stub then spins in spinning_queue_remove_first waiting for a timestamp slot that stack bytes keep at 0. That check sits before the timeout, so it never gives up. The Sentry waits on a context no stub ever acknowledged, and my 30s fail-stop, which kills a stuck stub process, only arms once an acknowledged context was interrupted. The sandbox never exits.

On Kubernetes it looks like this: a pod stuck in Terminating, one stub process pinned at 100% of a core, and runsc kill hanging.

Back of the envelope from the code: the gap is uniform below a random room of up to 1 TiB, so the odds come out roughly 1 in 100,000 starts for a 16-CPU sandbox, and about 1 in 35,000 at 64. That is an upper bound, since a hang also needs the overlapped stack in use. It had been there since the spinning queue landed in March 2023.

Forcing a small gap in an arm64 VM: stock build hung 2 of 2, with the one-line fix 20 of 20 clean.

If you run gVisor with systrap (release-20230417.0 through release-20261005.0): upgrade to a runsc build that contains c7f458aa3. Until then --systrap-disable-fast-path (release-20260622.0 or later) stops the spinning, at some syscall latency cost. To clear a hung pod, kill its runsc-sandbox process on the node.

Fix: https://github.com/google/gvisor/pull/15573

Write-up with the debugging steps and diagrams: https://www.catchkill9.dev/posts/gvisor-spinning-queue-gap.html

kubectl debug node/<name> puts the node's root filesystem at /host, but the pod isn't privileged, so chroot /host fails unless you add --profile=sysadmin. It also works one node at a time and leaves the pod behind until you delete it. When you need journalctl -u kubelet from every GPU node, what do you run, and who gets to run it?

for people running cnpg in prod, what caught you out that you didnt see coming?

one thing we noticed: a cluster with no backups configured at all still shows ContinuousArchiving=True, "Continuous archiving is working". the archive command just returns ok when theres nowhere to send the wal. kubectl cnpg status does say "Continuous Backup not configured", but an alert on the condition alone would miss it

curious about the rest. failovers, wal filling the disk, upgrades, restores that didnt work, whatever it was, and how you found out

full disclosure i work on radar (open source k8s ui). we've been gradually adding cnpg support and want to make sure it covers the stuff that actually bites people

Did you learn something new this week? Share here!