What Kubernetes Audit Logs Don't Show (and What Does)

TL;DR. The Kubernetes audit log sees requests to the API server, and nothing else. It does not see what happens inside a container or on a node, requests made directly to the kubelet or to etcd, cloud IAM changes, or network traffic. What it sees depends on your audit policy and on the managed provider's; some fields (source IP, user agent) are client-influenced; and pattern-based rules miss attackers who behave like your admins. Pair it with runtime detection, node and container logs, cloud audit logs and flow logs.

Every article in this series argues that the audit log is the centre of a Kubernetes investigation. This one explains where it stops, so a "clean" result is read for what it is.

Blind spot 1: inside containers and on nodes

The API server authorises and records the request that opens an exec session, including its initial command, then relays bytes it does not log. Everything after that is invisible:

• commands typed in an interactive shell;
• processes started by the application after an RCE (a reverse shell, a downloaded binary);
• file changes inside the container;
• anything done on the node after a container escape: new SSH keys, cron entries, kubelet credential reuse, containers started directly through a runtime socket.

The audit log shows the preparation (a privileged container, a hostPath of /, an exec with chroot /host), not the execution. For the rest you need runtime evidence: Falco or an EDR on the nodes, the nodes' own logs (journald, auth logs), container logs, and a disk snapshot of affected nodes.

The analyzer accepts Falco JSON alerts next to the audit logs and puts both on one timeline; alerts of priority Error and above become findings.

Blind spot 2: paths that bypass the API server

The cloud side has its own audit trail: CloudTrail for EKS, Cloud Audit Logs for GKE, the Azure Activity Log for AKS. The sibling sites AWS Forensics, GCP Forensics and Azure Forensics cover those.

Blind spot 3: what the policy did not keep

What you can detect is bounded by what was logged:

• Level Metadata removes request bodies. A DaemonSet creation is visible; whether it was privileged, which image it ran, which role a binding granted, is not. Several detections simply cannot fire. The audit policy guide shows what to log.
• Managed policies make their own choices: the EKS policy published in the EKS best practices guide does not log Events; GKE sends reads to Data Access logs that are off by default; AKS kube-audit-admin drops get and list. See the EKS, GKE and AKS guide.
• Delivery and size. EKS describes control-plane log delivery as best effort, and CloudWatch Logs caps entries at 1 MB, so very large objects can be truncated.
• Retention. Logs that expired before the investigation started are gone. Initial access often predates detection.
• Coverage. On self-managed clusters, each API server instance logs the requests it served; missing one control-plane node's file means missing part of the story.

The analyzer reports what it read (files, time range, sources, skipped and truncated records) precisely so these gaps are visible.

Blind spot 4: fields you cannot fully trust

• sourceIPs: the audit event reference warns that all but the last IP can be set by the client through X-Forwarded-For. Behind load balancers or private endpoints, the last IP may be an infrastructure address.
• userAgent: client-supplied. A careful attacker sends the same agent as the workload whose token they stole.
• Identity: the log names the credential, not the person. A stolen token and its legitimate workload look identical except for context (source, timing, behaviour).
• Impersonation: read user and impersonatedUser together; reporting only one misattributes actions.

Blind spot 5: attackers who look normal

Detection rules match patterns. They flag a service account running kubectl exec, a binding to cluster-admin, a miner image. They do not flag:

• an attacker using a stolen admin's credentials to do what that admin normally does;
• a malicious image with an ordinary name from your usual registry;
• data read slowly, one secret per hour, below any burst threshold;
• an identity named to resemble a system component. The analyzer allow-lists controller identities by prefix, which is necessary to avoid thousands of false positives but means a service account named to match an allow-listed prefix in kube-system could be excluded. Review who can create service accounts there.

Baselines help: first exec by an identity in 30 days, first request from a new network, a workload created outside the deployment pipeline. Those comparisons need history the analyzer does not keep between sessions; your SIEM can.

Reading a clean verdict

"No sign of compromise" means none of the rules matched in the data you provided. Before accepting it, check:

1. Does the time range cover the period of concern, with margin?
2. Were all control-plane sources included, and did any files get skipped?
3. Does the policy log bodies for workload and RBAC writes?
4. On GKE and AKS, were reads logged at all?
5. Is there independent evidence (runtime alerts, cloud bill, node behaviour) that contradicts the result?

If worries remain, escalate to an experienced incident responder and collect runtime evidence before the nodes are recycled.

The evidence map

FAQ

Do Kubernetes audit logs record commands run inside a container?

Only the initial command of a kubectl exec, which is part of the request URI. Anything typed in the session afterwards, and any process started by the application itself, is invisible to the API server and needs runtime tooling.

If the analyzer says no sign of compromise, is the cluster clean?

Not necessarily. It means no rule matched in the logs you provided. Missing time ranges, a metadata-only audit policy, activity on nodes or inside containers, and attackers who mimic normal behaviour can all leave no match.

What should I pair audit logs with?

Runtime detection on nodes (Falco or an EDR), node and container logs, the cloud provider's control-plane audit logs, network flow logs, and your deployment history from CI or GitOps.

Related articles

• Kubernetes incident response with audit logs
• Kubernetes audit policy for forensics
• Privileged pods and container escape indicators