Kubernetes Incident Response with API Server Audit Logs

TL;DR. In a Kubernetes incident, the API server audit log is your main source of truth: every kubectl exec, secret read, new binding and new workload went through the API server, and each can be recorded with who, from where, what and whether it was allowed. Preserve the logs first, then answer five questions in order: which identity, since when, what it touched, what it left behind, and what it can still reach. The free in-browser audit log analyzer gives you a first verdict, findings mapped to MITRE ATT&CK and a timeline without uploading anything; the rest of this guide explains what sits behind each step.

Kubernetes incidents rarely start with an alert that says "your cluster is compromised". They start with a cloud bill that doubled, a node pegged at 100% CPU, a Falco rule nobody has tuned, or a service account token found in a public repository. At that point you need to know quickly whether someone actually used the cluster, and what they did with it.

Why the audit log is the centre of the investigation

Every change to a cluster, and most reads, go through the Kubernetes API server. Kubernetes auditing records those requests as structured events: the authenticated user and groups, the source IPs, the user agent, the verb, the target resource, namespace, name and subresource, the response code, and, depending on the audit level, the request and response bodies.

That makes it the one log that covers most of an attacker's work through the control plane:

What it does not show is equally important: commands typed inside a shell once it is open, processes started on a node after an escape, or traffic between pods. The limitations article covers those gaps and which runtime sources fill them.

Step 0: preserve before you touch

The instinct is to delete the malicious pod. Resist it for a few minutes.

1. Export the audit logs for the full period and store them outside the cluster. On managed platforms they live in CloudWatch Logs (EKS), Cloud Logging (GKE) or Azure Monitor (AKS); the export guide for EKS, GKE and AKS has the exact commands. Check retention: logs that expire tomorrow are the first thing to save.
2. Save the specs of suspicious objects with kubectl get <kind> <name> -o yaml before deleting them: DaemonSets, CronJobs, pods, ClusterRoleBindings, ServiceAccounts.
3. Snapshot the disks of affected nodes if you suspect a container escape. A replaced node takes its evidence with it.
4. Write down what you have already done, with times in UTC. Your own kubectl commands will appear in the same audit log.

The AWS guidance on EKS incident response and forensics and Google's GKE security mitigations follow the same order: isolate, preserve, then remediate.

Step 1: get a first read of the logs

A cluster produces a lot of audit noise: kubelets renewing leases, controllers reconciling, monitoring agents listing pods every few seconds. Reading raw JSON lines is not realistic past a few thousand events.

Drop the export into the Kubernetes audit log analyzer. It normalises raw audit.k8s.io/v1 JSON lines and the EKS, GKE and AKS export formats into one event model, applies 28 reviewed detection rules, and returns:

• a verdict: Compromised when at least one critical rule fires or three different high-severity rules fire, Suspicious when a high rule or two medium rules fire, otherwise Clean;
• findings grouped by identity or image, each with a severity, an ATT&CK technique ID and remediation steps;
• an incident timeline in which the first occurrence of each rule is highlighted;
• an entity pivot (users and service accounts, source IPs, namespaces, pods, images, user agents).

Everything runs in the browser tab with WebAssembly; the files are never uploaded. The step-by-step guide walks through each panel.

A Clean verdict is not proof of safety. It means none of the rules matched in the data you provided, and the data depends heavily on your audit policy.

Step 2: answer the five questions

Which identity?

Start from the finding with the highest severity and note the user.username. A service account (system:serviceaccount:<namespace>:<name>) driven with kubectl/ or curl/ in the user agent, or from a public IP, is almost always a token that left its pod. A human identity doing something unusual needs a phone call to its owner before any conclusion.

Keep in mind that sourceIPs lists proxy headers first; the audit event reference warns that all but the last address can be set by the client.

Since when?

Filter the events on that identity and on its source IPs, then look for the earliest request. Attackers usually run reconnaissance (/version, /api, discovery calls, auth can-i) before anything noisy. If the earliest event is at the start of your export, the export is too short: go back further.

What did it touch?

List every verb and resource for the identity. Secrets read are the most urgent: each one is a credential to rotate, and some (cloud keys, CI tokens, registry credentials) reach outside the cluster. See secrets access and service account token theft.

What did it leave behind?

Look for writes: workloads, CronJobs, DaemonSets, ServiceAccounts, (Cluster)RoleBindings, Roles, webhooks, and new tokens minted via TokenRequest. Each of these is a way back in after you revoke the first credential. The RBAC escalation article covers the binding side, the privileged pods article the node side.

What can it still reach?

A cluster-admin who could exec into a privileged pod could read node credentials, cloud instance roles and every mounted secret. Scope beyond the cluster: the cloud account (CloudTrail, Cloud Audit Logs, Azure Activity Log), the CI system, the registry. For the cloud side, the sibling sites AWS Forensics, GCP Forensics and Azure Forensics cover those logs.

Step 3: contain, eradicate, recover

Order matters, because some steps invalidate others.

1. Cut the attacker's current access. Remove the malicious bindings, delete and recreate compromised service accounts (a token minted with TokenRequest stays valid until expiry unless the service account or its bound object is deleted, as the service account documentation explains), and restrict who can reach the API endpoint.
2. Remove persistence. Delete malicious DaemonSets, CronJobs, Jobs and pods, after saving their specs, and search every namespace for copies.
3. Rebuild nodes that ran privileged or host-mounting pods. Cordon, drain, replace; rotate kubelet and cloud instance credentials.
4. Rotate every secret that was read.
5. Close the entry point: exposed dashboard, leaked kubeconfig, over-privileged token, anonymous access.
6. Harden and monitor: Pod Security Admission on every namespace, least-privilege RBAC, audit logs retained outside the cluster. The NSA/CISA Kubernetes Hardening Guide and the Kubernetes security checklist are good baselines.

The analyzer's remediation checklist follows this order and only lists the steps that relate to the findings you actually have.

Map findings to ATT&CK, not to tools

Writing the report is easier when every finding has a technique ID. The MITRE ATT&CK Containers matrix covers the usual steps: T1609 Container Administration Command for exec, T1610 Deploy Container, T1611 Escape to Host, T1552.007 Container API for secret theft, T1528 Steal Application Access Token, T1098.006 Additional Container Cluster Roles, T1053.007 Container Orchestration Job, T1496 Resource Hijacking. Each detection rule of the analyzer carries these IDs, so the export is report-ready.

A realistic end-to-end example

If you want to see all of this on concrete events, the fictional incident walkthrough follows an exposed dashboard token to cluster-admin and a crypto-miner in 15 minutes of audit events. The same data is behind the "Try a sample" button on the analyzer page.

FAQ

What is the first thing to do when a Kubernetes cluster may be compromised?

Preserve evidence before changing anything: export the API server audit logs for the whole period, extend their retention, and save the specs of suspicious workloads and bindings. Deleting a pod or a binding first destroys context you will need to scope the incident.

Are Kubernetes audit logs enough to investigate an intrusion?

They are the primary record of what was done through the Kubernetes API, but they do not see processes inside containers or on nodes. Pair them with runtime alerts, node and container logs, and your cloud provider's control-plane logs.

How far back should I export audit logs?

Start a few days before the first suspicious event you know of, and extend backwards whenever you find an earlier trace of the same identity or source IP. Initial access often predates the noisy part of an attack.