Detect kubectl exec, attach and port-forward in Audit Logs

TL;DR. kubectl exec and kubectl attach are requests on the pods/exec and pods/attach subresources; kubectl port-forward hits pods/portforward. Match on objectRef.subresource, accept both create and get as verbs, expect response code 101, and read the command from the command= parameters in requestURI. Exec by a service account is rare and should be treated as high severity. Access to nodes/proxy reaches the kubelet directly and is worse. Nothing typed inside the session is logged.

Running a command in a container is ATT&CK technique T1609, Container Administration Command. It is also the most common thing a legitimate administrator does during an outage. The job is not to flag every exec, but to make every exec explainable.

What exec looks like in the audit log

kubectl exec -it api-6c9f7d8b5-q2w8x -n shop -- sh produces an event like this (trimmed, fictional names):

{
  "verb": "create",
  "stage": "ResponseStarted",
  "requestURI": "/api/v1/namespaces/shop/pods/api-6c9f7d8b5-q2w8x/exec?command=sh&container=api&stdin=true&stdout=true&tty=true",
  "user": { "username": "alice@example.com" },
  "objectRef": { "resource": "pods", "namespace": "shop", "name": "api-6c9f7d8b5-q2w8x", "subresource": "exec" },
  "responseStatus": { "code": 101 }
}

Points to know:

• Verb. The exec endpoint has traditionally been authorised and audited as create. Since Kubernetes 1.31, kubectl streams over WebSockets by default, and a WebSocket upgrade starts as an HTTP GET, so depending on versions you may see get. Search for both.
• Stage. Exec is long-running. The ResponseStarted event is written when the session opens; ResponseComplete only when it closes. If the audit policy omits nothing, you get both.
• Response code. 101 Switching Protocols means the session was established. A 403 means someone tried and RBAC said no, which is interesting in itself.
• Command. Each argument is a separate command= parameter, URL-encoded: command=chroot&command=%2Fhost&command=sh is chroot /host sh. It is in the URI, so it is logged even at Metadata level.

See the anatomy of an audit event for the other fields.

Attach, cp, debug and port-forward

kubectl cp is worth a separate look: copying a file into a pod is how tools get dropped without pulling a new image, and copying out is data exfiltration through the API server.

kubectl port-forward opens a tunnel from the client's machine into the pod network. It bypasses ingress controllers and, for the client's traffic, network policies that only filter pod-to-pod traffic. A port-forward to a database pod by an identity that never does it is a strong lead.

nodes/proxy: exec without an exec event

The nodes/proxy subresource lets a caller talk to the kubelet API through the API server. The kubelet can run commands in any container on that node and read its logs. The API server audits the nodes/proxy request, but not as a pods/exec, so exec-based detections miss it. The Kubernetes RBAC good practices list access to the node proxy subresource as a privilege escalation path.

Worse, anyone who can reach the kubelet port on a node with valid credentials talks to it directly: those calls never reach the API server and never appear in the audit log. Kubelet authentication and authorisation settings are the control there.

Telling admin work from abuse

The analyzer on the home page splits exec into four rules:

Why the split? Workloads almost never exec into other pods. When a service account does, it is usually a token taken from a pod and replayed by a person, often with a kubectl/ user agent from an unexpected IP, which two other rules catch (see service account token theft). CI systems and some operators do exec legitimately; allow-list them explicitly after checking.

For human identities, triage each session with four questions:

1. Who and from where? Known admin, known IP range, normal user agent?
2. When? During a change window or incident, or at an odd hour?
3. Where? Which namespace and pod; is it a privileged pod or a kube-system component?
4. What? The command: sh/bash interactive, or something like cat /var/run/secrets/kubernetes.io/serviceaccount/token, curl … | sh, chroot /host, nsenter?

A chroot /host or nsenter --target 1 inside a pod with a hostPath of / or hostPID is a container escape; see privileged pods and container escape.

Hunting queries

Raw JSON lines with jq:

jq -c 'select(.objectRef.subresource == "exec" or .objectRef.subresource == "attach"
              or .objectRef.subresource == "portforward")
       | {t: .requestReceivedTimestamp, user: .user.username, ip: .sourceIPs,
          ns: .objectRef.namespace, pod: .objectRef.name,
          sub: .objectRef.subresource, uri: .requestURI, code: .responseStatus.code}' audit.log

EKS, CloudWatch Logs Insights:

fields @timestamp, user.username, sourceIPs.0, objectRef.namespace, objectRef.name, requestURI
| filter @logStream like "kube-apiserver-audit"
| filter objectRef.subresource in ["exec", "attach", "portforward"]
| sort @timestamp desc

AKS, Log Analytics (resource-specific table):

AKSAudit
| where RequestUri has "/exec" or RequestUri has "/attach" or RequestUri has "/portforward"
| project TimeGenerated, User, SourceIps, UserAgent, RequestUri, ResponseStatus

On GKE, filter Cloud Logging on protoPayload.methodName:"pods.exec" (and pods.attach, pods.portforward).

Reduce the attack surface

• Restrict pods/exec, pods/attach and pods/portforward to a small break-glass group; they are separate RBAC resources, so you can grant get on pods without them.
• Do not grant nodes/proxy to people or workloads.
• Alert on every exec by a service account, and on every exec into kube-system.
• Keep the audit log: exec sessions are short and the pod may be gone tomorrow.

FAQ

How do I detect kubectl exec in Kubernetes audit logs?

Look for events whose objectRef.subresource is exec (or attach) on the pods resource. Depending on client and server versions the verb is create or get, and the response code is 101 for a successful streaming upgrade. The command is in requestURI as repeated command= parameters.

Can I see what was typed inside a kubectl exec shell?

No. The audit log records the request that opened the session and its initial command, not the bytes exchanged afterwards. Keystrokes and processes inside the container need runtime tooling such as Falco or an EDR on the node.

Does kubectl cp show up in audit logs?

Yes, as an exec. kubectl cp runs tar inside the container through pods/exec, so the audit event shows command=tar with its arguments in requestURI.

Related articles

• Secrets access and service account token theft
• Privileged pods and container escape indicators
• How to analyze Kubernetes audit logs in your browser