Using Kubernetes Audit Log to Identify Security Issues

In this post, I’ll explore Kubernetes audit logging and its importance in cloud-native security. I’ll also cover best practices for maintaining secure audit files.

Audit logs fundamentally help in the following areas:

Audit logs records each activity that happens in the cluster. For each activity, it adds metadata such as IP address from which the action was created, user-agent, etc. Using the audit log and the metadata, there are solutions that can look for Indication of Attack (IoA) and define policies. For example, You can create a policy allowing changes to the production cluster only from the organization’s approved IP addresses any action outside of this approved list will raise an alert.

Audit logging provides deep insight into a cluster’s actions and events, so it’s easy to reconstruct a problem if there’s a security incident. Teams can utilize audit trails to understand why, when, and how components of a cluster underperformed during operations. By understanding the conditions that lead to a security incident, security professionals can create enhanced monitoring, damage assessment, and remediation strategies.

Through forensic analysis and real-time alerts, log files help system administrators and security professionals identify malicious user actions and behavior. Audit trails also flag unusual user and bot activities in real-time, thereby helping with intrusion detection and unusual user behavior as they occur. There are solutions that use UEBA (User and Entity Behavior Analysis) in order to identify abnormal activity. For example, a new user is creating a lot of objects, the DevOps manager logs into the system from an abnormal location.

Kubernetes audit records are generated by the kube-apiserver component. Every client request generates an audit event, which is processed using an audit policy then written to the backend. Below is an outline of important fields covered in the audit log.

The audit log primarily records transactions between the Kubernetes API server and end-users. As a server processes client requests, it sends certain information to the log file, including:

Audit logs capture important account activities and information, such as:

In Kubernetes, you need to pass the API server the -audit-policy-file flag in order for the audit policy to be enforced. Policy is an object that defines the rules of events to be logged and what data the records should include. Once an event is logged, Kubernetes compares its characteristics against the list of rules. A sample audit policy specification would look similar to the following:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
— “RequestReceived”
rules:
— level: RequestResponse
resources:
— group: “”
resources: [“pods”]
— level: Metadata
resources:
— group: “”
resources: [“pods/log”, “pods/status”]

— level: None
resources:
— group: “”
resources: [“configmaps”]
resourceNames: [“controller-leader”]

— level: None
users: [“system:kube-proxy”]
verbs: [“watch”]
resources:
— group: “”
resources: [“endpoints”, “services”]

— level: None
userGroups: [“system:authenticated”]
nonResourceURLs:
— “/api*”
— “/version”

— level: Request
resources:
— group: “”
resources: [“configmaps”].
namespaces: [“kube-system”]

— level: Metadata
resources:
— group: “”
resources: [“secrets”, “configmaps”]

— level: Request
resources:
— group: “”
— group: “extensions”

— level: Metadata
omitStages:
— “RequestReceived”

Saving Logs

Kubernetes gives two options for saving the audit log:

If you are saving the audit log to the local filesystem, you need to pass the following to the API server flags:

If you are sending the audit logs to a 3 rdparty system, you need to pass the following to the API server flags:

Optimizing Audit Logs

You can define that the K8s API server will buffer the audit logs before saving/streaming it, you can define the buffer size, the batch size, the time the API server will wait before unconditionally batch events in the queue, batches per second, and in case of 3 rdparty system, the throttling burst (number of batches generated at the same moment).

There might be a case, where your API server receives a lot of requests per second and needs to handle and save/transmit a large number of records. You don’t want to define the audit log configuration parameters and cause logs to disappear due to a burst of requests which the API server can’t handle. The API server supports metrics to measure how often this happens. You can use these metrics in order to set the parameters rightfully.

Logs are only helpful if they are secure and untampered. A Kubernetes audit log becomes less effective if the information it records can be deleted or altered. As logs are essentially JSON files, they are commonly susceptible to theft, alteration, or corruption. Some practices that organizations can embrace to protect log files include:

Attackers target log files to keep their activities undetected. As a best practice, it’s important to record logs on a remote server making it harder for hackers to access. Use the webhook option to stream the audit logs records to a 3 rdparty solution that will not only store the records remotely, as required by some compliance frameworks but also will protect it., by adding security — policy, threat detection, abnormal activity detection, and incident response capabilities.

Kubernetes API server can audit all the requests it gets. Audit logging helps organizations implement visibility for these ecosystems, enabling regulatory compliance and security. You can use it as another security layer as it is not intrusive and does not affect the performance of your cluster and applications.

Related posts:

Find out how often you should water Rockwool seeding. Tips on how to grow seedlings successfully with hydroponics.

Sentada en el asiento del bus viendo por un pequeño reflejo empañado de suciedad y música de 103.5. Tomando con mi mano el libro del gran Saramago. Comencé a jugar con mi vista, cuando los cerraba…

Soy Erick Estrada y me encanta escribir y redactar cosas.. “Hola soy Erick Estrada.” is published by Erick Javier Estrada Ramón.