Splunk unknown experimental spl

Kubernetes Process with Resource Ratio Anomalies

The following analytic detects anomalous changes in resource utilization ratios for processes running on a Kubernetes node. It leverages process metrics collected via an OTEL collector and hostmetrics receiver, analyzed through Splunk Observability Cloud. The detection uses a lookup table containing average and standard deviation values for various resource ratios (e.g., CPU:memory, CPU:disk operations). Significant deviations from these baselines may indicate compromised processes, malicious activity, or misconfigurations. If confirmed malicious, this could signify a security breach, allowing attackers to manipulate workloads, potentially leading to data exfiltration or service disruption.

View Source

Detection Logic

| mstats avg(process.*) as process.* where `kubernetes_metrics` by host.name k8s.cluster.name k8s.node.name process.executable.name span=10s
| eval cpu:mem = 'process.cpu.utilization'/'process.memory.utilization'
| eval cpu:disk = 'process.cpu.utilization'/'process.disk.operations'
| eval mem:disk = 'process.memory.utilization'/'process.disk.operations'
| eval cpu:threads = 'process.cpu.utilization'/'process.threads'
| eval disk:threads = 'process.disk.operations'/'process.threads'
| eval key = 'k8s.cluster.name' + ":" + 'host.name' + ":" + 'process.executable.name'
| lookup k8s_process_resource_ratio_baseline key
| fillnull
| eval anomalies = ""
| foreach stdev_* [ eval anomalies =if( '<<MATCHSTR>>' > ('avg_<<MATCHSTR>>' + 4 * 'stdev_<<MATCHSTR>>'), anomalies + "<<MATCHSTR>> ratio higher than average by " + tostring(round(('<<MATCHSTR>>' - 'avg_<<MATCHSTR>>')/'stdev_<<MATCHSTR>>' ,2)) + " Standard Deviations. <<MATCHSTR>>=" + tostring('<<MATCHSTR>>') + " avg_<<MATCHSTR>>=" + tostring('avg_<<MATCHSTR>>') + " 'stdev_<<MATCHSTR>>'=" + tostring('stdev_<<MATCHSTR>>') + ", " , anomalies) ]
| eval anomalies = replace(anomalies, ",\s$", "")
| where anomalies!=""
| stats count values(anomalies) as anomalies by host.name k8s.cluster.name k8s.node.name process.executable.name
| where count > 5
| rename host.name as host
| `kubernetes_process_with_resource_ratio_anomalies_filter`

False Positives

⚠ No false positives have been identified at this time.

Field Validations

Loading…

Comments (0)

Loading comments...