Kubernetes Security Best Practices
Kubernetes Security Best Practices
A comprehensive security guide for safely operating Kubernetes in production environments. This guide covers practical security hardening techniques including RBAC configuration, network policies, Pod security, secrets management, and more.
Overview
Kubernetes has become an essential foundation for modern containerized applications, but its complex distributed architecture creates diverse security challenges. This article provides detailed guidance on implementing robust security in production environments, including the latest security features in Kubernetes 1.30 and beyond.
The Importance of Kubernetes Security
Modern Threat Landscape
In 2024, over 350 Kubernetes clusters belonging to organizations, open-source projects, and individuals were exposed due to misconfigurations, putting sensitive data and critical applications at risk. This incident demonstrates that Kubernetes security represents real-world threats, not just theoretical risks.
Why Security Matters
- Protecting Sensitive Data: Container environments contain sensitive code, system credentials, and customer data
- Complex Attack Surface: Complexity from distributed systems, microservices, and extensive configurability
- Regulatory Compliance: Legal requirements like GDPR, HIPAA, and CCPA
- Service Availability: Continuous operation of high-availability applications
- Financial & Reputation Risk Mitigation: Preventing losses from security incidents
Kubernetes Security Fundamentals
Defense in Depth Strategy
Kubernetes security requires a comprehensive approach consisting of multiple layers:
- Infrastructure Layer: Nodes, network, storage
- Cluster Layer: Control plane, etcd, API Server
- Workload Layer: Pods, containers, applications
- Application Layer: Code, runtime, data
Zero Trust Model
A principle of never trusting any component and always verifying, authenticating, and authorizing.
1. Pod Security Standards (PSS) - New Features in Kubernetes 1.30+
Overview of Pod Security Standards
Pod Security Standards have been significantly enhanced in Kubernetes 1.30 and later.
Three Security Levels
# 1. Privileged - No restrictions (avoid in production)
# 2. Baseline - Minimal restrictions
# 3. Restricted - Strict security controls
Namespace-Level Pod Security Standards Configuration
apiVersion: v1
kind: Namespace
metadata:
name: production-apps
labels:
# Enforce baseline, warn and audit on restricted
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/enforce-version: v1.30
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: v1.30
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: v1.30
Cluster-Wide Pod Security Configuration
# cluster-level-pss.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
defaults:
enforce: "baseline"
enforce-version: "latest"
audit: "restricted"
audit-version: "latest"
warn: "restricted"
warn-version: "latest"
exemptions:
usernames: []
runtimeClasses: []
namespaces: [kube-system]
Restricted Pod Implementation Example
apiVersion: v1
kind: Pod
metadata:
name: secure-app
namespace: production-apps
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: myapp:latest
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
seccompProfile:
type: RuntimeDefault
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "256Mi"
cpu: "250m"
volumeMounts:
- name: tmp
mountPath: /tmp
- name: app-data
mountPath: /app/data
readOnly: true
volumes:
- name: tmp
emptyDir: {}
- name: app-data
configMap:
name: app-config
Kubernetes 1.30 New Feature: User Namespaces
# User Namespace enabled Pod (Alpha feature)
apiVersion: v1
kind: Pod
metadata:
name: user-namespace-pod
spec:
hostUsers: false # Enable User Namespaces
securityContext:
runAsUser: 1000
runAsGroup: 3000
containers:
- name: app
image: busybox
command: ["sleep", "infinity"]
securityContext:
runAsNonRoot: true
2. Role-Based Access Control (RBAC)
RBAC Basic Principles
Based on the principle of least privilege, granting only the minimum access rights necessary.
Service Account Creation
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-service-account
namespace: production-apps
automountServiceAccountToken: false # Disable auto-mounting
Role Definition
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production-apps
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
resourceNames: ["app-config"] # Specific resources only
RoleBinding Configuration
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-binding
namespace: production-apps
subjects:
- kind: ServiceAccount
name: app-service-account
namespace: production-apps
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
ClusterRole Implementation (Cluster-wide Resources)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: monitoring-reader
rules:
- apiGroups: [""]
resources: ["nodes", "nodes/metrics", "services", "endpoints", "pods"]
verbs: ["get", "list", "watch"]
- apiGroups: ["metrics.k8s.io"]
resources: ["nodes", "pods"]
verbs: ["get", "list"]
RBAC Auditing and Best Practices
# Check current RBAC permissions
kubectl auth can-i --list --as=system:serviceaccount:production-apps:app-service-account
# Check specific operation permissions
kubectl auth can-i create pods --as=system:serviceaccount:production-apps:app-service-account
# RBAC analysis (using kubectl-who-can plugin)
kubectl who-can create pods
kubectl who-can get secrets --all-namespaces
3. Network Policies
Default Deny Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production-apps
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Layered Network Security
# Allow frontend to backend communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-access-policy
namespace: production-apps
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
- to: [] # For DNS resolution
ports:
- protocol: UDP
port: 53
External Connection Control
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: external-access-policy
namespace: production-apps
spec:
podSelector:
matchLabels:
app: api-gateway
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/8 # Internal network
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32 # Block metadata service
- 127.0.0.0/8 # Block localhost
ports:
- protocol: TCP
port: 443 # HTTPS only
4. Secrets Management
Kubernetes Secrets Best Practices
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
namespace: production-apps
annotations:
# Facilitate secret discovery
secret-type: "database-credentials"
type: Opaque
data:
# Base64 encoded
username: <base64-encoded-username>
password: <base64-encoded-password>
Secret Encryption in etcd
# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <32-byte-base64-encoded-key>
- identity: {}
External Secrets Operator
# Integration with external secret management systems
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: production-apps
spec:
provider:
vault:
server: "https://vault.company.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "app-role"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secret
namespace: production-apps
spec:
refreshInterval: 15s
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: app-secrets
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: database
property: password
Secrets Usage Best Practices
apiVersion: v1
kind: Pod
metadata:
name: app-with-secrets
spec:
containers:
- name: app
image: myapp:latest
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: password
volumeMounts:
- name: secret-volume
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: app-secrets
defaultMode: 0400 # Read-only
items:
- key: certificate
path: tls.crt
5. Container Image Security
Vulnerability Scanning with Trivy
# Image vulnerability scanning
trivy image myapp:latest
# Show HIGH and CRITICAL severities only
trivy image --severity HIGH,CRITICAL myapp:latest
# JSON format output
trivy image --format json myapp:latest > scan-results.json
# Kubernetes cluster scanning
trivy k8s cluster
Image Signing with Cosign
# Sign images
cosign sign --key cosign.key myregistry/myapp:latest
# Verify signatures
cosign verify --key cosign.pub myregistry/myapp:latest
# Keyless signing (OIDC)
cosign sign myregistry/myapp:latest
# Automatic verification with Admission Controller
Secure Base Images
# Use distroless images
FROM gcr.io/distroless/java:11
# Avoid privileged users
FROM node:18-alpine
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001
USER nextjs
# Multi-stage builds
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
FROM gcr.io/distroless/nodejs:18
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
USER 1001
CMD ["dist/index.js"]
Image Policy with Admission Controllers
# OPA Gatekeeper ConstraintTemplate
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: allowedrepositories
spec:
crd:
spec:
names:
kind: AllowedRepositories
validation:
properties:
repos:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package allowedrepositories
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not starts_with(container.image, input.parameters.repos[_])
msg := sprintf("Container image %v is not from allowed repository", [container.image])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: AllowedRepositories
metadata:
name: must-come-from-trusted-registry
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
repos:
- "myregistry.company.com/"
- "gcr.io/my-project/"
6. Runtime Security
Threat Detection with Falco
# Falco DaemonSet configuration
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: falco
namespace: falco-system
spec:
selector:
matchLabels:
app: falco
template:
metadata:
labels:
app: falco
spec:
serviceAccount: falco
hostNetwork: true
hostPID: true
containers:
- name: falco
image: falcosecurity/falco:latest
securityContext:
privileged: true
volumeMounts:
- name: dev
mountPath: /host/dev
- name: proc
mountPath: /host/proc
readOnly: true
- name: boot
mountPath: /host/boot
readOnly: true
- name: lib-modules
mountPath: /host/lib/modules
readOnly: true
- name: usr
mountPath: /host/usr
readOnly: true
- name: etc
mountPath: /host/etc
readOnly: true
volumes:
- name: dev
hostPath:
path: /dev
- name: proc
hostPath:
path: /proc
- name: boot
hostPath:
path: /boot
- name: lib-modules
hostPath:
path: /lib/modules
- name: usr
hostPath:
path: /usr
- name: etc
hostPath:
path: /etc
Custom Falco Rules
# Custom security rules
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-custom-rules
namespace: falco-system
data:
custom_rules.yaml: |
- rule: Suspicious shell in container
desc: Shell spawned in a container other than those in a whitelist
condition: >
spawned_process and container and
shell_procs and
not proc.pname in (docker_binaries) and
not container.image.repository in (allowed_images)
output: >
Shell spawned in container (user=%user.name container_id=%container.id
container_name=%container.name shell=%proc.name parent=%proc.pname
cmdline=%proc.cmdline image=%container.image.repository)
priority: WARNING
tags: [container, shell, mitre_execution]
- list: allowed_images
items: [
"alpine/git",
"busybox",
"debug-tools"
]
- rule: Unauthorized process in production
desc: Unexpected process started in production namespace
condition: >
spawned_process and k8s.ns.name=production-apps and
not proc.name in (allowed_processes)
output: >
Unauthorized process in production (user=%user.name proc=%proc.name
parent=%proc.pname cmdline=%proc.cmdline container=%container.name
image=%container.image.repository)
priority: CRITICAL
- list: allowed_processes
items: [
"node",
"nginx",
"java",
"python"
]
Security Response Automation
# Automatic response with Falco Sidekick
apiVersion: apps/v1
kind: Deployment
metadata:
name: falco-sidekick
namespace: falco-system
spec:
replicas: 1
selector:
matchLabels:
app: falco-sidekick
template:
metadata:
labels:
app: falco-sidekick
spec:
containers:
- name: falco-sidekick
image: falcosecurity/falco-sidekick:latest
env:
- name: SLACK_WEBHOOK_URL
valueFrom:
secretKeyRef:
name: alerting-config
key: slack-webhook
- name: KUBELESS_NAMESPACE
value: "kubeless"
- name: KUBELESS_FUNCTION
value: "security-response"
ports:
- containerPort: 2801
7. Service Mesh Security
mTLS with Istio
# Enforce mTLS with PeerAuthentication
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: production-apps
spec:
mtls:
mode: STRICT # Enforce mTLS for all communications
---
# Port-specific mTLS configuration
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: api-server
namespace: production-apps
spec:
selector:
matchLabels:
app: api-server
portLevelMtls:
8080:
mode: STRICT
9090:
mode: PERMISSIVE # Gradual migration for metrics port
Authorization Policies
# Fine-grained authorization control
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: frontend-to-backend
namespace: production-apps
spec:
selector:
matchLabels:
app: backend
rules:
- from:
- source:
principals: ["cluster.local/ns/production-apps/sa/frontend-sa"]
- to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/v1/*"]
- when:
- key: request.headers[authorization]
values: ["Bearer *"]
---
# API rate limiting
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: rate-limit-policy
namespace: production-apps
spec:
selector:
matchLabels:
app: api-gateway
rules:
- to:
- operation:
methods: ["POST"]
paths: ["/api/auth/*"]
when:
- key: source.ip
notValues: ["10.0.0.0/8"] # Exclude internal IPs
Gateway and VirtualService Security
# Secure Gateway configuration
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: secure-gateway
namespace: production-apps
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: api-tls-secret
hosts:
- api.company.com
- port:
number: 80
name: http
protocol: HTTP
hosts:
- api.company.com
tls:
httpsRedirect: true # Redirect HTTP to HTTPS
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-routes
namespace: production-apps
spec:
hosts:
- api.company.com
gateways:
- secure-gateway
http:
- match:
- headers:
user-agent:
regex: ".*bot.*" # Block bot access
fault:
abort:
percentage:
value: 100
httpStatus: 403
- match:
- uri:
prefix: "/api/v1/"
route:
- destination:
host: backend-service
port:
number: 8080
8. Auditing and Compliance
Kubernetes Audit Logs
# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log sensitive operations in detail
- level: RequestResponse
resources:
- group: ""
resources: ["secrets", "configmaps"]
- group: "rbac.authorization.k8s.io"
resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
# Log authentication failures
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources: ["*"]
namespaces: ["production-apps"]
# Monitor exec/attach operations
- level: Request
verbs: ["create"]
resources:
- group: ""
resources: ["pods/exec", "pods/attach", "pods/portforward"]
# Monitor certificate requests
- level: Request
resources:
- group: "certificates.k8s.io"
resources: ["certificatesigningrequests"]
CIS Kubernetes Benchmark Auditing
# CIS compliance check with kube-bench
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
# Check results
kubectl logs job/kube-bench
# Comprehensive assessment with Kubescape
kubescape scan --submit --account-id YOUR_ACCOUNT_ID
# Assessment with specific frameworks
kubescape scan framework nsa,mitre
# YAML configuration file scanning
kubescape scan *.yaml
Continuous Policy Management with OPA Gatekeeper
# ConstraintTemplate to enforce resource limits
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: containerlimits
spec:
crd:
spec:
names:
kind: ContainerLimits
validation:
properties:
cpu:
type: string
memory:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package containerlimits
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.cpu
msg := "Container must have CPU limits"
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.memory
msg := "Container must have memory limits"
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: ContainerLimits
metadata:
name: must-have-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
excludedNamespaces: ["kube-system", "kube-public"]
parameters:
cpu: "1000m"
memory: "1Gi"
9. Continuous Security Monitoring
Security Metrics with Prometheus + Grafana
# ServiceMonitor for security-related metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: falco-metrics
namespace: monitoring
spec:
selector:
matchLabels:
app: falco
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
# Alert rules
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: security-alerts
namespace: monitoring
spec:
groups:
- name: security
rules:
- alert: PrivilegedContainerRunning
expr: increase(falco_events_total{rule_name="Run shell untrusted"}[5m]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "Privileged container detected"
description: "Falco detected a privileged container running"
- alert: FailedAuthenticationSpike
expr: rate(apiserver_audit_total{verb="create",objectRef_resource="tokenreviews"}[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "High rate of authentication failures"
description: "Authentication failure rate is above normal threshold"
Security Automation and Response
#!/bin/bash
# security-automation.sh
# Automatically isolate pods with critical vulnerabilities
kubectl get pods --all-namespaces -o json | \
jq -r '.items[] | select(.metadata.annotations."vulnerability-scan" == "critical") | "\(.metadata.namespace) \(.metadata.name)"' | \
while read namespace pod; do
echo "Isolating critical vulnerability pod: $namespace/$pod"
kubectl label pod $pod -n $namespace security.isolation=true
kubectl annotate pod $pod -n $namespace network-policy.isolation=deny-all
done
# Detect abnormal network activity
kubectl logs -n falco-system daemonset/falco | \
grep "Outbound or inbound traffic not to or from" | \
tail -10 | \
while read line; do
echo "Suspicious network activity detected: $line"
# Send Slack or email notifications
done
10. Incident Response and Forensics
Security Incident Response Playbook
# Emergency namespace for incident response
apiVersion: v1
kind: Namespace
metadata:
name: incident-response
labels:
security-level: high
pod-security.kubernetes.io/enforce: restricted
---
# Forensic pod
apiVersion: v1
kind: Pod
metadata:
name: forensic-toolkit
namespace: incident-response
spec:
hostNetwork: false
hostPID: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- name: forensic-tools
image: forensic-toolkit:latest
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
volumeMounts:
- name: evidence
mountPath: /evidence
- name: tmp
mountPath: /tmp
volumes:
- name: evidence
emptyDir: {}
- name: tmp
emptyDir: {}
Compromised Pod Isolation
# Immediate isolation of compromised pods
isolate_compromised_pod() {
local namespace=$1
local pod_name=$2
echo "Isolating compromised pod: $namespace/$pod_name"
# Network isolation
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: isolate-$pod_name
namespace: $namespace
spec:
podSelector:
matchLabels:
app: $pod_name
policyTypes:
- Ingress
- Egress
EOF
# Labeling and annotation
kubectl label pod $pod_name -n $namespace security.status=compromised
kubectl annotate pod $pod_name -n $namespace security.incident-id=$(date +%s)
# Evidence preservation
kubectl logs $pod_name -n $namespace > /evidence/logs-$pod_name-$(date +%s).log
kubectl describe pod $pod_name -n $namespace > /evidence/describe-$pod_name-$(date +%s).yaml
}
Summary
Kubernetes security is not just a configuration issue—it's a continuous process. By combining the best practices introduced in this article, you can expect the following outcomes:
Key Performance Indicators
- Reduced Incident Rate: Proper security controls can reduce security incidents by 70-80%
- Shortened Mean Detection Time: Runtime monitoring can reduce threat detection time from hours to minutes
- Compliance Adherence: Automated policy management can reduce audit preparation time by 90%
- Improved Development Velocity: Automated security checks maintain development speed while strengthening security
Implementation Roadmap
- Phase 1 (1-2 weeks): Implement Pod Security Standards, basic RBAC, and network policies
- Phase 2 (3-4 weeks): Configure image scanning, secrets management, and audit logs
- Phase 3 (1-2 months): Deploy runtime security, Service Mesh, and OPA policy management
- Phase 4 (Ongoing): Expand continuous monitoring, incident response, and security automation
Key Success Factors
- Gradual Implementation: Don't implement everything at once; apply high-priority items first
- Developer Collaboration: Integrate security into development processes, not as an afterthought
- Continuous Improvement: Continuously update security measures as threats evolve
- Education and Training: Improve security awareness and skills across the entire team
Kubernetes security is complex, but with a systematic approach and proper tools, you can build a robust and secure production environment. Remember that security is not a one-time setup but a continuous process requiring ongoing monitoring and improvement.