Why Kubernetes Clusters Get Expensive
Kubernetes gives teams flexibility, but it also makes it easy to spend more than necessary. The most common reason is overprovisioning. Teams set high resource requests because they want to avoid incidents, but those requests reserve capacity even when workloads do not use it. The cluster then needs more nodes than the application actually requires.
Idle nodes are another major cost driver. A cluster may scale up during a traffic spike, batch job, or deployment, then remain larger than needed because autoscaling is not configured correctly. In some environments, nodes run for weeks with low CPU and memory utilization because nobody owns the cleanup process.
Persistent volumes can also become expensive. Teams create large volumes "just in case," keep old volumes after workloads are deleted, or use premium storage classes for workloads that do not need high performance. Container registries add hidden cost when old images accumulate across branches, environments, and releases.
Kubernetes cost control is not about making everything smaller. It is about matching capacity to real usage while protecting reliability.
Step 1: Understand Where Your Money Goes
Before changing anything, find the top cost drivers. Start with kubectl top to review live CPU and memory usage across nodes and pods. It will not give you the full financial picture, but it quickly shows which workloads are using resources and which are mostly idle.
The Kubernetes Dashboard or observability tools such as Prometheus and Grafana can show usage over time. That historical view matters because a pod that looks idle at noon may be busy at night during batch processing. Cost optimization should be based on patterns, not a single snapshot.
Cloud cost explorer tools are the next layer. AWS Cost Explorer, GCP Billing, Azure Cost Management, and third-party tools can show which node groups, volumes, load balancers, NAT gateways, and registries are driving the bill. The goal is to identify the top five cost drivers first. Optimizing small items may feel productive, but the biggest savings usually come from compute, storage, and data transfer.
Once the top drivers are clear, map them to workloads and teams. Cost ownership improves when engineers can see which services create which expenses.
Step 2: Right-Size Resource Requests
Requests and limits are often misunderstood. A request is the amount of CPU or memory Kubernetes reserves for a container when scheduling it. A limit is the maximum the container can use. If requests are too high, the scheduler spreads workloads across more nodes than necessary. If limits are too low, applications can throttle or crash.
Right-sizing starts with actual usage data. Look at p50, p90, and p95 CPU and memory usage over a representative period. For stable services, requests can usually be set near normal usage with room for expected spikes. For bursty services, combine careful requests with autoscaling rather than reserving peak capacity all day.
Memory needs special care because memory exhaustion can kill containers. CPU is usually more forgiving because throttling slows the service instead of immediately crashing it. Set memory requests and limits with enough margin, then monitor restarts and latency after changes.
The best process is incremental. Adjust a few workloads, deploy, observe, and repeat. Do not rewrite every request in the cluster in one afternoon.
Step 3: Implement Horizontal Pod Autoscaling
Horizontal Pod Autoscaling lets Kubernetes adjust the number of pod replicas based on demand. CPU-based autoscaling is the simplest option and works well for many web services. If average CPU passes the configured target, Kubernetes adds replicas. When usage drops, it scales down.
Custom metrics are better for some workloads. Queue depth, requests per second, response time, active sessions, or job backlog may represent load more accurately than CPU. For example, a worker service should often scale based on queue length rather than processor usage.
Sensible minimum and maximum replicas matter. The minimum should handle normal baseline traffic and protect availability during node disruption. The maximum should prevent runaway cost during abnormal spikes or metric bugs. Autoscaling should be paired with alerts so teams know when workloads frequently hit max replicas.
Autoscaling is not a substitute for efficient code, but it prevents teams from paying for peak capacity during quiet periods.
Step 4: Use Spot/Preemptible Nodes for Non-Critical Workloads
Spot or preemptible nodes can reduce compute cost significantly, but they can be interrupted. They are useful for non-critical workloads such as batch jobs, preview environments, CI runners, data processing, and stateless workers that can retry safely.
They are risky for stateful databases, critical production APIs, or workloads that cannot tolerate sudden interruption. If spot nodes are used in production, workloads need disruption budgets, retry behavior, multiple replicas, and scheduling rules that keep critical services on stable nodes.
A common pattern is to create separate node pools: on-demand nodes for critical workloads and spot nodes for interruptible workloads. Taints, tolerations, and node selectors keep the right pods in the right place.
Step 5: Clean Up What You Are Not Using
Cleanup is boring, but it saves money. Orphaned persistent volume claims can keep storage running after applications are deleted. Unused namespaces often contain services, secrets, jobs, and volumes that nobody remembers. Old load balancers and ingress resources can continue generating charges.
Container registries should have retention policies. Keeping every image forever creates storage cost and makes incident response harder because nobody knows which images matter. Preview environments should expire automatically. Old Helm releases, completed jobs, and unused secrets should be reviewed regularly.
Create a monthly cleanup checklist. The process does not need to be complex. It needs ownership.
Real Numbers from a Client Engagement
In one engagement with a Berlin-based startup, the Kubernetes bill was about $8,400 per month. The product was stable, but the infrastructure had grown through years of urgent releases. Resource requests were set far above real usage, several namespaces were no longer active, and every workload ran on on-demand nodes.
We started by identifying the top cost drivers, then right-sized the busiest services based on 30 days of usage. We added horizontal pod autoscaling to web and worker workloads, moved non-critical jobs to spot nodes, reduced oversized persistent volumes, and cleaned old environments.
The monthly bill dropped to about $4,900 without reducing production reliability. The important lesson was not a single trick. It was the process: measure, change carefully, monitor, and repeat.