How Horizontal Scaling Became a Cover-Up for Terrible Code
Ez Eldeen Mushtaha · Engineering ManifestoFirst Principles · 2025
Introduction
One question is enough to embarrass any DevOps engineer or cloud architect:
"Why do you need 50 servers to serve 10,000 users, when a single server in the 1990s served twice that load on a Pentium processor?"
The answer no one wants to hear:
Horizontal scaling is no longer a tool for building large systems. It has become a cover for slow code, bad engineering, and the fear of optimizing a single server.
This article dismantles the scaling illusion and shows why adding 50 Kubernetes nodes does not solve the problem—it multiplies it.
Chapter One · The History of Scaling
When Was Horizontal Scaling Necessary?
In the beginning, horizontal scaling was a real solution to a real problem:
Google (1998): Millions of pages, a single search required thousands of servers because the data was too large for one machine.
Facebook (2004): Hundreds of millions of users, each generating data. Storage and distribution required thousands of nodes.
Amazon (2000s): Millions of products, millions of orders. One system could not handle the load.
In these cases, horizontal scaling was necessary because:
Data exceeded the capacity of a single server.
Load exceeded the capacity of a single server.
Geographic distribution required servers close to users.
These are legitimate use cases for horizontal scaling.
When Did Scaling Become a Cover for Failure?
The shift began when the philosophy became:
"No need to optimize. Just add servers."
This philosophy spread with:
AWS (2006): Adding a new server became easy and cheap.
Docker (2013): Running 100 containers became easier than optimizing one piece of code.
Kubernetes (2014): Managing thousands of containers became the "solution" to performance problems.
The result: Developers stopped optimizing code and started adding servers.
Chapter Two · Why Horizontal Scaling Does Not Solve the Problem
1. Amdahl's Law — The Mathematical Ceiling
This law is the mathematical weapon that destroys the scaling illusion:
S = 1 / ((1 - p) + (p / n))
Where:
S: Theoretical speedup
p: Parallelizable portion of the code
n: Number of processors/servers
(1 - p): Serial portion — this is the bottleneck
What does this mean?
If 10% of your code is serial (not parallelizable):
Servers (n)
Theoretical Speedup (S)
1
1.0×
2
1.8×
4
3.1×
8
4.7×
16
6.4×
32
7.6×
64
8.3×
∞
10.0×
Conclusion: Even with 64 servers, you cannot exceed 10× speedup because 10% of your code is serial.
If your code has:
Locks (Mutex)
Shared database (Database)
Synchronous I/O
Shared state
Then you are bounded by the serial portion. All additional servers will not exceed this limit.
2. Coordination Overhead — The Hidden Tax
When you add servers, you add:
Inter-node Communication: Each node talks to others over the network.
Consistency: Maintaining consistent state across nodes requires complex protocols (e.g., Paxos, Raft).
Load Balancing: Distributing requests across nodes requires an additional layer.
Failure Handling: More nodes = more failures.
Nodes
Coordination Overhead (% of time)
1
0%
2
~5–10%
10
~15–25%
50
~30–50%
100
~50–70%
Result: With 50 nodes, half of the processing time is wasted on coordination between nodes, not on processing actual requests.
3. Network is the Bottleneck
In a distributed system, every request travels over the network:
Client → Load Balancer → Service A → Service B → Service C → Database → Response
Operation
Time
Relative
Local function call
~5–10 ns
1×
Same server (shared memory)
~50–100 ns
~10×
Network call (same datacenter)
~100–500 μs
~50,000×
Multi-hop microservices
~1–10 ms
~1,000,000×
Local Call (Pure POSIX):
┌─────────────────────────────────────────────────────────────┐
│ App ──(function call)──▶ Data ──(return)──▶ App │
│ Time: 10 nanoseconds │
└─────────────────────────────────────────────────────────────┘
Distributed Call (Microservices):
┌─────────────────────────────────────────────────────────────┐
│ App ──(HTTP)──▶ Service A ──(gRPC)──▶ Service B │
│ │ │ │
│ ▼ ▼ │
│ Service C ──(TCP)──▶ Database │
│ │ │ │
│ ▼ ▼ │
│ Response ──────────────▶ App │
│ Time: 2.5 milliseconds │
└─────────────────────────────────────────────────────────────┘
The difference:
Local function call: 10 nanoseconds
Network call: 500 microseconds (50,000× slower!)
The problem: When you split your application into 10 microservices, each request requires 5–10 network calls.
Total response time:
5 calls × 500 microseconds = 2.5ms
Instead of 10 nanoseconds (local call), the latency becomes 2.5ms — 250,000× slower.
4. Cascading Failure
In a distributed system, failure propagates:
Service A slows down
↓
Service B waits for Service A
↓
Service C waits for Service B
↓
Requests pile up everywhere
↓
All services become unresponsive
↓
User sees 500 error
In a monolithic system:
If a service fails, the whole system fails.
But debugging is straightforward.
In a distributed system:
Service A slows down, Service B waits, Service C waits.
Requests pile up everywhere.
It becomes impossible to identify the root cause.
Each team blames the other.
This is not scalability. This is distributed failure.
5. The Thundering Herd Problem
When multiple replicas of a service are behind a load balancer, and a sudden burst of traffic arrives:
Burst of 10,000 requests
↓
Load balancer distributes to 10 replicas
↓
Each replica receives 1,000 requests
↓
All replicas try to handle them simultaneously
↓
Database connections pile up
↓
Database connection pool exhausted
↓
All replicas fail
The irony: More replicas made the problem worse, not better.
6. The Hidden Cost of Orchestration (Kubernetes)
Kubernetes itself consumes resources:
Component
Memory
CPU
API Server
1–4GB
1–4 cores
etcd
1–4GB
1–2 cores
Scheduler
200–500MB
0.5–1 cores
Controller Manager
200–500MB
0.5–1 cores
kubelet (per node)
100–200MB
0.5–1 cores
kube-proxy (per node)
100–200MB
0.5–1 cores
Total (10-node cluster)
~6–12GB
~8–16 cores
That is 6–12GB of RAM and 8–16 CPU cores consumed purely by the orchestration layer—before any application runs.
In a pure POSIX system: Zero orchestration overhead.
Chapter Three · The Alternative — Single-Node Optimization
What Can a Single Server Achieve?
With optimized code, a single server can achieve:
Task
Performance with Optimized Code
HTTP requests (JSON)
50,000–100,000 req/s
Static file serving
200,000–500,000 req/s
In-memory database ops
1,000,000+ ops/s
Message processing
500,000–1,000,000 msg/s
What does this mean?
A single server with optimized code can handle the equivalent of 10–20 servers with unoptimized code.
Cost comparison:
1 optimized server: $100/month
20 unoptimized servers: $2,000/month
Difference: 20× cost for the same performance.
How to Optimize a Single Server
1. Eliminate Dynamic Memory Allocation in the Hot Path
Use memory pools.
Pre-allocate memory.
Avoid malloc() and free() during request processing.
2. Use Non-blocking I/O
Use epoll (Linux) or kqueue (BSD).
Avoid select() and poll() (slow).
Use io_uring for zero-copy.
3. Optimize Cache Usage
Align data structures to cache lines (64 bytes).
Avoid false sharing.
Use SIMD (AVX-512) for data processing.
4. Avoid Locks in the Hot Path
Use lock-free data structures.
Use atomic operations.
Use ring buffers.
5. Use Zero-Copy
Use sendfile() for file transfers.
Use splice() for data transfer between sockets.
Avoid copying data between user space and kernel space.
6. Tune Kernel Parameters
Increase net.core.somaxconn.
Increase net.ipv4.tcp_max_syn_backlog.
Enable TCP_NODELAY.
Increase ulimit -n.
7. CPU Pinning (sched_setaffinity)
Pin processes to specific CPU cores.
Keep L1/L2 cache warm.
Avoid cache thrashing.
8. Real-time Scheduling (PREEMPT_RT)
Use real-time kernel patches.
Guarantee deterministic latency.
Avoid scheduler jitter.
Chapter Four · Comprehensive Comparison
Metric
Horizontal Scaling (Kubernetes/Docker)
Single-Node Optimization (POSIX/C)
Number of servers
20–50
1–2
Monthly cost
$2,000–$10,000
$100–$200
Average latency
10–100ms (unstable)
0.3–1ms (stable)
Peak latency
Unpredictable (100ms–1s)
Predictable (1–5ms)
Operational complexity
Very high
Low
Debugging
Hard (50+ servers)
Easy (1 server)
Failure mode
Cascading
Isolated
Scalability limit
Amdahl's Law ceiling
Hardware limit (one node)
Orchestration overhead
6–12GB RAM, 8–16 cores
0MB, 0 cores
Chapter Five · The Root Cause
1. The Industry Chose Convenience
Easier to add a library than to write code.
Easier to add a server than to optimize.
Easier to add a layer than to understand one.
2. Developers Stopped Learning Fundamentals
Many cannot explain how malloc() works.
Many cannot explain what a system call is.
Many do not know the kernel, the network stack, or the filesystem.
3. Cloud Providers Profit from Inefficiency
AWS, GCP, and Azure make more money when you consume more resources.
Inefficient code = more resource consumption = more revenue.
There is no financial incentive for cloud providers to encourage efficiency.
4. The Philosophy Became "Ship Everything"
Instead of shipping the application, we ship the entire environment.
Instead of solving the dependency problem, we encapsulate it.
Instead of writing clean code, we wrap the garbage and call it "modern."
5. "Good Enough" Became the Standard
"It works, so ship it."
"We can scale later."
"Hardware is cheap."
"We'll fix it when it breaks."
Except it breaks. And when it breaks, it breaks everywhere.
Chapter Six · The Conclusion
The Problem Is Not Kubernetes
Kubernetes is a tool. It solves real problems at massive scale.
The problem is using Kubernetes when you do not need it.
The Problem Is Not Microservices
Microservices are an architectural pattern. They solve organizational problems.
The problem is using microservices when a monolith would work better.
The Problem Is Not Cloud Providers
Cloud providers are businesses. They sell resources.
The problem is that we voluntarily consume more resources than necessary.
The Problem Is the Fear of Fundamentals
We have built an entire industry on the premise that:
Memory management is hard, so let's use garbage collection.
Networking is hard, so let's add layers.
System programming is hard, so let's use interpreted languages.
Scaling is hard, so let's add more servers.
The result is not engineering. It is hiding.
What Is the Solution?
1. Optimize the Single Node First
Before scaling, make sure the single node is fully optimized.
Measure. Profile. Improve.
2. Understand Amdahl's Law
Know your serial portion.
Do not add servers if the serial portion is the bottleneck.
3. Avoid Microservices Unless Necessary
If your team is small, a monolith is often better.
Microservices add complexity, not performance.
4. Learn the Fundamentals
Understand the kernel, memory, network, and filesystem.
Understand what a system call does.
Understand how the scheduler works.
5. Question the Defaults
Do you really need Kubernetes?
Do you really need 50 microservices?
Do you really need to scale horizontally?
Final Words
Horizontal scaling was invented to solve real problems:
Google needed it because one server could not store the internet.
Facebook needed it because one server could not store all user data.
Amazon needed it because one server could not handle all orders.
Your CRUD application does not need 50 servers.
Your simple API does not need Kubernetes.
Your 10,000 users do not need 50 microservices.
Optimize the single node first. Then, if necessary, scale. But only if necessary.
The industry does not need more servers. It needs better code.
It does not need more layers. It needs fewer layers.
It does not need more abstraction. It needs understanding.
Because adding servers does not fix bad code. It just spreads it around.