The Containers Fallacy

How a Joke Became an Industry Standard
Ez Eldeen Mushtaha · Engineering Manifesto First Principles · 2025

Introduction

Fifteen years ago, there was a joke among engineers:

"The code doesn't work on the server!"
"Strange! It works fine on my machine!"
"Then take your machine and give it to the customer!"

Everyone laughed. The implication was clear: a developer who cannot manage their environment, does not understand dynamic linking, and does not know how the kernel works, is someone who needs to return to fundamentals.

Then came the cloud era.

The industry had a choice:

They chose option two.

Containers are not a solution. They are a workaround. They do not fix the problem—they encapsulate it, and ship it to production.

This article dissects the layers of abstraction we have built, the cost of each layer, why the industry's obsession with "adding layers" is not engineering, and why the "isolation" that containers claim to provide is an illusion.


Chapter One · The Five Layers of the Disaster

Layer One · The Networking Disaster

What Happens in Pure POSIX?

In a normal POSIX system, the application opens a socket and talks directly to the kernel network stack.

Application → Kernel Network Stack → Physical NIC

One hop. Zero translation. Zero overhead.

The application calls socket(), bind(), listen(), accept(), and recv()/send(). These are system calls. They go straight to the kernel. The kernel manages the TCP/IP stack, routing, and the NIC driver directly.

What Happens in Docker?

To run your application with -p 8080:80, every packet travels through:

Application → veth pair → docker0 bridge → iptables DNAT → conntrack → Kernel Network Stack → Physical NIC

That is six hops. Each hop has a cost.

Sub-Layer 1A: veth pair and Context Switching

Each container runs in its own network namespace. To connect this namespace to the host, the kernel creates a virtual Ethernet pair (veth). One end lives inside the container; the other lives in the host's root namespace.

Every packet that crosses between the container and the host must be transferred between two network namespaces. This transfer requires a context switch.

MetricValue
Cost of a single context switch~1,000–2,000 CPU cycles
Packets per second under moderate load50,000–100,000
CPU cycles wasted on context switches alone50–200 million cycles per second
Percentage of CPU spent on switching vs processing15–25%

Under higher load (200,000+ packets/second), this can reach 40% of CPU time wasted purely on moving packets between namespaces.

Sub-Layer 1B: iptables and DNAT/SNAT

Docker uses iptables to forward ports. When you map -p 8080:80, the kernel sets up DNAT rules:

iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80
OperationCPU Cost
IP header modification~50–100 cycles
Port checksum recalculation~50 cycles
Routing table lookup~100–200 cycles
conntrack entry creation/lookup~200–500 cycles
Total per packet~400–900 CPU cycles

For 100,000 packets/second, that is 40–90 million CPU cycles per second spent purely on NAT and routing—before any application logic.

Sub-Layer 1C: The conntrack Table

The kernel maintains a connection tracking table (nf_conntrack) for NAT. Every active connection is tracked.

Default limit: 65,536 entries.

When the table is full, the kernel drops packets silently.

# You will see this in dmesg: nf_conntrack: table full, dropping packet.
Connections/secondEntries created per minuteTime to fill default table
1,00060,000~1 minute
5,000300,000~13 seconds
10,000600,000~6 seconds

Once the table is full, new connections are dropped without any application-level error. The developer sees unexplained timeouts, connection resets, and failures—but the application logs show nothing.

This is a silent failure. No exception. No stack trace. Just packets vanishing.

Sub-Layer 1D: Ephemeral Port Exhaustion

When Docker uses NAT for outgoing connections, it consumes host ephemeral ports.

sysctl net.ipv4.ip_local_port_range # Output: 32768 60999

That is 28,232 ports.

Connections per secondPorts consumedTime to exhaust
1,00028,23228 seconds
5,00028,2325.6 seconds
10,00028,2322.8 seconds

Once exhausted, new connections fail with:

EADDRNOTAVAIL (Cannot assign requested address)

Important distinction:

Layer One: Summary Table

MetricPure POSIXDocker Networking
Network hops16+
Context switches per packet02 (container ↔ host)
NAT overheadNoneDNAT + SNAT per packet
conntrack entriesNone1 per active connection
Ephemeral ports consumedNone1 per NAT mapping
Packet drop behaviorClear EMFILE/ENFILE errorsSilent drops (no app-level error)
Latency penalty~0.1–0.3ms~1–5ms additional

Layer Two · The Filesystem Disaster

What Happens in Pure POSIX?

The application calls open(), read(), write(), close(). The Virtual File System (VFS) looks up the file in the kernel's directory cache (dcache). The kernel reads the file from the disk or from the page cache.

Application → VFS → dcache → Page Cache → Physical Disk

What Happens in Docker OverlayFS?

Docker uses OverlayFS (or Overlay2). This is a union filesystem that combines multiple layers into a single view.

Container View │ ▼ ┌─────────────┐ │ Upperdir │ ← Writable layer (container-specific) │ (writable) │ └─────────────┘ │ ▼ ┌─────────────┐ │ Lowerdir 1 │ ← Read-only (image layer 1) └─────────────┘ │ ▼ ┌─────────────┐ │ Lowerdir 2 │ ← Read-only (image layer 2) └─────────────┘ │ ▼ ┌─────────────┐ │ Lowerdir N │ ← Read-only (image layer N) └─────────────┘

Sub-Layer 2A: Layer Lookup Overhead

When the application calls open("/some/file"):

Number of layersAverage lookup time (relative to POSIX)
11.0×
5~2.5×
10~4.0×
20~7.0×

Sub-Layer 2B: Copy-on-Write (CoW) Disaster

When the application modifies a file that exists in a lower layer:

File SizeCopy Time (SSD)Copy Time (HDD)
1 MB~5ms~20ms
10 MB~20ms~100ms
100 MB~100ms~800ms
1 GB~1s~8s

Sub-Layer 2C: Page Cache Deduplication Loss

ScenarioPage Cache Behavior
50 containers, same image, same layer hashShared
50 containers, same base image, different app layersPartially shared
50 containers, different base imagesNot shared

Layer Two: Summary Table

MetricPure POSIXOverlayFS (Docker)
File openO(1) dcache lookupO(N) layer walk
File modifyIn-place writeCopy entire file + write
Page cache sharingFullPartial (inode-based)
I/O latencyStableSpikes on CoW operations
Memory efficiencyOptimalRedundant copies in mixed environments

Layer Three · The CPU Scheduler Disaster

What Happens in Pure POSIX?

The Linux Completely Fair Scheduler (CFS) schedules processes based on their nice values and CPU usage. No artificial limits. No throttling.

Process → CFS → CPU Core

What Happens with cgroups and CFS Bandwidth Control?

Docker uses cgroups to limit CPU usage. This uses the CFS Bandwidth Control feature.

cpu.cfs_period_us = 100000 (100ms) cpu.cfs_quota_us = 150000 (150ms) # When --cpus=1.5 is set

Sub-Layer 3A: CFS Throttling Spikes

Without throttling (Pure POSIX): ┌──────────────────────────────────────────────────────┐ │ ████████████████████████████████████████████████████ │ │ 100ms of continuous execution │ └──────────────────────────────────────────────────────┘ With CFS throttling (Docker --cpus=1.5): ┌──────────────┬──────────────────────────────────────┐ │ ████████████ │ │ │ 15ms burst │ 85ms FORCED FREEZE (Throttled) │ └──────────────┴──────────────────────────────────────┘

Layer Three: Summary Table

MetricPure POSIXDocker with cgroups
CPU schedulingDirect CFS, no limitsCFS with quota/period throttling
Latency predictabilityDeterministicUnpredictable spikes
CPU freeze behaviorNeverUp to 100ms when quota exceeded
Memory reclaimSystem-wide, efficientcgroup-specific, more expensive
nproc limitsPer-processShared across containers with same UID
Failure visibilityClear errors (logs)Silent, confusing failures

Layer Four · The Daemon Bloat

Docker CLI → dockerd → containerd → containerd-shim → runc → Your Application
ComponentLanguageRoleMemory Consumption
dockerdGoREST API, image management100–300MB
containerdGoContainer lifecycle management50–150MB
containerd-shimGoPer-container process5–15MB per container
runcCLow-level execution~5MB (transient)
Total base overhead150–450MB (before any app)
Pure POSIX: ┌─────────────┐ │ Your App │ │ (PID) │───────────▶ Kernel (Direct Syscalls) └─────────────┘ Docker: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ docker │────▶│ dockerd │────▶│ containerd │────▶│ containerd│────▶│ runc │────▶│ Your App │ │ CLI │ │ (Daemon) │ │ (Daemon) │ │ -shim │ │ │ │ (PID) │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ REST API gRPC Pipes System Calls Memory consumption: 100MB+ + 50MB+ + 5-10MB/container

Layer Four: Summary Table

MetricPure POSIXDocker
DaemonsNonedockerd + containerd + containerd-shim
RAM overhead (base)0MB150–450MB
RAM overhead (per container)0MB5–15MB
IPC complexityNoneREST + gRPC + pipes + syscalls
Attack surfaceApplication only3+ additional daemons
DebuggingSimple (strace/gdb)Complex (5 layers)

Layer Five · The Dependency Bloat

Pure POSIX: Static binary, 2–5MB, zero runtime dependencies.

Node.js + Docker:

my-app/ ├── index.js (2KB) ├── package.json (1KB) ├── node_modules/ (300–800MB) │ ├── express/ │ ├── body-parser/ │ ├── cors/ │ ├── dotenv/ │ ├── 200+ packages...
MetricPure POSIXNode.js + Docker
DependenciesNone (static binary)Hundreds of packages
Deployment size200KB–5MB300MB–2GB
Library duplicationNoneMultiple versions of same lib
Memory managementManual (malloc/free)Garbage Collection (with pauses)
GC pausesNone10–500ms
Memory leaksNone (if coded correctly)Common from poorly written libs

Chapter Two · The Isolation Illusion

Docker claims to provide "isolation" between containers. This claim is technically false.

1. Security Isolation is Not Real

All containers on the same host share the same kernel.

CVEComponentYearSeverity
CVE-2019-5736runc2019Critical
CVE-2020-15257containerd-shim2020Critical
CVE-2022-0492cgroup v12022High
CVE-2023-25173containerd2023Critical
CVE-2024-21626runc2024Critical
"Containers do not provide the same level of security as virtual machines. The kernel is shared between containers and the host." — Docker Documentation

2. Network Isolation is Not Real

3. Filesystem Isolation is Not Real

4. CPU and Memory Isolation are Not Real

5. Kernel Resource Isolation is Not Real

ResourceVM (Hypervisor)Docker (Containers)
Kernel✅ Full (separate kernel)❌ Shared (same kernel)
CPU✅ Full (dedicated core)❌ Shared (contended)
RAM✅ Full (physically isolated)❌ Shared (contended)
Network✅ Full (VFIO/SR-IOV)❌ Shared (veth + NAT)
Page Cache✅ Full (isolated)❌ Shared
conntrack✅ Full (isolated)❌ Shared
Security✅ Full (hypervisor)❌ Partial (escape possible)

Chapter Three · The Cumulative Cost

Each layer interacts with the others, amplifying the total cost.

Dependency bloat (400MB node_modules) ↓ Container memory usage increases ↓ cgroup memory reclaim activates ↓ CPU cycles are consumed by reclaim logic ↓ Application consumes CPU quota faster ↓ CFS throttling freezes the process ↓ Latency spikes occur ↓ Sockets stay open longer ↓ conntrack table fills ↓ Silent packet drops ↓ Users see timeouts ↓ Cloud bill increases (more instances) ↓ (loop)
ParameterPOSIX/C ArchitectureDocker/Node.js Architecture
Deployment size2MB500MB
RAM per service10MB400MB
Avg latency0.3ms15ms
Peak latency1ms100ms+
Throughput (req/s, 1 core)100,000+5,000–15,000
Cloud cost (monthly)$100$5,000
Debugging time10 minutes1 day
Security surfaceApp only5+ daemons + shared kernel

Chapter Four · When Containers Make Sense

When Containers Make Sense

When Containers Do NOT Make Sense


Chapter Five · Root Cause Analysis


Chapter Six · The Alternative

Pure POSIX Architecture


Chapter Seven · The Conclusion

The joke we used to laugh at—"It works on my machine"—became an industry standard.

We stopped teaching developers how to write portable, standards-compliant code.

We stopped teaching them what malloc() does, how the kernel schedules processes, or why the network stack is designed the way it is.

We replaced engineering rigor with layers.

We replaced understanding with abstraction.

We replaced real isolation with administrative illusion.

And now we wonder why our systems are slow, unstable, insecure, and expensive to operate.

The industry does not need another layer. It needs a return to fundamentals.

It needs to stop shipping the environment and start shipping the solution.

It needs to stop wrapping the garbage and start fixing it.

It needs to stop pretending that containers provide isolation and start understanding what isolation actually means.

Because the joke stopped being funny a long time ago.


Appendix A · System Limits Reference

ParameterDefaultWhen ExceededError
net.ipv4.ip_local_port_range32768–60999Port exhaustionEADDRNOTAVAIL
net.netfilter.nf_conntrack_max65536Table fullSilent packet drops
ulimit -n (soft)1024Too many open filesEMFILE
ulimit -n (hard)4096Too many open filesEMFILE
ulimit -u (nproc)1024–4096Too many processesEAGAIN
vm.max_map_count65530Too many memory mapsENOMEM
fs.file-maxSystem-dependentToo many files system-wideENFILE
kernel.pid_max32768PID exhaustionProcess creation fails
net.core.somaxconn128Listen queue overflowECONNREFUSED
net.core.netdev_max_backlog1000Packet dropsSilent drops
net.ipv4.tcp_max_syn_backlog1024SYN queue overflowSYN flood drops

Appendix B · Container Escape CVEs

CVEComponentYearSeverity
CVE-2019-5736runc2019Critical
CVE-2020-15257containerd-shim2020Critical
CVE-2022-0492cgroup v12022High
CVE-2023-25173containerd2023Critical
CVE-2024-21626runc2024Critical

This work is licensed under a pragmatic engineering perspective — share freely, understand deeply.
All numbers based on Linux kernel documentation, public benchmarks, and real-world production data.