🔒 Topic 28: Distributed Locking

SINGLE SERVER (LOCKS WORK FINE) ═══════════════════════════════════════════════════════════════════ ┌─────────────────────────────────────────────────────────────────┐ │ ONE SERVER │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Process A: lock acquired ✓ │ │ │ │ Process B: wait (blocked) │ │ │ │ Process C: wait (blocked) │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ Works perfectly. One source of truth. MULTIPLE SERVERS (LOCKS ARE DANGEROUS) ═══════════════════════════════════════════════════════════════════ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Server 1 │ │ Server 2 │ │ Server 3 │ │ │ │ │ │ │ │ Lock acquired ✓ │ │ Lock acquired ✓ │ │ Lock acquired ✓ │ │ (thinks it's │ │ (thinks it's │ │ (thinks it's │ │ the only one) │ │ the only one) │ │ the only one) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ REDIS (Single) │ │ │ │ Server 1: SET lock:user:123 "server1" NX EX 10 → SUCCESS │ │ Server 2: SET lock:user:123 "server2" NX EX 10 → FAIL (locked) │ │ Server 3: SET lock:user:123 "server3" NX EX 10 → FAIL (locked) │ │ │ │ ✅ Redis as single source of truth — works! │ └─────────────────────────────────────────────────────────────────┘ BUT WHAT HAPPENS DURING NETWORK PARTITION? ═══════════════════════════════════════════════════════════════════ ┌──────────────────┐ ┌──────────────────┐ │ Server 1 │ ✗✗✗ │ Server 2 │ │ (can reach │ │ (can't reach │ │ Redis) │ │ Redis) │ └──────────────────┘ └──────────────────┘ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Redis │ │ (No │ │ Alive │ │ Redis) │ └─────────────┘ └─────────────┘ Server 1: Can acquire locks normally Server 2: Can't acquire locks (Redis unreachable) → But what if it THINKS it can? Without proper distributed locking, Server 2 might assume the lock is free and acquire it locally — now TWO servers have the lock!

REDLOCK: ACQUIRE LOCK FROM MULTIPLE REDIS INSTANCES ═══════════════════════════════════════════════════════════════════ Instead of one Redis, use 5 independent Redis masters: ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Redis 1 │ │ Redis 2 │ │ Redis 3 │ │ Redis 4 │ │ Redis 5 │ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │ │ │ │ │ └────────────┴────────────┼────────────┴────────────┘ │ ┌──────▼──────┐ │ Server 1 │ │ tries to │ │ acquire │ └─────────────┘ Algorithm: 1. Generate unique lock ID (UUID) 2. Try to acquire lock from ALL 5 Redis instances 3. Use a short timeout (e.g., 5ms per Redis) 4. If lock acquired from MAJORITY (≥3 Redis instances): → Lock is acquired successfully 5. If not, release any locks you have and retry Why it's more reliable: • Even if 2 Redis instances die, you still have majority (3/5) • Network partition affecting 2 instances won't break the lock

// Install package
composer require redlock-php/redlock

use RedLock\RedLock;

$servers = [
    ['host' => 'redis1.example.com', 'port' => 6379],
    ['host' => 'redis2.example.com', 'port' => 6379],
    ['host' => 'redis3.example.com', 'port' => 6379],
    ['host' => 'redis4.example.com', 'port' => 6379],
    ['host' => 'redis5.example.com', 'port' => 6379],
];

$redlock = new RedLock($servers);

$lock = $redlock->lock('user_processing_123', 10000); // 10 seconds TTL

if ($lock) {
    try {
        // Perform critical operation
        $this->processUser(123);
    } finally {
        $redlock->unlock($lock);
    }
}

ZOOKEEPER: STRONGLY CONSISTENT DISTRIBUTED LOCKS ═══════════════════════════════════════════════════════════════════ ZooKeeper uses the ZAB protocol (ZooKeeper Atomic Broadcast) for strong consistency: ┌─────────────────────────────────────────────────────────────────┐ │ ZooKeeper Ensemble (3/5 nodes) │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Leader │ │Follower │ │Follower │ │ │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ │ │ │ │ │ └────────────┴────────────┘ │ │ │ │ │ Consensus (ZAB) │ └────────────────────┼────────────────────────────────────────────┘ │ ┌────────▼────────┐ │ Application │ │ Server 1..N │ └─────────────────┘ Guarantees: • Linearizable writes (strong consistency) • Sequential consistency for reads • No split-brain (network partition? writes stop) • Fencing tokens (prevent lock from dead processes)

// Install PHP ZooKeeper extension
sudo apt-get install php-zookeeper

use Zookeeper;

class ZooKeeperLock
{
    private Zookeeper $zk;
    
    public function __construct(string $hosts)
    {
        $this->zk = new Zookeeper($hosts);
    }
    
    public function acquireLock(string $lockName, int $timeout = 30): bool
    {
        $lockPath = "/locks/{$lockName}";
        
        // Create sequential, ephemeral node
        $path = $this->zk->create(
            $lockPath . '/lock-',
            '',
            Zookeeper::EPHEMERAL | Zookeeper::SEQUENCE
        );
        
        // Check if this is the smallest sequence number
        $children = $this->zk->getChildren($lockPath);
        sort($children);
        
        if (basename($path) === $children[0]) {
            return true;  // We have the lock
        }
        
        // Wait for the previous lock to be released
        // ... watch implementation omitted for brevity
        
        return false;
    }
}

THE GHOST LOCK PROBLEM (No Fencing Token) ═══════════════════════════════════════════════════════════════════ Time 0ms: Process A acquires lock (token=1) and starts processing Time 100ms: Process A has GC pause (or network delay) Time 200ms: Lock expires, Process B acquires lock (token=2) Time 300ms: Process A wakes up, continues processing Time 400ms: Process A and Process B both think they have lock! Result: CORRUPTED DATA! WITH FENCING TOKEN (Solution): ═══════════════════════════════════════════════════════════════════ Time 0ms: Process A acquires lock (token=1) and starts processing Time 100ms: Process A has GC pause Time 200ms: Lock expires, Process B acquires lock (token=2) Time 300ms: Process A wakes up, tries to write to database Database rejects write because token=1 < current token=2 Time 400ms: Only Process B writes. Data is safe! LARAVEL IMPLEMENTATION (with ZooKeeper/etcd):

// Fencing token in action
class FencedResource
{
    private int $currentToken;
    
    public function write(string $data, int $fencingToken): void
    {
        if ($fencingToken < $this->currentToken) {
            throw new \Exception("Fencing token expired. Write rejected.");
        }
        
        // Perform write with current token
        DB::table('resource')->update([
            'data' => $data,
            'last_write_token' => $fencingToken,
        ]);
        
        $this->currentToken = $fencingToken;
    }
}