Distributed Task Orchestrator

Role: Systems Architect & Lead Engineer
Timeline: 14 weeks
Stack: Go, RabbitMQ, PostgreSQL, Kubernetes, Prometheus

Problem

The client operated a video processing pipeline that required reliable execution of 500K–2M tasks per day across heterogeneous worker nodes. Tasks included transcoding, thumbnail generation, metadata extraction, and quality analysis. The existing solution used a naive HTTP polling pattern that frequently lost tasks during node failures, required manual retries, and provided zero visibility into queue depth or task latency.

Downtime was unacceptable: a failed task could delay content delivery by hours. Recovery was manual. The operations team spent 40% of their time firefighting task failures.

System Architecture

                         ┌─────────────────┐
                         │  API Gateway     │
                         │  (rate-limited)  │
                         └────────┬────────┘
                                  │
                         ┌────────▼────────┐
                         │  TaskIngestor   │
                         │  (Go, validates │
                         │   & enqueues)   │
                         └────────┬────────┘
                                  │
                         ┌────────▼────────┐
                         │  RabbitMQ       │
                         │  (4 queues,     │
                         │   DLQ per type) │
                         └──┬──────┬───────┘
                            │      │
            ┌───────────────┘      └───────────────┐
            │                                      │
   ┌────────▼────────┐                   ┌─────────▼────────┐
   │  Worker Pool A  │                   │  Worker Pool B   │
   │  (transcoding)  │    ...            │  (analysis)      │
   └────────┬────────┘                   └─────────┬────────┘
            │                                      │
            └───────────────┬──────────────────────┘
                            │
                     ┌──────▼──────┐
                     │  PostgreSQL │
                     │  (task DB,  │
                     │  audit log) │
                     └─────────────┘

Key design decisions:

Idempotency keys: Every task carries a UUID4 idempotency key. Workers check task_events table before processing. Duplicate delivery is safe.
Dead letter queues: Each task type has a DLQ with configurable max-retry (default: 3). After exhaustion, tasks route to a manual-remediation queue.
Backpressure: RabbitMQ queue depth metrics feed into a feedback loop that throttles the ingestor when any queue exceeds 100K pending tasks.

Task Lifecycle Flow

INGESTED → QUEUED → CLAIMED → PROCESSING → COMPLETED
                        │                        │
                        │ (heartbeat timeout)    │ (error)
                        ▼                        ▼
                    RETRY_CLAIMED            FAILED
                        │                        │
                        │ (retries < max)        │ (retries >= max)
                        ▼                        ▼
                    QUEUED (re-enqueue)      DEAD_LETTER

Every state transition is recorded in PostgreSQL with a timestamp and worker ID. This enables point-in-time recovery and audit trails.

Failure Modes & Mitigations

| Failure | Detection | Mitigation | |---|---|---| | Worker crash mid-task | Heartbeat timeout (30s) | Task auto-re-queues with retry count += 1 | | Queue partition loss | RabbitMQ mirrored queues | Automatic failover to replica node | | PostgreSQL outage | Connection pool exhaustion | Circuit breaker trips, tasks remain in queue | | Network partition between workers and DB | Health check failure | Workers enter degraded mode (local queue) |

Results

99.97% task completion rate over 6 months
Zero data loss during 3 Kubernetes node failures
Recovery time reduced from hours to < 2 minutes (auto-remediation)
Operations overhead cut by 80% — no more manual task reconciliation

Lessons

The single most valuable investment was the idempotency key design. We spent two weeks debating the exact data model, and it saved us countless hours when we discovered that RabbitMQ's at-least-once delivery guarantees produced duplicates in exactly the scenarios we'd modeled. Always design for at-least-once. Always.