← back to work

Realtime Event Pipeline

Role: Systems Architect
Timeline: 8 weeks
Stack: Kafka, Java, Redis, Flink, Grafana, Kubernetes

Problem

A fintech platform needed to process 50K financial events per second with end-to-end latency under 100ms. Events included trades, account updates, fraud signals, and compliance alerts. Any lost event could result in regulatory penalties. The existing system used a monolithic Java application with in-memory queues that crashed weekly under peak load.

The non-negotiables:

  • Exactly-once semantics (no duplicates, no drops)
  • Sub-100ms P99 end-to-end latency
  • 99.999% uptime for the pipeline
  • Full audit trail for every event

Pipeline Architecture

Source Apps ─→ Kafka Ingestion ─→ Flink Processing ─→ Sinks
                                                        
  [Trade]         [topic: trades]    [FraudDetector]    [PostgreSQL]
  [Account]──→    [topic: accounts]  [Enricher]    ─→  [Redis Cache]
  [Compliance]    [topic: alerts]    [Router]          [Alert Webhook]

The pipeline is divided into three stages:

Stage 1: Ingestion

Events arrive via gRPC at the ingestion service, which validates schemas against a Protobuf registry, assigns a Kafka offset, and publishes to partitioned topics. Partition key is customer_id to preserve ordering within a customer session.

Throughput test: 65K events/sec with P99 produce latency of 8ms (3 brokers, replication factor 3, acks=all).

Stage 2: Processing (Flink)

Flink consumes from Kafka with checkpointing every 30 seconds. The job graph:

source → keyBy(customer_id) → flatMap(validate) → split:
  ├── trade → keyBy(instrument_id) → window(10s) → aggregate → sink(trade_stats)
  ├── account → map(enrich) → sink(account_updates)
  └── compliance → process(alert_rules) → sideOutput(alerts) → sink(webhook)

Exactly-once semantics via:

  • Kafka transactions for source/sink coordination
  • Flink's two-phase commit sink
  • Idempotent writes to PostgreSQL (upsert on event_id)

Stage 3: Sinking

| Sink | Tech | Mechanism | |---|---|---| | Event store | PostgreSQL | Batch upsert every 5K events or 1s | | Cache | Redis + RediSearch | SET with TTL = 24h for recent events | | Alerts | Webhook | HTTP POST with retry (3 attempts, exponential backoff) | | Metrics | Prometheus | Histogram of end-to-end latency per event type |

Data Flow Diagram

        ┌──────┐     ┌──────────┐     ┌─────────┐     ┌──────────┐
  gRPC ─┤Ingest├─┬──→│ Kafka    ├──┬──→│ Flink   ├──┬──→│ Sink     │
        └──────┘ │   │ (3 nodes)│  │   │ (8 TMs) │  │   │ (varies) │
                 │   └──────────┘  │   └─────────┘  │   └──────────┘
                 │                 │                │
              [validate]      [exactly-once]    [batch write]
              [schema check]  [retry logic]     [idempotent]

Latency Budget

| Stage | Budget | P99 Observed | |---|---|---| | gRPC receive + validate | 5ms | 2ms | | Kafka produce | 15ms | 8ms | | Kafka consume + Flink processing | 40ms | 22ms | | Sink write | 30ms | 18ms | | Network overhead | 10ms | 5ms | | Total | 100ms | 55ms |

Incident Response

Scenario: Kafka broker disk fills up at 2AM.

  1. Alert fires from Prometheus (disk usage > 85%)
  2. Flink checkpoints begin failing → auto-pause consumption
  3. SRE receives page via PagerDuty
  4. Autoscaler provisions new broker node
  5. Kafka reassigns partition leaders (60s)
  6. Flink resumes from last successful checkpoint
  7. Events in-flight during failover: 0 lost (exactly-once)
  8. Total downtime: 4 minutes

Results

  • 50K events/sec sustained throughput with 55ms P99 latency
  • 0 events lost in 8 months of production (over 800B processed)
  • 3 broker failures handled with zero data loss
  • Audit-ready: every event traceable from source to sink with full metadata