API Gateway Design

Role: Creator & Maintainer
Timeline: Ongoing (open source, 1.2K GitHub stars)
Stack: Go, Envoy, Redis, OpenTelemetry, Vue.js (dashboard)

---

Motivation

Most API gateways fall into two categories: heavyweight all-in-one platforms (Kong, Apigee) or minimal reverse proxies (nginx, Caddy). There was no ergonomic middle ground that offered:

• Dynamic route configuration without restarts (no nginx reloads)
• First-class observability (traces, metrics, logs out of the box)
• Pluggable middleware in a sandboxed runtime (Lua, WASM)
• Configuration as code (GitOps-friendly YAML/JSON)

So I built one.

---

Architecture

                      ┌──────────────────────┐
                      │   Control Plane       │
                      │ (Go, reads from       │
                      │  etcd / file / k8s)   │
                      └──────────┬───────────┘
                                 │ gRPC stream
                                 │ (route config)
                      ┌──────────▼───────────┐
                      │   Data Plane          │
                      │ (Go reverse proxy,    │
                      │  per-core event loop) │
                      │                      │
                      │  ┌────────────────┐  │
                      │  │ Middleware      │  │
                      │  │ Chain:         │  │
                      │  │  JWT Auth      │  │
                      │  │  Rate Limiter  │  │
                      │  │  Circuit Brkr  │  │
                      │  │  Request Log   │  │
                      │  └────────────────┘  │
                      └──────────────────────┘

---

Rate Limiting Strategy

The rate limiter supports three modes:

| Mode | Algorithm | Use Case | |---|---|---| | Fixed window | counter(key, window) per Redis | Simple per-IP limits | | Sliding window | ZREMRANGEBYSCORE + ZCARD | Burst-aware per-user limits | | Token bucket | SETEX + DECR | Precise per-route limits |

The sliding window mode is the default. It uses a Redis sorted set per key, adding a timestamp member per request and removing entries outside the window. This gives accurate per-second limits without the thundering herd of a centralized counter — at the cost of slightly more Redis memory.

func SlidingWindow(key string, limit int, window time.Duration) bool {
    now := time.Now().UnixMilli()
    cutoff := now - window.Milliseconds()
    // Redis: ZREMRANGEBYSCORE key 0 cutoff
    // Redis: ZADD key now now
    // Redis: ZCARD key
    // Return ZCARD <= limit
}

---

Circuit Breaker

The circuit breaker implements the standard closed/open/half-open state machine with one addition: a probing state between half-open and closed.

CLOSED → (failure threshold exceeded) → OPEN
  ↑                                          │
  │                                    (timeout expires)
  │                                          ↓
  └──── (probe succeeds) ← PROBING ←─── HALF_OPEN

In probing state, the gateway sends 1% of traffic through while keeping the breaker open for 99% of requests. If the probe failure rate stays below 5% for 30 seconds, it transitions to closed. This prevents the thundering herd problem where a single success in half-open state immediately floods a recovering service.

---

Observability

Every request produces:

• Trace: OpenTelemetry span with route_id, upstream_latency, status_code
• Metric: Histogram of latency by route, counter of status codes per upstream
• Log: Structured JSON record with request_id, method, path, duration

The dashboard (built with Vue.js) displays: RPS per route, error rate by upstream, P50/P95/P99 latency, circuit breaker state, and rate limiter hit count.

---

Results

• ~2M RPM throughput on a single 4-core instance (benchmark)
• ~85μs overhead added per request (middleware chain of 4 plugins)
• Zero-config hot reloads — routes update within 500ms of config change
• Adopted by 3 companies in production

---

Key Takeaway

The most impactful design decision was the separation of control and data planes. It meant we could iterate on route configuration tooling without touching the hot path. The data plane stayed lean, the control plane could be as complex as needed.