A streaming ingestion pipeline built to scale out without ever breaking an upstream quota. 50 async workers pull from a Redis job queue, and Redis Streams consumer groups make sure nothing is lost when a worker crashes.
The hard part is the shared limit. One atomic token bucket written in Lua inside Redis is shared by every replica, so adding workers adds throughput without adding risk. In a scale test, 300 workers across 6 pods held a hard cap of 8 requests a second, 167 grants in 20 seconds against 168 allowed.
It ships with Kubernetes manifests, Prometheus metrics, health checks, idempotent writes and exponential backoff with full jitter.