Replay, Scheduling, and Throttling: Engineering a Self-Service Kafka Streaming Tool at Scale

What started as a recovery tool for streaming pipelines ended up as a full-fledged self-service platform, with scheduled watermark-based incremental ingestion, ad-hoc/one-time data replay, and throughput control built in along the way. The origin was narrow: streaming pipelines fail. A downstream consumer has an outage, a schema changes, a job... Weiterlesen: Replay, Scheduling, and Throttling: Engineering a Self-Service Kafka …