Spark Structured Streaming Can’t Save You From Bad Architecture

Roughly 80% of the "exactly-once" guarantees I see implemented in production financial pipelines are actually "at-least-once" disguised by a fragile post-processing cleanup script. We obsess over the Spark checkpointing mechanism while ignoring the fact that our sink is a leaky bucket. Why I chose this topic: I’ve spent the last six years cleaning... Weiterlesen