Engineering decisions behind dsweep — a three-stage detection pipeline, reversible quarantine, and why I keep reaching for .NET when I want a fast CLI.
The obvious approach, and why it's wrong
The naive implementation of a duplicate-file scanner goes like this: walk the directory tree, hash every file with SHA-256, group files by hash, done.
It works. It's also brutally expensive. If you're scanning 50,000 files and most of them are different, you've computed full SHA-256 hashes of every single one — reading every byte of every file — to find a few hundred matches.
The interesting engineering problem isn't "how do I hash files." It's "how do I avoid hashing files I don't need to."
This post walks through the design of — MIT licensed. C#, .NET 10, ~600 lines.
The design applies beyond duplicate finding: any "compare by content" problem benefits from a cheap-first funnel. Dedup systems in databases (MinHash, LSH), spell checkers (edit distance gating on length difference first), image similarity search (perceptual hash before deep feature comparison) all use the same shape. Cheap gate first, expensive confirmation only for survivors.
I'm a full-stack engineer building .NET microservices and TypeScript frontends. Most of what I write about comes from real implementation work rather than tutorial derivation.
Open-source: a11y-scope — a free, self-hosted WCAG 2.2 accessibility monitor.
SOCIAL SHARE CARD GENERATOR