Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance

We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME is a different sport, and it broke us in a way that took a heap profiler to see.

The symptom

The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.

The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, node_modules, Library, build output), so surely we were not touching millions of files.

We were not. But something else was.

The finding

We use the excellent notify crate, specifically notify-debouncer-full, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache.

RecommendedCache resolves to FileIdMap. From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.

To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a (PathBuf, FileId) for every entry it finds.

Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events. The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME, so it faithfully walked all of $HOME, including everything we thought we had excluded, following symlinks as it went.

Under malloc_history, the damage was specific:

• 23.6 GB of live heap in the cache


• 4.16 GB of that in the hashbrown table itself


• 19.4 GB spread across 54.2 million path allocations


• roughly 6.4 million files stat-ed at boot, which is the pinned core and the ten minutes

54 million allocations to remember the inode of files we had explicitly said we did not care about.

The fix, and what it costs

We switched the cache to NoCache.

That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap exists for a reason: with NoCache, the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B.

We could absorb that, for two reasons that are specific to our design:

1. Our indexer is idempotent. Indexing the same file twice converges to the same row, so a spurious create is cheap and harmless.


2. We already run a rescan and reconcile pass that removes index entries whose files no longer exist. Deletions get collected there regardless of whether the watcher paired them correctly.

So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.

Result: 24.8 GB down to 1.45 GB.

The second bug we found on the way

While measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per configured root.

Our roots were nested. We watched $HOME, and we also watched Documents, Downloads, and Desktop because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated lines per document.

The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.

The lesson worth stealing

Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.

So, two things:

Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.

Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc_history found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.

If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.