🔧 I Was Optimizing Ranking While the Real Problem Was Selection

I mistook movement for improvement.




For three months, I changed our ranking algorithm every two weeks.


BM25.
Hybrid BM25 + vector search.
A cross-encoder reranker.
A different embedding model.
Yet another embedding model.


Every change moved a few documents up or down the list.

None of them made the answers better.







The Same...