Yes, another tree classifier. Here's what building one from scratch taught me.

I spent way too long building a tabular classifier from scratch. It doesn't

beat XGBoost. I'm publishing it anyway — with honest benchmarks — because

the honest result seems more useful than a drawer full of code.

Repo:

The honest benchmarks

One note before the tables. Everything below runs the basic version of

the algorithm, on defaults — I kept it that way so the idea stays visible

and the code stays readable (~900 lines of NumPy). There are plenty of

tweaks left on the table (just prompt an AI to improve the score). I tried

a bunch of them in another version of this algorithm — and with every tweak

it started looking closer and closer to the methods that already exist. So

this version stays as close to the original idea as possible.

5-fold stratified CV, default parameters everywhere, cells are

accuracy / ROC-AUC / mean fit time.

And the same protocol over 14 OpenML datasets:

Mean accuracy rank (1 = best): HistGradientBoosting 1.46, RandomForest

1.79, DecisionTree 3.25, HypothesisTree 3.50.

Reading guide: accuracy lands in single-decision-tree territory (it beats

the tree outright on 5 of 14 and is best of all four models on exactly one

dataset, blood-transfusion). The ensembles win, as they do against nearly

everything on tabular data. On imbalanced datasets accuracy holds up but

AUC collapses toward 0.5 — probabilities come from per-cluster confidence,

and most clusters saturate at 1.0, so there's almost no ranking signal.

That one is the weakest part of the model, and it's documented in the

README rather than hidden.

What surprised me

1. A hyperparameter I designed, tuned, and then proved does nothing (as
   
   probably most of them). The match score is exp(-d/softness) — which is
   
   monotonic in d, so within any competition the ranking never changes,
   
   no matter what softness is. I tuned that knob more than once before
   
   noticing.




2. I tried to make a general learning algorithm — I made a tree. Boxes
   
   in feature space, growth driven by errors, parent-child structure...
   
   every design decision that worked pulled the thing closer to the shape
   
   of the methods I was trying to out-do. There's probably a lesson in
   
   there about why trees keep winning on tabular data.




3. The tree still overfits to noise, and a neural net handles it much
   
   better (what a real shock) Error-driven carving means every noisy
   
   point eventually earns its own little box if you let it — you can see it
   
   on the moons dataset (noise=0.25), where my model drops below even a
   
   plain decision tree while smoother models shrug.

Was it worth it?

I started this when nobody around cared about AI, and I genuinely hoped it would beat the established methods. It didn't, and watching the benchmark table say so, fold after fold, was not a great time.

But the gains don't fit in a table. I can derive every decision this model makes from first principles. I learned to benchmark honestly instead of hopefully.

And where a drawer full of code used to be, there's now a tested, documented repo with benchmark tables I don't have to apologize for. That trade I'd take again.

What's next

I'm probably done with the current tree. Maybe PyPI, if anyone cares.

The next thing I want to try is merging MLPs and trees — trees are fast and

well-optimized, neural nets deal with noise and unstructured data, and

surprise #3 suggests they'd cover each other's blind spots.

Repo: https://github.com/cloudlesson95-arch/hypothesis-tree — issues,

benchmarks disputes, and pointers to related work I've missed (RCE networks

and PRIM box-hunting are the closest relatives I found) are all welcome.