Why GridSearchCV Wastes Most of Its Time — And What I Did About It

If you've ever tuned hyperparameters on a large grid, you know the pain. You kick off a GridSearchCV, go make coffee, come back, and it's still running. Maybe you go to lunch. Maybe it's still running.

I got frustrated enough to build something different.

The Problem with GridSearchCV

GridSearchCV is brute force by design. For a grid with k parameters and n values each, it evaluates nᵏ × cv_folds configurations — every single one, regardless of how poorly a value performs early on.

Here's what that actually means in practice:

The last point is the killer. Your grid doesn't have to be huge for this to hurt — it just has to grow.

The Idea: Eliminate Instead of Enumerate

What if instead of evaluating everything upfront, we tested parameter values in rounds — and dropped the bad ones before they compound?

That's EliminationSearchCV.

It works like this:

• 
  Round 1: Test each parameter value in isolation. Score them per-parameter and eliminate the worst performers.


• 
  Round 2: Test surviving pairs. Rank all combinations globally, keep the top fraction.


• 
  Round 3+: Repeat with triples, then full combinations, until one winner remains.

Bad values get cut early. They never get the chance to multiply into thousands of useless combinations.

A Concrete Example

Let's tune a LogisticRegression with 4 parameters:

param_grid = {

    'C':        [0.001, 0.01, 0.1, 1, 10, 100],   # 6 values

    'penalty':  ['l1', 'l2'],                       # 2 values

    'solver':   ['liblinear', 'saga'],              # 2 values

    'max_iter': [1000, 2000],                       # 2 values

}

# GridSearchCV: 6 × 2 × 2 × 2 = 48 combos × 5 folds = 240 fits

With EliminationSearchCV and elimination_rate=0.8 (keep best 20%):

Same best params. A fraction of the work.

Drop-in Replacement

The API is intentionally identical to GridSearchCV:

from EliminationSearchCV import EliminationSearchCV



# Before

search = GridSearchCV(model, param_grid, cv=5)



# After — just swap the class name

search = EliminationSearchCV(

    estimator=model,

    param_grid=param_grid,

    scoring='accuracy',

    cv=5,

    elimination_rate=0.8,  # eliminate worst 80% each round

)



search.fit(X_train, y_train)



# Same interface as GridSearchCV

print(search.best_params_)

# → {'C': 1, 'penalty': 'l1', 'solver': 'liblinear', 'max_iter': 1000}



print(search.best_score_)

# → 0.9248



# Already refitted on full training set — ready to predict

search.best_estimator_.predict(X_test)

One Thing I'm Proud Of: Invalid Combo Handling

Sklearn can raise errors for incompatible combinations — like penalty='l1' with solver='lbfgs'. GridSearchCV crashes on these. You have to manually filter them out.

EliminationSearchCV catches any exception during fit(), scores that combination 0.0, and lets the elimination logic handle it naturally. Invalid combos just die in Round 1. No special handling needed from you.

Benchmark Results

Tested across 5 models and 3 datasets (cv=2, elimination_rate=0.8, 10,000 samples):

Full grids are where this shines. The accuracy trade-off is minimal — under 0.02 across all models, often zero.

Honest caveat: Light grids (small search spaces) are actually slower with this approach. The elimination overhead doesn't pay off when there are only a few combinations to begin with. If your grid is small, stick with GridSearchCV.

Architecture: How It's Built

The library is two files:

src/EliminationSearchCV/

├── EliminationSearchCV.py   ← Core class: fit(), elimination logic, scoring

└── Utils.py                 ← Stateless utilities: fold creation, combination generation, metrics

The flow inside fit():

EliminationSearchCV.fit(X, y)

    │

    ├─▶ Utils.create_cv_data_sets()         — StratifiedKFold/KFold splits

    │

    └─▶ [For each round i = 1 … n_params]

             │

             ├─▶ generate_param_combinations_with_limit(grid, limit=i)

             │

             ├─▶ _score_candidates(candidates)

             │         — per-fold metric evaluation

             │

             └─▶ _eliminate_low_scoring_values(candidates, scores)

                       ├─▶ _eliminate_single_param_values()   — Round 1

                       └─▶ _eliminate_multi_param_values()    — Rounds 2+

A key design decision: in Round 1, each parameter's values are scored and compared in isolation — so C values compete only against other C values, not against penalty values. This prevents interference between parameters that are on completely different scales.

In later rounds, all combinations are ranked globally and the top (1 - elimination_rate) fraction survives.

What's Working and What's Not Yet

Currently supported:

• 
  fit(), best_params_, best_score_, best_estimator_
  


• Round 1: per-parameter isolation and elimination


• Rounds 2+: global combination ranking


• StratifiedKFold / KFold cross-validation


• Invalid combination handling


• Scoring: accuracy, precision, recall, f1, roc_auc
  

On the roadmap:

• 
  cv_results_ (per-fold score breakdown)


• 
  n_jobs parallel evaluation via joblib


• 
  verbose logging


• Full pytest test suite


• Scikit-learn BaseEstimator compatibility

Try It

pip install elimination-search-cv

Requirements: Python ≥ 3.8. scikit-learn and numpy install automatically.

GitHub: https://github.com/thisal-d/elimination-search-cv

Honest Disclaimer

This is an experimental approach. The quality of results depends heavily on the dataset and model. I'm actively benchmarking it and the results so far are promising — but I wouldn't call it production-ready yet.

What I'd genuinely love is feedback on edge cases where it fails. If you try it on a grid where it gives clearly wrong results or behaves unexpectedly, please open an issue. That's more useful to me right now than praise.

If you found this interesting, a ⭐ on the repo helps a lot — it keeps the motivation alive to keep building.

Tags: python machinelearning datascience opensource