Stop Trusting Your Accuracy Score: A Practical Guide to Evaluating Logistic Regression Models

"Accuracy lied to you. Here's the complete toolkit—confusion matrix, precision, recall, F1, ROC/AUC, log loss, and cross-validation—that separates models that look good from models that actually work."

You trained your first classifier, ran .score(), and got 97% accuracy. You shipped it. Three weeks later, your fraud team tells you it's catching zero fraudulent transactions.

Sound familiar? You fell into the accuracy trap—and it's the most common mistake from developers moving into ML.

This guide will give you the mental model and the code to evaluate binary classifiers properly. By the end, you'll know which metrics to reach for, when accuracy actively lies to you, how to read a ROC curve, and the seven pitfalls that silently kill production models.

Why Linear Regression Breaks for Classification

Before we get to evaluation, one minute on why we use logistic regression at all—because understanding the limitation it solves makes the evaluation choices clearer.

When you apply linear regression to a yes/no problem, you get predictions like 1.3 or -0.2. These aren't probabilities. They can't be thresholded reliably. And a single outlier in your training set can physically shift your decision boundary by several units:

import numpy as np

from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt



# Binary labels: 0 or 1

X = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)

y = np.array([0, 0, 0, 0, 1, 1, 1, 1])



model = LinearRegression().fit(X, y)

print(model.predict([[0]]))   # Predicts -0.36 — not a valid probability

print(model.predict([[10]]))  # Predicts 1.36 — also not valid

The sigmoid squashes any real number into the interval (0, 1), giving you an actual probability. It also models the log-odds of the positive class linearly, which is the statistician's way of saying "we get interpretable coefficients."

Under the hood, the model is optimized with Maximum Likelihood Estimation, minimizing cross-entropy loss (not squared error). The decision boundary is linear—a straight line in 2D feature space—but the output is a calibrated probability.

from sklearn.linear_model import LogisticRegression

from sklearn.datasets import make_classification

from sklearn.model_selection import train_test_split



X, y = make_classification(

    n_samples=1000, n_features=10,

    n_informative=5, random_state=42

)

X_train, X_test, y_train, y_test = train_test_split(

    X, y, test_size=0.2, random_state=42, stratify=y

)



# Note: C is the inverse of regularization strength (C = 1/λ)

# Smaller C = stronger regularization = less overfitting

model = LogisticRegression(C=1.0, max_iter=1000, random_state=42)

model.fit(X_train, y_train)



# Hard class predictions

y_pred = model.predict(X_test)



# Calibrated probabilities — use these for most evaluation tasks

y_prob = model.predict_proba(X_test)[:, 1]

Quick note on regularization: LogisticRegression in scikit-learn uses L2 regularization by default (penalty='l2'). Use penalty='l1' with solver='liblinear' if you want automatic feature selection via sparsity.

The Problem: Accuracy Actively Misleads You on Imbalanced Data

Here's the scenario that trips up almost everyone.

Fraud detection dataset:

• 10,000 transactions


• 9,900 legitimate (99%)


• 100 fraudulent (1%)

Build a model that predicts "legitimate" for every single transaction:

import numpy as np

from sklearn.metrics import accuracy_score



y_true = np.array([0]*9900 + [1]*100)

y_dummy = np.zeros(10000)  # Predicts "not fraud" always



print(f"Dummy accuracy: {accuracy_score(y_true, y_dummy):.1%}")

# Output: Dummy accuracy: 99.0%

scikit-learn convention (rows = actual, columns = predicted):

Plain English:

• 
  True Positive (TP): You predicted fraud. It was fraud.


• 
  True Negative (TN): You predicted legit. It was legit.


• 
  False Positive (FP): You cried wolf. Customer was innocent. (Type I error)


• 
  False Negative (FN): You missed the fraudster. (Type II error)

⚠️ Heads up: Some textbooks and tools swap the axis convention. When reading someone else's confusion matrix, always check the axis labels before drawing conclusions.

Precision, Recall, F1, and Specificity

Once you have the confusion matrix, every classification metric is just arithmetic on those four numbers.

Precision: "When I fire, do I hit?"

Precision = TP / (TP + FP)

Of all the positives you predicted, what fraction were actually positive? High precision means you rarely raise false alarms.

Reach for precision when false positives are expensive: spam filtering (you don't want to delete legitimate emails), content moderation (you don't want to wrongly remove posts).

Recall (Sensitivity): "Do I catch everything?"

Recall = TP / (TP + FN)

Of all the positives that actually exist, what fraction did you catch? High recall means you miss very few real positives.

Reach for recall when false negatives are dangerous: cancer screening (missing a tumor is catastrophic), fraud detection (missing fraud costs money), churn (missing a leaving customer means lost revenue).

The Unavoidable Trade-off

Lower your classification threshold → you predict positive more often → recall goes up, precision goes down. Raise it → fewer positive predictions → precision goes up, recall goes down. They move in opposite directions; there's no free lunch.

from sklearn.metrics import precision_recall_curve



precisions, recalls, thresholds = precision_recall_curve(y_test, y_prob)



plt.figure(figsize=(8, 5))

plt.plot(thresholds, precisions[:-1], label='Precision', color='blue')

plt.plot(thresholds, recalls[:-1], label='Recall', color='red')

plt.xlabel('Classification Threshold')

plt.ylabel('Score')

plt.title('Precision-Recall Trade-off vs Threshold')

plt.legend()

plt.grid(True, alpha=0.3)

plt.show()

🔑 Read classification_report carefully. The accuracy row at the bottom tells you almost nothing here. Look at the per-class precision, recall, and F1 for your minority class.

Choosing the Right Metric for Your Situation

Here's the decision framework I use before I even start training:

For fraud, churn, and disease: optimize recall first, then set a precision floor your business can tolerate. For spam filters and recommendation engines: optimize precision, accept some misses.

ROC Curve and AUC: Threshold-Independent Evaluation

All the metrics above assume a fixed decision threshold (typically 0.5). But the right threshold depends on your business context and changes as requirements evolve. How do you compare two models before you've even decided on a threshold?

Enter: the ROC (Receiver Operating Characteristic) curve.

ROC plots True Positive Rate (Recall) on the Y-axis against False Positive Rate on the X-axis, across every possible threshold. Each point on the curve is one threshold value.

from sklearn.metrics import roc_curve, roc_auc_score

import matplotlib.pyplot as plt



fpr, tpr, thresholds = roc_curve(y_test, y_prob)

auc = roc_auc_score(y_test, y_prob)



plt.figure(figsize=(7, 6))

plt.plot(fpr, tpr, color='steelblue', lw=2, label=f'ROC Curve (AUC = {auc:.3f})')

plt.plot([0, 1], [0, 1], 'k--', lw=1, label='Random Guessing (AUC = 0.5)')

plt.fill_between(fpr, tpr, alpha=0.15, color='steelblue')

plt.xlabel('False Positive Rate (1 - Specificity)')

plt.ylabel('True Positive Rate (Recall / Sensitivity)')

plt.title('ROC Curve')

plt.legend(loc='lower right')

plt.grid(True, alpha=0.3)

plt.tight_layout()

plt.show()

Lower log loss = better calibrated probabilities. A model with log loss > 0.693 is effectively worse than random probability assignment.

Cross-Validation: Getting Evaluation You Can Trust

Single train/test splits are noisy. If you got lucky (or unlucky) with how data was randomly partitioned, your metrics don't generalize. Cross-validation gives you a reliable estimate.

k-Fold Cross-Validation

Split data into k folds. Train on k-1 folds, test on the remaining fold. Repeat k times (once per fold as the test set). Average the k results.

from sklearn.model_selection import cross_val_score, StratifiedKFold

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler



# Always put preprocessing inside the pipeline!

# This prevents data leakage from the scaler.

pipe = Pipeline([

    ('scaler', StandardScaler()),

    ('clf', LogisticRegression(C=1.0, max_iter=1000, random_state=42))

])



cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)



# Evaluate multiple metrics

for metric in ['accuracy', 'f1', 'roc_auc', 'neg_log_loss']:

    scores = cross_val_score(pipe, X, y, cv=cv, scoring=metric)

    label = metric.replace('neg_', '-')

    print(f"{label:>15}: {scores.mean():.4f} ± {scores.std():.4f}")

       accuracy: 0.8920 ± 0.0145

             f1: 0.8911 ± 0.0163

        roc_auc: 0.9587 ± 0.0098

      -log_loss: -0.2734 ± 0.0121

Why Stratified, Not Regular k-Fold?

With imbalanced classes, a random split might put almost all the minority class examples in one fold—making some folds impossible to evaluate meaningfully.

StratifiedKFold preserves the class ratio in each fold. Use it by default for classification, especially with imbalanced data. It's almost always the right choice.

7 Pitfalls That Will Silently Break Your Evaluation

1. Reporting Only Accuracy

Symptom: Your model scores 97% accuracy and gets shipped. It catches nothing useful.


Fix: Always report precision, recall, F1, and AUC alongside accuracy. If classes are imbalanced, accuracy is a secondary metric at best.

2. Data Leakage in Preprocessing

Symptom: Suspiciously high validation metrics that don't hold up in production.


Cause: Fitting your scaler, imputer, or feature selector on the full dataset before splitting, letting test-set information influence your transforms.

# ❌ WRONG — scaler sees test data, leaks information

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)           # fit on everything

X_train, X_test = train_test_split(X_scaled) # split afterward



# ✅ CORRECT — use a pipeline or manually split first

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)  # fit on train only

X_test_scaled = scaler.transform(X_test)        # transform test with train params

3. Using Default 0.5 Threshold Without Questioning It

Symptom: Good AUC, terrible precision or recall in production.


Fix: Find the threshold that matches your business cost ratio, then tune it on a validation set.

# Find threshold that maximizes F1 on validation data

from sklearn.metrics import f1_score



best_f1, best_threshold = 0, 0.5

for threshold in np.arange(0.1, 0.9, 0.01):

    y_pred_t = (y_prob >= threshold).astype(int)

    f = f1_score(y_test, y_pred_t)

    if f > best_f1:

        best_f1 = f

        best_threshold = threshold



print(f"Best threshold: {best_threshold:.2f}, F1: {best_f1:.4f}")

4. Ignoring Class Imbalance in CV

Symptom: Cross-validation folds have inconsistent class distributions; some folds fail or give wild metric swings.


Fix: Use StratifiedKFold (shown above). Also consider class_weight='balanced' in your model:

model = LogisticRegression(class_weight='balanced', max_iter=1000)

5. Evaluating on the Test Set More Than Once

Symptom: You iterate by checking test metrics, making changes, re-checking—and unknowingly over-fit to the test set.


Fix: Use a three-way split or cross-validation for development; touch the test set exactly once for final reporting.

# Three-way split

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)

# Train on X_train, tune on X_val, final eval on X_test (once!)

6. Over-Relying on AUC-ROC with Severe Imbalance

Symptom: ROC-AUC looks great; actual fraud/disease detection rate is awful.


Fix: Switch to AUC-PR (average_precision_score) for heavily imbalanced problems.

7. Skipping a Baseline Comparison

Symptom: You report 0.87 AUC with no context.


Fix: Always compare against a dummy baseline.

from sklearn.dummy import DummyClassifier



dummy = DummyClassifier(strategy='most_frequent')

baseline_auc = cross_val_score(dummy, X, y, cv=5, scoring='roc_auc').mean()

model_auc = cross_val_score(pipe, X, y, cv=5, scoring='roc_auc').mean()



print(f"Baseline AUC: {baseline_auc:.3f}")

print(f"Model AUC:    {model_auc:.3f}")

print(f"Improvement:  {model_auc - baseline_auc:+.3f}")

The Complete Evaluation Workflow

Here's what we should actually run before declaring a model ready:

from sklearn.metrics import (

    classification_report, confusion_matrix, ConfusionMatrixDisplay,

    roc_auc_score, average_precision_score, log_loss, f1_score

)



def evaluate_classifier(model, X_test, y_test, threshold=0.5):

    y_prob = model.predict_proba(X_test)[:, 1]

    y_pred = (y_prob >= threshold).astype(int)



    print("=" * 50)

    print(f"EVALUATION REPORT (threshold = {threshold})")

    print("=" * 50)



    # 1. Classification report (precision, recall, F1 per class)

    print("\n--- Per-Class Metrics ---")

    print(classification_report(y_test, y_pred))



    # 2. Probabilistic metrics

    print(f"ROC-AUC:       {roc_auc_score(y_test, y_prob):.4f}")

    print(f"PR-AUC:        {average_precision_score(y_test, y_prob):.4f}")

    print(f"Log Loss:      {log_loss(y_test, y_prob):.4f}")



    # 3. Confusion matrix

    print("\n--- Confusion Matrix ---")

    cm = confusion_matrix(y_test, y_pred)

    disp = ConfusionMatrixDisplay(cm)

    disp.plot(cmap='Blues')

    plt.title(f"Confusion Matrix (threshold={threshold})")

    plt.tight_layout()

    plt.show()



evaluate_classifier(model, X_test, y_test, threshold=0.5)

Practical Takeaways Checklist

Before you call any classifier production-ready:

• [ ] Look at the confusion matrix first. Numbers before plots.


• [ ] Report precision, recall, and F1 for the minority class, not just overall accuracy.


• [ ] Use StratifiedKFold cross-validation to get reliable metric estimates.


• [ ] Compare ROC-AUC and PR-AUC. If classes are imbalanced, PR-AUC is your primary signal.


• [ ] Check log loss to verify your probabilities are well-calibrated, not just your hard predictions.


• [ ] Question the 0.5 threshold. Tune it to match the real cost of FP vs. FN in your domain.


• [ ] Use a Pipeline to prevent data leakage from preprocessing steps.


• [ ] Run a DummyClassifier baseline before celebrating your AUC score.


• [ ] Reserve your test set. If you've looked at it more than once during development, it's a validation set.


• [ ] Tie your metric choice to a business outcome. "We want to catch 90% of churners while maintaining > 60% precision" beats "maximize F1."

Final Thought

Model evaluation isn't about finding the best model in the abstract—it's about finding the right model for your specific problem. A 95% accuracy model can be completely useless. An 80% accuracy model can save lives or prevent fraud, depending on where it's wrong.

The metrics are just tools. The judgment—knowing which errors your system can tolerate and which it can't—is what makes you a useful engineer, not just a code runner.

Go measure wisely.

Found this useful? I'd love to hear which pitfall stung you hardest—drop it in the comments.