Stop Writing .isnull() — Audit Your Dataset in One Line with OMR

python

df.isnull().sum()

df.duplicated().sum()

df.dtypes

df.describe()

df[df['age'] < 0]

# ... 40 more lines

After doing this for the hundredth time, I built OMR (Omni Data Refinement) — a Python library that replaces all of that with a single call.



What is OMR?

OMR is an open-source Python framework for dataset quality, validation, profiling, and monitoring. Think of it as a health check for your data — before you do any ML, EDA, or transformation, you should know how healthy your data actually is.



Installation

bash

pip install omni-data-refinement

Only needs pandas, numpy, and rich. No cloud, no LLMs.



The Core Feature: Health Score

python

import pandas as pd

from omr import Dataset

df = pd.read_csv("your_data.csv")

report = Dataset(df).health()

print(report.score)  # e.g. 87/100

You get a 0-100 quality score covering 5 dimensions:



Pillar  What it checks

Completeness    Missing values

Uniqueness  Duplicates

Consistency Type mismatches

Validity    Value ranges and format rules

Conformity  Schema adherence

Auto-Cleaning

python

dataset = Dataset(df)

dataset.clean()

dataset.explain_changes()  # See exactly what was fixed

OMR auto-resolves missing values, duplicates, and type mismatches — and gives you a full transformation log so nothing is a black box.



Schema Validation

python

from omr import schemas

schema = {

    "age":    schemas.PositiveInteger(max=120),

    "salary": schemas.PositiveFloat(min=10000),

    "status": schemas.OneOf("active", "inactive"),

    "email":  schemas.Email()

}

dataset.validate(schema)

Drift Detection

Compare your current dataset against production or a previous version:



python

prod_dataset = Dataset(pd.read_csv("prod_data.csv"))

dataset.compare(prod_dataset)

Uses PSI, KS Test, and JS Divergence under the hood.



Statistical Analysis

python

dataset.analyze()

# Detects: outliers, multicollinearity, skewness, class imbalance, zero variance

Column Profiling (replaces .describe())

python

dataset.profile()

Unlike .describe() which only covers numeric columns, OMR profiles every column — numeric, categorical, and boolean — in a clean terminal table.



The Fluent API

Chain everything together:



python

clean_df = (Dataset(df)

    .health()

    .clean()

    .analyze()

    .export())  # Exports HTML, Markdown, or JSON report

Why I Built This

The goal is for OMR to become the first thing you run after pd.read_csv() — not a replacement for Pandas, but the quality layer on top of it.



Links

PyPI: pip install omni-data-refinement

GitHub: https://github.com/Omar-Alshafai2/omni-data-refinement

Docs: https://Omar-Alshafai2.github.io/omni-data-refinement/

Examples: https://Omar-Alshafai2.github.io/omni-data-refinement/examples/





What data quality problems do you run into most? I would love to know what to build next.