Every time I start a new data project, I write the same boilerplate:
python
df.isnull().sum()
df.duplicated().sum()
df.dtypes
df.describe()
df[df['age'] < 0]
# ... 40 more lines
After doing this for the hundredth time, I built OMR (Omni Data Refinement) — a Python library that replaces all of that with a single call.
What is OMR?
OMR is an open-source Python framework for dataset quality, validation, profiling, and monitoring. Think of it as a health check for your data — before you do any ML, EDA, or transformation, you should know how healthy your data actually is.
Installation
bash
pip install omni-data-refinement
Only needs pandas, numpy, and rich. No cloud, no LLMs.
The Core Feature: Health Score
python
import pandas as pd
from omr import Dataset
df = pd.read_csv("your_data.csv")
report = Dataset(df).health()
print(report.score) # e.g. 87/100
You get a 0-100 quality score covering 5 dimensions:
Pillar What it checks
Completeness Missing values
Uniqueness Duplicates
Consistency Type mismatches
Validity Value ranges and format rules
Conformity Schema adherence
Auto-Cleaning
python
dataset = Dataset(df)
dataset.clean()
dataset.explain_changes() # See exactly what was fixed
OMR auto-resolves missing values, duplicates, and type mismatches — and gives you a full transformation log so nothing is a black box.
Schema Validation
python
from omr import schemas
schema = {
"age": schemas.PositiveInteger(max=120),
"salary": schemas.PositiveFloat(min=10000),
"status": schemas.OneOf("active", "inactive"),
"email": schemas.Email()
}
dataset.validate(schema)
Drift Detection
Compare your current dataset against production or a previous version:
python
prod_dataset = Dataset(pd.read_csv("prod_data.csv"))
dataset.compare(prod_dataset)
Uses PSI, KS Test, and JS Divergence under the hood.
Statistical Analysis
python
dataset.analyze()
# Detects: outliers, multicollinearity, skewness, class imbalance, zero variance
Column Profiling (replaces .describe())
python
dataset.profile()
Unlike .describe() which only covers numeric columns, OMR profiles every column — numeric, categorical, and boolean — in a clean terminal table.
The Fluent API
Chain everything together:
python
clean_df = (Dataset(df)
.health()
.clean()
.analyze()
.export()) # Exports HTML, Markdown, or JSON report
Why I Built This
The goal is for OMR to become the first thing you run after pd.read_csv() — not a replacement for Pandas, but the quality layer on top of it.
Links
PyPI: pip install omni-data-refinement
GitHub: https://github.com/Omar-Alshafai2/omni-data-refinement
Docs: https://Omar-Alshafai2.github.io/omni-data-refinement/
Examples: https://Omar-Alshafai2.github.io/omni-data-refinement/examples/
What data quality problems do you run into most? I would love to know what to build next.
SOCIAL SHARE CARD GENERATOR