Stop Wasting Cloud Budget: Ingest & Chunk Unstructured Local Data in Milliseconds

Let's be honest: building Retrieval-Augmented Generation (RAG) pipelines, context-aware LLM agents, or local semantic search engines is incredibly exciting. What isn't exciting is wasting days writing boilerplate ingestion logic or pulling in bloated, multi-gigabyte frameworks just to split a folder of local Markdown and text files.

When did parsing a local directory and breaking text into clean chunks become so heavy? Monolithic AI orchestrators introduce massive dependency chains, require internet connectivity to calculate basic token metrics, and frequently mangle structural context by splitting text arbitrarily mid-word or mid-sentence.

To solve this exact engineering friction point, I built NexusFlow—a high-performance, zero-configuration local data pipeline engine designed to turn messy local text streams into optimized, context-preserving semantic chunks entirely offline.

Interactive Repository

Check out the full source code, architecture logs, and contribution guidelines natively on GitHub:

pip install ragmill[all]   # includes PDF + DOCX + embeddings support

# or

pip install ragmill        # core only (txt/md), zero dependencies

pip install -e ".[dev]"

pytest tests/ -v

from ragmill import RAGEngine



engine = RAGEngine(chunk_size=500, overlap=50)

chunks = engine.execute_pipeline("./my_documents")



for chunk in chunks:

    print(chunk["metadata"]["filename"], chunk["content"][:80])