Using LLMs to Automate Data Cleaning and Transformation Pipelines

A new paradigm, not a replacement of data engineering, but a fundamental shift in where engineering effort concentrates.

If you were to ask a data engineer about their week, I am sure they would not speak about anything exciting. Most of their time is spent on data wrangling, messy upstream data, inconsistent date formats, null values that are not really null, and vendor exports that rename columns without noting those changes in the documentation. These are the kinds of data engineering tasks that nobody actually wants to do, yet they end up occupying the majority of their week.