go-intake v0.1.0: A Small Go Library for Messy Data Intake

I released go-intake v0.1.0.

It is a small Go library for a very specific problem:

Turning unknown or messy flat input data into validated, transformed, record-oriented output.

Most data systems do not start with Kafka, Iceberg, dbt, Airflow, or a clean warehouse.

They usually start with something much less glamorous:

• CSV exports


• JSONL files


• manual uploads


• dirty headers


• missing fields


• mixed types


• numeric values stored as strings


• bad rows that should not silently disappear

That first layer is usually not “big data engineering”.

It is data intake.

And in many Go applications, I found myself needing the same primitives again and again:

• read records


• normalize headers


• parse fields


• validate business rules


• separate invalid rows


• write clean output


• keep the pipeline testable

So I built go-intake.

Repository:

https://github.com/firfircelik/go-intake

What problem does go-intake solve?

A lot of ETL tools focus on orchestration, scheduling, connectors, DAGs, streaming infrastructure, or warehouse transformations.

Those are important problems.

But before any of that, there is often a smaller and more boring problem:

Can I safely accept this file into my system?

For example:

• Are the headers usable?


• Are required fields present?


• Are values parseable?


• Are numbers actually numbers?


• Are dates valid?


• Are there missing or malformed records?


• What should happen to invalid rows?


• Can I inspect the file before I trust it?

go-intake focuses on this early intake layer.

It is not trying to be a full data platform.

It is designed to be embedded inside your own Go application when you need a small, explicit, testable way to process incoming records.

Core design

The pipeline model is intentionally simple:

Source → Transformer → Validator → Quarantine → Sink

A typical pipeline looks like this:

p := intake.New().

    From(source.CSV("input.csv")).

    Transform(

        transform.NormalizeHeaders(transform.SnakeCase),

        transform.TrimStrings(),

        transform.ParseFloat("price"),

    ).

    Validate(

        validate.Required("product"),

        validate.Min("price", 0),

    ).

    OnInvalid(quarantine.JSONL("bad-records.jsonl")).

    To(sink.JSONL("output.jsonl"))

The important part is not just that it reads CSV or writes JSONL.

The important part is the contract.

Each stage has a small responsibility:

• 
  Source reads records


• 
  Transformer cleans or reshapes records


• 
  Validator enforces rules


• 
  Quarantine captures invalid records


• 
  Sink writes valid records

This keeps the pipeline easy to reason about and easy to test.

What go-intake is

go-intake is:

• Go-native


• library-first


• streaming


• dependency-free


• record-oriented


• explicit about validation


• designed to be embedded inside applications

It is useful when you need to process flat input data in a controlled way without bringing in a large framework.

What go-intake is not

go-intake is not:

• a CLI


• a DAG engine


• a scheduler


• a dataframe library


• a connector marketplace


• a YAML-driven orchestration tool


• an Airflow replacement


• an Airbyte replacement

This is intentional.

There are already excellent tools for orchestration, declarative streaming, and connector-heavy workflows.

go-intake focuses on a smaller layer:

accepting, cleaning, validating, and routing records before the rest of your system depends on them.

Why Go?

Go is a good fit for this kind of library because:

• it has simple interfaces


• it is easy to ship as a single binary


• it has strong standard library support for files and encoding


• it performs well for streaming record processing


• it encourages explicit error handling


• it is easy to test

For internal tools, ingestion services, data quality checks, and file-processing workflows, Go is often a very practical choice.

I wanted go-intake to feel like a normal Go package, not a mini-platform hidden inside a package.

Streaming by default

One design goal is that the full dataset should not be loaded into memory.

The pipeline processes records one at a time.

That means the same code can work for:

• a small test file


• a manual business upload


• a larger production export


• a recurring intake job

This is especially important for boring operational data work, where file sizes may grow over time.

Invalid data is first-class

One thing I care about a lot:

Invalid data should not disappear silently.

In many small ingestion scripts, bad rows are either skipped, logged vaguely, or cause the whole job to fail.

Sometimes failing the whole job is correct.

But often, especially in intake workflows, you want to continue processing valid records while preserving invalid ones for inspection.

That is where quarantine comes in.

Invalid records can be written with structured metadata such as:

_errors

_stage

_timestamp

This gives you a better audit trail.

Instead of asking “why did 40 rows disappear?”, you can inspect the quarantine output and see what failed and where.

Transformers return fresh records

Another design decision:

Transformers should not mutate input records in place.

A transformer receives a record and returns a new one.

That makes pipelines easier to reason about.

It also makes testing safer because each stage has a clearer contract.

For data processing code, accidental mutation can create subtle bugs, especially when multiple transformations are chained together.

Keeping transformations explicit helps avoid that.

Validation should be explicit

Schema inference is useful, but it should not replace business rules.

go-intake includes discovery/profiling functionality for unknown files, but production validation should be written directly.

For example:

validate.Required("product")

validate.Min("price", 0)

This makes the rule obvious.

The goal is not to guess everything automatically.

The goal is to help you inspect unknown input, then write clear validation rules for the fields that matter.

How is this different from other Go ETL tools?

There are Go-based tools and libraries that focus on broader ETL workflows, DAGs, streaming, connectors, or declarative pipelines.

Those are useful when you need a larger data movement or processing platform.

go-intake is different because it intentionally stays smaller.

It does not try to own orchestration.

It does not try to define your whole data platform.

It does not require YAML configuration.

It does not include a connector marketplace.

It does not introduce third-party dependencies.

Instead, it gives you a small set of primitives that you can compose inside your own application.

The difference is scope.

Many ETL tools ask:

How do we orchestrate and move data across systems?

go-intake asks:

How do we safely accept and validate records before they enter the system?

That smaller problem is still important.

And it appears everywhere.

Zero third-party dependencies

For v0.1.0, go-intake has no third-party dependencies.

That is intentional.

For an intake layer, I want the dependency surface to stay as small as possible.

This matters when the library is used inside:

• internal business tools


• ingestion services


• audit workflows


• data quality checks


• lightweight automation jobs

A small dependency surface makes the library easier to understand, audit, and maintain.

Example use cases

Some places where this kind of library can be useful:

• internal CSV upload processing


• validating supplier/customer exports


• cleaning flat files before loading them into a database


• building small ingestion services


• writing testable data quality checks


• separating valid and invalid records


• inspecting unknown files before wiring them into a larger pipeline


• converting CSV to JSONL after normalization and validation

This is not meant to replace a full data platform.

It is meant to handle the intake layer cleanly.

Current status

This is still v0.1.0.

So I am not pretending it is a mature data platform.

The API may evolve.

The scope may become sharper.

But the direction is clear:

• small surface area


• composable interfaces


• streaming record processing


• explicit validation


• quarantine for invalid data


• no unnecessary framework magic

Sometimes the best tool is not another platform.

Sometimes it is a small, boring primitive that does one job clearly.

Repository:

https://github.com/firfircelik/go-intake

Feedback from Go developers and data engineers is very welcome.