Let's go back to the buffet and compare to another validation tool. Today we'll look at . Just from the name of the exercise you know this is more up CsvPath Framework's alley than it is Validatar's.

The Validatar Example

Here's the problem description at the top of the Validatar example:

This Standard Test is designed to demonstrate the concept of how to create a uniqueness template test for all CSV files in multiple folders.

The Standard Test here compares the Row Count per account_id value in the account_data.csv to make sure all account_id's only have 1 row. The test only keeps failures and stops after 100 failure records.

Spoiler alert: in FlightPath this is a trivial example (as I think it is meant to be)

Validatar starts by having you create a test template. Before you can do that, though, you need a project. Here are the instructions for that step:

• Choose your project


• Make sure your Data Source is Mapped correctly

The first bullet sounds simple. I'm not sure what the second bullet means because I'm not a Validatar expert and that one isn't explained on that page.

Moving on, let's create that template. Most of this exercise is forms based. The setup is shown in the image below. Their ask is that you notice:

• Note that the column specified to group by is account_id


• Note that it is comparing the ROW_COUNT to a fixed value of 1


• Note that the Result Configuration is set so that only Failures are kept and to abort after 100 failures are found

Good requirements for us to use on the CsvPath side.

At this point we have our test. Now we need to create a template from it so that we can apply it to each CSV file. This is how we get to a single action we can apply to multiple files in a uniform way.

I'm going to just add the bullets because the screenshot is in the link above, which is of course a more complete description. We do:

• Click Build Template


• Update the Folder input to {{schema.name}}
  


• Update the File input to {{table.name}}
  


• Update the Column input to {{#replace table.name "_data.csv" "_id"}}
  


• Update the Metadata Links to {{schema.name}}.{{table..name}}
  


• Change the Generate column list using to Dynamic Template Configuration


• Update the Dynamic Script

The dynamic script is pretty simple:

    [

    {"name":"{{#replace table.name "_data.csv" "_id"}}","sequence":1,"type":"Numeric","role":"Key"},

        {"name":"ROW_COUNT","sequence":2,"type":"Numeric","role":"Value"}

    ]

Now, we're going to use some metadata to filter down to the files we care about.

• Switch to the Metadata Selection Tab


• Change to the Use Filters option


• Add a Filter on the Table Name Field that contains "_data.csv"

In the load dialog give the named-paths group the name dups and click Create.

Next stage your data. In the example, each file is in its own folder and its folder is one of many in the same directory. We'll just add the parent folder and let FlightPath find the files for us.

To do that right click the parent directory and select Stage data. In the stage data dialog uncheck the Separate named-files checkbox because we're going to have every physical file be one version of the same named-file. Think of a named-file as a category that has one file assigned to it at a time, in sequence. We say named-files have versions.

The result looks like:

And here's the Run dialog:

And that is it!

There is, of course, much more you can do with CsvPath Framework. Likewise, Validatar has a ton more functionality than what we showed. But now you've had a taste of both.

What I'd hope you come away with is that CsvPath Framework is the better tool for CSV, JSONL, and Excel file validation. The ease of using is a complete data preboarding solution, not just a validation engine. Preboarding inbound data files is a big deal. If you need that (and who doesn't?) you owe it to yourself to take a look at CsvPath Framework.

To whet your appetite, here's a post on data preboarding build or buy. Enjoy!