Open-Source Data Observability with Elementary — From Zero to Hero (Part 1)

Open-Source Data Observability with Elementary — From Zero to Hero (Part 1)

A step-by-step hands-on guide I wish I had when I was a beginner

Data observability and its importance have often been discussed and written about as a crucial aspect of modern data and analytics engineering. Many tools are available on the market with various features and prices. In this 2 part article, we will focus on the open-source version of Elementary, one of these data observability platforms, tailored for and designed to work seamlessly with dbt. We will start by setting up from zero and aiming to understand how it works and what is possible in different data scenarios by the end of part 2. Before we start, I also would like to disclose that I have no affiliation with Elementary, and all opinions expressed are my own.

In part 1, we will set up the Elementary and check how to read the Elementary’s daily report. If you are comfortable with this part already and interested in checking different types of data tests and which one suits bests for which scenario, you can directly jump into part 2 here:

• . Let us start by setting up the open-source version first.
  • How to Install Elementary

  Installing Elementary is as easy as installing any other package in your dbt project. Simply add the following to your packages.yml file. If you don't have one yet, you can create a packages.yml file at the same level as your dbt_project.yml file. A package is essentially another dbt project, consisting of additional SQL and Jinja code that can be incorporated into your dbt project.

  packages:
    - package: elementary-data/elementary
      version: 0.15.2
      ## you can also have different minor versions as:
        ##  version: [">=0.14.0", "<0.15.0"]
      ## Docs: https://docs.elementary-data.com

  We want Elementary to have its own schema for writing outputs. In the dbt_project.yml file, we define the schema name for Elementary under models. If you are using dbt Core, by default all dbt models are built in the schema specified in your profile's target. Depending on how you define your custom schema, the schema will be named either elementary or <target_schema>_elementary.

  models:
    ## see docs: https://docs.elementary-data.com/
    elementary:
      ## elementary models will be created in the schema 'your_schema_elementary'
      +schema: "elementary"
      ## If you dont want to run Elementary in your Dev Environment Uncomment following:
      # enabled: "{{ target.name in ['prod','analytics'] }}"

  From dbt 1.8 onwards, dbt depreciated the ability of installed packages to for different methods that fit your setup/where you want to host it in your production.

  - name: generate-elementary-report
      container:
        image: "{{inputs.parameters.elementary_image}}" ##pre-defined elemantary image in configmap.yaml
        command: ["edr"] ##run command for elemantary report
        args: ["report", "--profile-target={{inputs.parameters.target}}"]
        workingDir: workdir ##working directory
      inputs:
        parameters:
          - name: target
          - name: elementary_image
          - name: bucket
        artifacts:
          - name: source
            path: workdir ##working directory
      outputs:
        artifacts:
          - name: state
            path: /workdir/edr_target
            gcs:
              bucket: "{{inputs.parameters.bucket}}" ##here is the bucket that you would like to host your dashboard output
              key: path_to_key
            archive:
              none: {}

  By using the template above in our Argo workflows, we would create the Elementary HTML report and save it in the defined bucket. We can later take this report from your bucket and send it with your alerts.

  Now we know how we set up our report and hopefully know the basics of Elementary, next, we will check different types of tests and which test would suit the best in which scenario. Just jump into

• dbt Labs. (n.d.). dbt Documentation. Retrieved September 5, 2024, from
• dbt Labs. (2024). Upgrading to dbt v1.8. Retrieved September 5, 2024, from

on Medium, where people are continuing the conversation by highlighting and responding to this story.