Filter by Value, Not by Path
In this post, I use the example of a parcel-shipping gateway integrating with multiple carriers.
This example is purely hypothetical and chosen to clearly illustrate the technical idea.
Integrating with N shipping carriers often means dealing with N different JSON (if you’re lucky!) responses.
Each carrier’s responses can sprinkle customer PII into your logs—until your CISO sees it.
Most teams try to predict where the PII will be:
"recipient.email"for Carrier A
"ship_to.contact.email"for Carrier B
"error.stacktrace[0].ctx.email"when Carrier C blows up 😱
That game never ends.
Let’s flip the problem: we already know the values before we send any request.
So let’s redact by value, not by path.
Step 1 — Our Canonical Parcel Struct
defmodule Parcel do
@enforce_keys ~w(id name email phone)a
defstruct [:id, :name, :email, :phone, :address, :items]
end
Everything downstream is derived from %Parcel{}.
Step 2 — Collect the Secrets Once
defmodule Secrets do
@doc """
Accepts a Parcel struct and returns a list of
values that are secrets that we don't want to see in our logs.
"""
@spec from_parcel(Parcel.t()) :: [String.t()]
def from_parcel(%Parcel{} = p) do
[
p.name,
p.email,
p.phone,
p.address.street,
p.address.city,
p.address.postcode
]
|> Enum.reject(&is_nil/1)
end
end
Step 3 — A Drop-in Logger Helper
defmodule SafeLog do
require Logger
@doc """
Logs a payload at the given level, safely redacting any known secret values.
If the payload is a map, it is encoded as JSON before logging.
If the payload is a printable string, it is used directly.
"""
@spec scrub_and_log(
:debug | :info | :warn | :error,
String.t(),
map() | String.t(),
[String.t()]
) :: :ok
def scrub_and_log(level, message, payload, secrets)
def scrub_and_log(level, message, payload, secrets) when is_map(payload) do
payload
|> Jason.encode!()
|> redact_and_log(level, message, secrets)
end
def scrub_and_log(level, message, payload, secrets) when is_binary(payload) do
if String.printable?(payload) do
redact_and_log(level, message, payload, secrets)
else
Logger.warn("SafeLog attempted to log a non-printable binary, skipping.")
:ok
end
end
@spec redact_and_log(
:debug | :info | :warn | :error,
String.t(),
String.t(),
[String.t()]
) :: :ok
defp redact_and_log(level, message, text, secrets) do
redacted =
secrets
|> Enum.uniq()
|> Enum.sort_by(&String.length/1, :desc)
|> Enum.reduce(text, fn secret, acc ->
String.replace(acc, secret, "[FILTERED]")
end)
Logger.log(level, fn -> "#{message} #{redacted}" end)
end
end
Step 4 — Use It in an Adapter
defmodule ShippingExpressAdapter do
@doc """
Creates a shipping label by sending a request to the ShippingExpress API.
Logs the request and response with PII safely scrubbed out,
using the known secret values extracted from the Parcel struct.
"""
@spec create_label(Parcel.t()) :: {:ok, map()} | {:error, any()}
def create_label(%Parcel{} = parcel) do
secrets = Secrets.from_parcel(parcel)
request = build_request(parcel)
SafeLog.scrub_and_log(:debug, "ShippingExpress request ⇢", request, secrets)
with {:ok, raw_resp} <- HTTPClient.post(url(), request),
{:ok, resp} <- Jason.decode(raw_resp) do
SafeLog.scrub_and_log(:debug, "ShippingExpress response ⇠", resp, secrets)
parse_label(resp)
end
end
end
No carrier-specific filter modules.
If ShippingExpress decides to throw "InternalError: phone=123-456" somewhere, it still gets nuked.
Considerations
| Concern | Mitigation |
|---|---|
| Large payloads? | Replacing many secrets in large payloads could introduce small slowdowns. Benchmark if you process very large logs. |
| Secrets overlapping or partial matches? | Deduplicate secrets (Enum.uniq/1) and sort them by descending length (Enum.sort_by/2) before replacing. This ensures longer secrets are redacted first, avoiding partial leaks (e.g., preventing "[email protected]" from becoming "[FILTERED][email protected]"). |
| Encoding issues | If sensitive values are encoded (e.g., Base64) before being logged, simple string replacement won't find them. Make sure sensitive fields are logged in plain form, or exclude them entirely. |
If you can think of scenarios where this approach might fail, I'd love to learn from your experience.
Conclusion
Third-party APIs are unpredictable — but the data you send is something you control.
Building your filters around known values, rather than brittle paths, can make your systems more robust and easier to trust.