Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. sitting in a repo I was poking around, and it broke my brain a little.
So I read the source, dragged my devops brain through it, and now I'm going to explain it to you the way I wish someone had explained it to me.
No kernel PhD required.
Some puns required.
The problem: packets don't have a call stack you can just console.log
As a web dev, when something's broken, I add a console.log, or I set a breakpoint, or worst case I print my way to enlightenment.
The request comes in, flows through my middleware, my handlers, my ORM, and I can trace every step because it's all just... functions calling functions, in my process, in my language.
A packet inside the Linux kernel does not work like that.
It gets handed from function to function across the network stack (ip_rcv, tcp_v4_rcv, netfilter hooks, qdiscs, the works), it can get cloned, NAT'd, encapsulated in a tunnel, handed to a virtual ethernet pair, sent through an eBPF program of its own, and dropped at literally any of a few thousand points, often silently.
There's no stack trace.
There's no packet.log().
The tool you reach for as a network person is tcpdump, which is great for seeing a packet arrive and a packet leave, but useless for seeing what happened to it in between, inside the kernel, especially when it never arrives at all.
That's the gap pwru (packet, where are you?) fills.
It's basically an eBPF-powered console.log for the kernel's packet path.
Attaching a debugger to (checks notes) every function in the kernel
Here's the first thing that made my jaw drop reading this codebase.
Instead of the pwru authors hand-picking a list of "interesting" kernel functions to trace (which is what I assumed, naively), they let BTF (BPF Type Format, basically the kernel's own type metadata) do the work for them.
In
Here's roughly what that discovery-and-attach pipeline looks like:
to translate cBPF into modern eBPF instructions, and then splices those raw instructions directly into the loaded eBPF program, at a specific marker function it planted for exactly this purpose:
for idx, inst := range program.Instructions {
if inst.Symbol() == "filter_pcap_ebpf"+suffix {
injectIdx = idx
break
}
}
In bpf/kprobe_pwru.c there's a deliberately empty, __noinline placeholder function called filter_pcap_ebpf_l3 / _l2 whose only job is to exist as an injection point:
static __noinline bool
filter_pcap_ebpf_l3(void *_skb, void *__skb, void *___skb, void *data, void* data_end)
{
return data != data_end && _skb == __skb && __skb == ___skb;
}
pwru finds that function in the compiled instruction stream and literally performs bytecode surgery, cutting it out and stitching in your compiled filter instead, before the program is even loaded into the kernel.
It's like if your reverse proxy let you write location blocks in nginx syntax, but under the hood it was actually recompiling and hot-patching the C binary per request. Deranged. Also very cool.
The hard part: knowing it's "the same packet"
Okay here's the thing that actually made me appreciate why this tool needed to exist and isn't just "kprobe everything and print."
A packet does not keep a fixed identity as it flows through the kernel.
- NAT rewrites its IP/port, so your 5-tuple filter (
host 1.1.1.1) can stop matching halfway through the journey, right after the thing you were trying to debug happens. - It can get cloned (
skb_clone) or copied (skb_copy) for things like local delivery plus forwarding. - It can cross a veth pair into another network namespace, get handed to an XDP program, or get bridged, where the original
sk_buffmight legitimately get freed and a new one used to represent conceptually "the same" packet.
So pwru maintains a small state machine to keep following a packet even after your original filter would technically stop matching.
Once a packet matches your filter once, its pointer gets remembered in a skb_addresses BPF hash map (bpf/kprobe_pwru.c), and every subsequent kprobe hit checks that map first, before even bothering with your filter again:
if (cfg->track_skb && bpf_map_lookup_elem(&skb_addresses, &skb_addr)) {
tracked_by = _stackid ? TRACKED_BY_STACKID : TRACKED_BY_SKB;
goto cont;
}
When the skb gets cloned, an fexit probe on skb_clone/skb_copy propagates the "yes, we're tracking this one" flag from the old pointer to the new one.
When the kernel finally frees it (kfree_skbmem), pwru cleans up the tracking entry so the map doesn't grow forever.
For the gnarlier case, like bridging, where the pointer identity is genuinely lost, there's a --filter-track-skb-by-stackid mode that fingerprints by call stack instead of pointer, so it can pick the "same" packet back up under a new address.
This is basically the eBPF equivalent of trying to correlate a request across microservices without a trace ID: you're stitching identity back together from context clues because nobody handed you a clean handle to hold onto.
Turning "packet vanished" into an actual reason
The best part for someone coming from iptables-land: when a packet dies, pwru doesn't just say "the trace stopped here, good luck."
It hooks kfree_skb_reason and decodes the kernel's skb_drop_reason enum, so you get an actual human string like SKB_DROP_REASON_NETFILTER_DROP instead of silence.
That's the difference between "curl hung and I have no idea why" and "curl hung, and here is the literal function and the literal drop reason, with a timestamp."
The
Why this clicked for a devops brain
If you've ever debugged distributed systems, this whole tool is basically:
Structured, filtered logging (pcap-filter syntax you already know), applied to
Every relevant call site automatically (via BTF introspection instead of a hand-maintained list), with
Identity tracking across mutation and forking (NAT, clone, tunnel, bridge) so your trace doesn't just stop the moment the packet changes shape, and
A resolved root cause at the end (the drop reason) instead of a shrug emoji.
That's a debugging methodology, not just a tool.
It's the "attach a debugger to the whole system, but keep it filtered so it doesn't drown you, and correlate identity across boundaries" pattern, and it happens to be implemented with kprobes and BTF instead of trace IDs and Jaeger.
Here's the shape of what happens end to end, from the kprobe firing to a line on your screen:
and
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
Free, Micro AI Code Reviews That Run on Git Commit
| | | | | |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
·
SOCIAL SHARE CARD GENERATOR