Hey everyone! I'm working on an eBPF-based dependency tracer that monitors file syscalls (openat, stat, etc.) and I'm running into kernel event drops when my load generator hits around 600,000 operations per second. The kernel keeps logging "lost samples" which means my userspace isn't draining the perf buffer fast enough. My setup:
- eBPF program attached to syscall tracepoints
- ~4KB events (includes 4096-byte filename field)
- 35MB perf buffer (system memory constraint - can't go bigger)
- Single perf reader → processing pipeline → Kafka publisher
- Go-based userspace application
The problem:At 600k ops/sec, my 35MB buffer can theoretically only hold ~58ms worth of events before overflowing. I'm getting kernel drops which means my userspace processing is too slow.What I've tried:
- Reduced polling timeout to 25ms
My constraints:
- Can't increase perf buffer size (memory limited)
- Can't use ring buffers (using kernel version 4.2)
- Need to capture most events (sampling isn't ideal)
- Running on production-like hardware
Questions:
- What's typically the biggest bottleneck in eBPF→userspace→processing pipelines? Is it usually the perf buffer reading, event decoding, or downstream processing?
- Should I redesign my eBPF program to send smaller events? That 4KB filename field seems wasteful but I need path info.
- Any tricks for faster perf buffer drainage? Like batching multiple reads, optimizing the polling strategy, or using multiple readers?
- Pipeline architecture advice? Currently doing: perf_reader → Go channels → classifier_workers → kafka. Should I be using a different pattern?
Just trying to figure out where my bottleneck is and how to optimize within my constraints. Any war stories, profiling tips, or "don't do this" advice would be super helpful! Using cilium/ebpf library with pretty standard perf buffer setup.
[link] [comments]