Stop using tcpdump for packet capture! Kyanos helps you troubleshoot network issues in seconds.

Why I Developed Kyanos

Have you ever faced a problem like this?

You just joined a new company and are responsible for a backend service. Everything is going smoothly...

On a Friday afternoon at 5 PM, with just an hour left before you clock out, suddenly, trouble hits. Your upstream is angrily asking you 😡 why your interface is timing out.

You panic 😩 but try to stay calm and check the monitoring, only to find that your service's interface latency is normal.

Just as you’re about to argue back, you suddenly realize that the company's monitoring only tracks server-side application latency, with no visibility into kernel and network latency!

As a result, neither side can convince the other 👿, leading to a blame game and the issue remains unresolved...

Conversely, if you experience a timeout with downstream interfaces but their monitoring shows no issues, a new blame game begins, only this time you’re on the other side...

So, how can we solve this problem?

1. Use tcpdump for packet capture! However, the biggest drawback of tcpdump is its troubleshooting process: it is too slow.
   
   For example, if you need to troubleshoot an issue in production:
   
   a. First, you need to install tcpdump in the production environment 💤,
   
   b. Then find the IP and port of the problematic server 👁👁
   
   c. Write a filter expression yourself (which you might have forgotten how to write, spending 5 minutes searching~😂)
   
   d. After much effort, you finally download a several hundred MB pcap file from the production machine 💤
   
   e. Install a tcpdump client (if you haven’t already) 💤
   
   f. Load the pcap file and watch your CPU fan spin wildly 😡
   
   g. After a few minutes, you squint 👁👁 to manually verify whether the interface you want is in this pcap file, only to find that it isn’t due to too much irrelevant information 🤡...
   
   h. Then you start capturing packets again, checking if your tcpdump command was correct... 😔

2. Ask the operations team to install monitoring tools! Fortunately, there’s a powerful technology called eBPF that allows for deep kernel-level data collection and full-stack observability. Modern tools like Skywalking, Pixie, and Deepflow offer impressive products.
   
   But this isn’t easy! These monitoring tools either require a high kernel version (5.x), only monitor K8s traffic, or generate TBs of monitoring data per hour, needing heavy storage dependencies.

So, is there a lightweight, compatible with older kernel versions, and highly efficient tool for troubleshooting network issues?

Kyanos is here!

What is Kyanos

Kyanos is an open-source command-line tool 👉 Kyanos repository 👈 that supports kernels as low as version 3.10 and runs without any additional dependencies. All you need to do is download the executable file (Release download link).

So, what can Kyanos do?

Before diving into the details, let’s see an example of what Kyanos can do.

You don’t need to understand any filter syntax. Just run a single command (kyanos stat http ...), and Kyanos will find the slowest HTTP requests and provide details on their latency (imagine how much time it would take with tcpdump):

Here, one command finds several of the slowest requests.

If you want to print the content of the request and response, you can do it like this:

As you can see, the request and response content is printed directly.

Kyanos is not only easy to install, but also aligns perfectly with our troubleshooting needs. Unlike packet-based tools, which are too granular and often contain a lot of irrelevant information, Kyanos is completely non-intrusive and works at the application layer, filtering out unnecessary data and retaining only the most valuable information for troubleshooting.

So, what exactly can Kyanos do? The main features of Kyanos are:

1. Capturing request and response data for various protocols (HTTP, MySQL, Redis, etc.).

2. Performing higher-dimensional analysis of the captured traffic through aggregation.

I won’t go into too much detail here—let’s dive straight into examples! 🤞

Overview - stat

While the watch command offers a granular analysis perspective, stat provides more flexible and higher-dimensional analytical capabilities. In simple terms, it aggregates request and response metrics by certain dimensions. For example, if you want to know which remote IPs have the slowest HTTP interfaces, you can aggregate requests and responses by the same IP to find the slowest remote IP.

So it can answer questions like:

Help me find out which remote IPs on this machine have the slowest HTTP interfaces?

One command does it all: ./kyanos stat http --side client -i 5 -m n -l 10 -g conn. This command outputs the top 10 HTTP connections with the longest network latency every 5 seconds. The output looks like this:

The result shows two connections along with latency information such as avg, max, and pxx.

My machine has very high outbound traffic. Which HTTP requests are causing this?

One command does it all: ./kyanos stat http --side client -i 5 -m p -s 10 -g none. This command outputs the top 10 HTTP request responses with the largest response sizes every 5 seconds. The output looks like this:

The results include average, maximum, and pxx values of response sizes, as well as the HTTP request path information.

General Steps for Using the stat Command

1. First, determine the metrics you're interested in and specify them using --metrics. Kyanos supports aggregation for the following metrics:

1. Next, specify the aggregation dimension using --group-by or -g. For example, if you're interested in whether different remote services provide different service quality, you can specify -g remote-ip. This will aggregate the request and response statistics by different remote IP addresses, making it easier to identify which remote service is causing issues. Kyanos supports the following aggregation dimensions:

For a more detailed usage guide, see the GitHub page: Kyanos GitHub

The stat command aggregates the results observed by watch according to the user-specified aggregation dimension (--group-by), and finally outputs according to the user’s most concerned metrics (--metrics).

Using tcpdump would take so much time—with Kyanos, it’s done in seconds!

Conclusion

Although kyanos actually can't completely replace tcpdump just yet. For instance, Kyanos currently supports a limited set of application-layer protocols, and support for more complex network environments still needs improvement. However, it certainly can enhance the efficiency of troubleshooting in our development process. 👍

Why am I so confident? Because I have truly solved problems using Kyanos. If you've read my articles, you might know that I work with Redis. Kyanos helped me resolve a very peculiar issue where the Redis client was timing out but the Redis server showed no anomalies (I will be publishing an article about this process soon, so stay tuned if you're interested). I was able to diagnose the issue within 30 minutes, and Kyanos proved its worth. That's why I am confident in open-sourcing it, and I believe Kyanos can help others too!

Finally, could you please give a star to encourage me? 👉 Kyanos