-
zhenwei pi authored
On a server side, a server listens a local port and accepts lots of connections. To seperate histogram into views by each remote address, so add --byraddr to support this. Suggested-by:
Brendan Gregg <brendan.d.gregg@gmail.com> Signed-off-by:
zhenwei pi <pizhenwei@bytedance.com>
Unverified4cbcd9af
tcprtt_example.txt 5.71 KiB
Demonstrations of tcprtt, the Linux eBPF/bcc version.
This program traces TCP RTT(round-trip time) to analyze the quality of
network, then help us to distinguish the network latency trouble is from
user process or physical network.
For example, wrk show the http request latency distribution:
# wrk -d 30 -c 10 --latency http://192.168.122.100/index.html
Running 30s test @ http://192.168.122.100/index.html
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 86.75ms 153.76ms 1.54s 90.85%
Req/Sec 160.91 76.07 424.00 67.06%
Latency Distribution
50% 14.55ms
75% 119.21ms
90% 230.22ms
99% 726.90ms
9523 requests in 30.02s, 69.62MB read
Socket errors: connect 0, read 0, write 0, timeout 1
During wrk testing, run tcprtt:
# ./tcprtt -i 1 -d 10 -m
Tracing TCP RTT... Hit Ctrl-C to end.
msecs : count distribution
0 -> 1 : 4 | |
2 -> 3 : 0 | |
4 -> 7 : 1055 |****************************************|
8 -> 15 : 26 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 18 | |
128 -> 255 : 14 | |
256 -> 511 : 14 | |
512 -> 1023 : 12 | |
The wrk output shows that the latency of web service is not stable, and tcprtt
also shows unstable TCP RTT. So in this situation, we need to make sure the
quality of network is good or not firstly.
Use filter for address and(or) port. Ex, only collect local address 192.168.122.200
and remote address 192.168.122.100 and remote port 80.
# ./tcprtt -i 1 -d 10 -m -a 192.168.122.200 -A 192.168.122.100 -P 80
Tracing at server side, show each clients with its own histogram.
For example, run tcprtt on a storage node to show initiators' rtt histogram:
# ./tcprtt -i 1 -m --lport 3260 --byraddr
Tracing TCP RTT... Hit Ctrl-C to end.
Remote Address: = 10.131.90.16
msecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 2 |****************************************|
Remote Address: = 10.131.90.13
msecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 4 |************************** |
8 -> 15 : 6 |****************************************|
Remote Address: = 10.131.89.153
msecs : count distribution
0 -> 1 : 120 |****************************************|
2 -> 3 : 31 |********** | 4 -> 7 : 32 |********** |
Remote Address: = 10.131.89.150
msecs : count distribution
0 -> 1 : 12 |****************************************|
2 -> 3 : 12 |****************************************|
4 -> 7 : 9 |****************************** |
8 -> 15 : 3 |********** |
Remote Address: = 10.131.89.148
msecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 4 |****************************************|
....
Full USAGE:
# ./tcprtt -h
usage: tcprtt.py [-h] [-i INTERVAL] [-d DURATION] [-T] [-m] [-p LPORT]
[-P RPORT] [-a LADDR] [-A RADDR] [-b] [-B] [-D]
Summarize TCP RTT as a histogram
optional arguments:
-h, --help show this help message and exit
-i INTERVAL, --interval INTERVAL
summary interval, seconds
-d DURATION, --duration DURATION
total duration of trace, seconds
-T, --timestamp include timestamp on output
-m, --milliseconds millisecond histogram
-p LPORT, --lport LPORT
filter for local port
-P RPORT, --rport RPORT
filter for remote port
-a LADDR, --laddr LADDR
filter for local address
-A RADDR, --raddr RADDR
filter for remote address
-b, --byladdr show sockets histogram by local address
-B, --byraddr show sockets histogram by remote address
-D, --debug print BPF program before starting (for debugging
purposes)
examples:
./tcprtt # summarize TCP RTT
./tcprtt -i 1 -d 10 # print 1 second summaries, 10 times
./tcprtt -m -T # summarize in millisecond, and timestamps
./tcprtt -p # filter for local port
./tcprtt -P # filter for remote port
./tcprtt -a # filter for local address
./tcprtt -A # filter for remote address
./tcprtt -b # show sockets histogram by local address
./tcprtt -B # show sockets histogram by remote address
./tcprtt -D # show debug bpf text