Network connection slowness is one of the interesting scenarios I came across. It might help someone.
My application servers are placed in one VLAN. And backup server placed in another VLAN. Since it is LAN bandwidth supposed to be 40-50 Mbps. But the connection between both application and backup server were at ~100KBPs transfer rate. I have checked router, firewall and VLAN settings. Everything looks good. No ware bandwidth limit identified. So, how do I fix network connection slowness issue?
Symptoms
Using tcpdump capture the network data flow and analyse it using wireshark utility.
# yum install tcpdump wireshark # tcpdump -nqi <interface-name> -s0 -w /tmp/client-issue.pcap (from another window start transfer some data) # scp test_data root@10.20.90.40:/tmp # tshark -r client-issue.pcap -Y "tcp.stream eq 5 and tcp.analysis.retransmission"
Are you seeing more number of retransmission attempts like this? If so you are at right place. This post would help you in resolving network connection slowness or intermittent disconnection.
5724 60.132229 10.20.31.137 -> 10.20.90.40 SSHv2 1434 0.000002000 Client: [TCP Retransmission] , Encrypted packet (len=1368) 5725 60.132234 10.20.31.137 -> 10.20.90.40 SSHv2 1434 0.000005000 Client: [TCP Retransmission] , Encrypted packet (len=1368) 5726 60.132237 10.20.31.137 -> 10.20.90.40 SSHv2 1434 0.000003000 Client: [TCP Retransmission] , Encrypted packet (len=1368)
Disable TCP SACK (Selective Acknowledgements)
Selective acknowledgement (SACK) defines how the host should send acknowledgement once the data received from sender. A basic TCP ACK says “I received all bytes up to X” But Selective ACK allows you to say “I received bytes X-Y, and V-Z.”
For example, sender sends 1000bytes of data. First 200bytes received. Next 201-400bytes lost due to some issue. Again rest sequence of data received by receiver. Here if you use
- Basic TCP ACK – client acknowledges like received only till 200 bytes. But actually server had sent all 1000 bytes. Now server has to resend 201-1000 bytes again.
- TCP SACK – client acknowledges like received 0-200 bytes and 401-1000 bytes. Sender understands and resends only 201-400 bytes.
Tcp_sack enabled by default. Enabling this sysctl parameter supposed to improve performance. But here my data packets will cross over many network devices. Some of them are does not support selective acknowledgement (SACK). So did disabled tcp_sack parameter.
#echo 0 > /proc/sys/net/ipv4/tcp_sack
To disable permanently
#echo “net.ipv4.tcp_sack = 0” >> /etc/sysctl.conf #sysctl –p
Disable TCP window scaling
In TCP communication, the receive window (RWIN) is the amount of data receiver can receive without acknowledging the sender. This is to ensure reliable communication and not to overwhelm receiver by keep sending data.
For example, a sender receives a TCP_ACK which acknowledges byte 1000 and specifies a receive window of 10000 (bytes), the sender will not send packets after byte 14000. Post receiving next acknowledgement sender sends further data. The 10000bytes receive window always maintained. The maximum receive window supported is 65535.
Assume you are in high bandwidth networks. To utilize full network space receive window size need to be increased beyond 65535. But some router/firewall do not handle this TCP scaling properly causes slowness in data transfer rate or intermittent disconnection.
TCP window scaling is a sysctl parameter used to increase “receive window” size. This feature made available since Linux kernel 2.6.8. Disable TCP window scaling and try establishing network connection.
#echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
To disable permanently
#echo “net.ipv4.tcp_window_scaling = 0” >> /etc/sysctl.conf #sysctl –p
The above two parameters will be enabled by default in RHEL7, SUSE 12 and other latest distributions. Both the parameters exist for better performance but it depends on what network you are connected with. Post applying above solution try sending data using SCP. Is there any improvement?
Have any one come across this situation? Any comments post here.
References
If you are looking for more details on any of above please refer these pages.