I'm copying a file larger than 10 GB from server A to server B with SCP. Both servers have 10 GbE network cards, and ethtool reports a 10,000 Mb/s link on each interface. Both links are up and the servers use static IP addresses. They connect through a Netgear M4300 switch. However, the transfer consistently reaches only about 110 MB/s, which is roughly the practical limit of a 1 GbE connection. What should I check to determine whether the bottleneck is the switch, network path, storage, or SCP itself?
4 Answers
110 MB/s is about 880 Mb/s, which is very close to a saturated 1 GbE connection after protocol overhead. The reported NIC speed only confirms the interface’s negotiated physical link; it does not prove that the complete path between the servers is 10 GbE. Check the switch port status, cabling, transceivers, and any intermediate connections.
Check the actual switch ports the servers are connected to. The M4300 model in question may have mostly 1 GbE access ports and only a few 10 GbE uplink ports. The NIC can report a 10 GbE capability or link, but traffic will still be limited if the servers are plugged into gigabit ports or if there is a 1 GbE link anywhere between them. Also verify the negotiated speed on the switch itself, not only with ethtool.
Storage and SCP can also be limiting factors. The source and destination drives should be benchmarked independently, and system CPU usage should be watched during the transfer because SCP encrypts the stream and generally uses one main data stream. Since both drives appear to be SSDs, test with iperf3 first and then compare tools such as rsync or a faster parallel transfer method.
Start by testing the network separately with iperf3. Run it between the two servers, then repeat with the reverse-direction option. If iperf3 reaches close to 10 Gb/s but SCP remains around 110 MB/s, the network is probably fine and the limitation is elsewhere—such as disk performance, CPU encryption, or SCP’s single-stream behavior.
That may be the issue. I’m going to inspect the switch model and port configuration more closely.