Why do file copies top out around 110 MB/s on a 10 GbE network?
Asked by
83_bushel
I'm copying a file larger than 10 GB from server A to server B with scp. Both servers have 10 GbE NICs, and ethtool reports a 10,000 Mb/s link with static IP addresses configured. However, the transfer consistently reaches only about 110 MB/s, which is roughly gigabit Ethernet speed. The servers are connected through a Netgear M4300 switch. What should I check to find the bottleneck?
Answers
A 10 GbE NIC report from ethtool only confirms the interfaceโs negotiated or configured link information; it doesnโt measure end-to-end throughput. Check the switchโs port counters and negotiated rate, then use iperf3 before investigating disk performance or scp tuning.
Reply to this answer
First test the network independently with iperf3. Run it between the two servers, then repeat with the reverse-direction option. If iperf3 gets close to 10 Gb/s, the network is probably fine and scp, the CPU, or the storage is limiting the copy. Also remember that 10,000 Mb/s is about 1,250 MB/s before protocol overhead, so 110 MB/s is much closer to a real 1 Gb/s path than normal 10 GbE overhead.
Reply to this answer
Check the actual switch ports the servers use. On some M4300 models, most ports are 1 GbE while only a few uplink or designated ports support 10 GbE. The NIC can report a 10 GbE capability or link while the path still includes a gigabit port. Verify the negotiated speed for both switch ports and make sure both servers are connected to genuine 10 GbE ports.
Reply to this answer
The disks and the copy program can also be the limiting factor. Both source and destination storage devices need to sustain the required read and write rates, and scp normally uses a single encrypted stream, which can become CPU-bound depending on the hardware and cipher. Test read and write performance separately, check CPU usage during the transfer, and compare scp with a tool or method that can use multiple streams. The SSD model information alone doesnโt prove that the complete storage path can sustain 10 GbE speeds.
Reply to this answer
Add your answer
Answers are reviewed before publication.
83_bushel: That may be the issue. Iโm going to inspect the switch model and port configuration more closely.