Why do file copies top out around 110 MB/s on a 10 GbE network?

The disks and the copy program can also be the limiting factor. Both source and destination storage devices need to sustain the required read and write rates, and scp normally uses a single encrypted stream, which can become CPU-bound depending on the hardware and cipher. Test read and write performance separately, check CPU usage during the transfer, and compare scp with a tool or method that can use multiple streams. The SSD model information alone doesn’t prove that the complete storage path can sustain 10 GbE speeds.

Check the actual switch ports the servers use. On some M4300 models, most ports are 1 GbE while only a few uplink or designated ports support 10 GbE. The NIC can report a 10 GbE capability or link while the path still includes a gigabit port. Verify the negotiated speed for both switch ports and make sure both servers are connected to genuine 10 GbE ports.

First test the network independently with iperf3. Run it between the two servers, then repeat with the reverse-direction option. If iperf3 gets close to 10 Gb/s, the network is probably fine and scp, the CPU, or the storage is limiting the copy. Also remember that 10,000 Mb/s is about 1,250 MB/s before protocol overhead, so 110 MB/s is much closer to a real 1 Gb/s path than normal 10 GbE overhead.

A 10 GbE NIC report from ethtool only confirms the interface’s negotiated or configured link information; it doesn’t measure end-to-end throughput. Check the switch’s port counters and negotiated rate, then use iperf3 before investigating disk performance or scp tuning.