How should I troubleshoot an AWS CLI transfer that seems stuck?

First check whether it is truly stalled or just spending a long time enumerating files or waiting on the source storage. Watch disk activity, network traffic, CPU use, and the CLI output, and try a representative smaller directory. Millions of small files can make filesystem scans and per-file requests the bottleneck, even when the total size is not especially large.

Test a few CLI concurrency settings on a sample and compare results rather than assuming more parallelism will help. Keep logs, and verify the destination with a follow-up listing or sync check. For a one-off transfer, AWS DataSync is also worth evaluating because it is designed for managed file transfers.

For lots of objects, I’d benchmark s5cmd or rclone against the AWS CLI using the same subset of files. They can offer a different transfer workflow, but no tool can make millions of filesystem lookups and network operations disappear. Measure scan time, throughput, retries, and resource use, not just the final copy speed.

Use a tool that can resume safely and record progress. Before running a full job, confirm how it handles existing objects, metadata, and deletions so a retry does not do something surprising.

DataSync is the first alternative I’d investigate for this scale, especially if you want a managed process with monitoring and retry behavior. Check its setup requirements and pricing for your environment, then run a small test before committing to the full dataset.

If you stay with the CLI, split the work into stable batches and save enough progress information to restart cleanly. A hand-written script can add overhead or accidentally serialize work, so compare it with the built-in tools on a realistic sample.

There may be two separate bottlenecks here: reading and walking the source filesystem, and uploading objects. If the source has high latency or weak metadata performance, increasing upload concurrency may not help much. Check whether the machine can enumerate and read the same files quickly, and whether network use stays active during the apparent freeze.

If the data is already packaged into larger files and preserving individual object names is not required, archiving can reduce per-file overhead. But that changes how the data is stored and accessed, so it is only useful when that tradeoff fits.

Don’t tune blindly. Run a small but representative transfer, capture CLI diagnostics, and note whether the delay happens during listing, file reads, or uploads. Then change one setting at a time and compare. If the job must be reliable across interruptions, prioritize resumability and verification over a setting that only improves peak speed.