Hey everyone! I ran into an issue while moving some data between io1 volumes, each of which is supposed to handle 20,000 IOPS, all on an r5.16xlarge instance. Ideally, we should have had plenty of IOPS and IO bandwidth available, but I noticed we were capped at 4000 IOPS, translating to about 530MB/s. According to the official documentation, the r5.16xlarge should easily provide a baseline throughput of 1700MB/s for a 128kb block size, and usually, that's what we see. However, today was a different story across two separate instances in the eu-central-1 region; performance was pretty awful, and it seemed to stay stuck at the 4k mark based on our monitoring graphs. Has anyone else experienced something similar? Could there be a specific issue in that zone?
1 Answer
From what I gather, the 4,000 IOPS performance you're hitting isn't too surprising given the specs. According to AWS documentation, io1 volumes can only provide maximum IOPS based on I/O size; for instance, at the max size of 256 KiB, you'd max out throughput at about 2,000 IOPS. So your 4,000 IOPS seems consistent if you're working with 128 KiB blocks. This might explain why you're seeing that limitation—it matches up with the performance characteristics AWS outlines.
That helps clarify things! I see how those specs can impact performance. I was previously using io2 where throughput can be specified, but we switched back to io1 since some in our team 'broke' io2 during their tests. I’ll have to push for us to look at gp3 next time.