I need to load a large set of S3 records into a DynamoDB table that's already serving traffic. What does the managed bulk put feature handle, and what should I watch out for?
How does DynamoDB bulk import from S3 work for an existing table?
Answers
It's a managed write job, not a way to fetch S3 objects through DynamoDB. You point it at files in S3 and it puts their records into an existing table. That can replace a fair bit of custom ingestion orchestration for backfills or migrations.
The job uses dedicated capacity, so its writes don't compete with your application's reads and writes. They also don't count toward the table's CloudWatch read and write metrics, so those metrics won't show the import traffic.
Reply to this answer
Pay close attention to overwrites if the table is live. Each record can include a PutItem condition expression. For example, attribute_not_exists on the partition key gives you an insert-only write, while a version check can stop an older export from replacing a newer item.
A condition that doesn't pass is a skip, not a job failure. The report goes to an S3 bucket you choose and includes records that weren't written and why. For condition failures, it shows the attempted item alongside the item already in the table.
Reply to this answer
The input format is worth checking before you plan around it. The files need to be line-delimited DynamoDB JSON or Amazon Ion. Gzip and zstd compression are supported, so ordinary JSON lines may need conversion first.
Reply to this answer
There are Standard and Accelerated tiers. Standard is cheaper for background work. Accelerated is intended for jobs with a deadline.
In US East, N. Virginia, the stated rates are $0.10 per GB for Standard and $0.20 per GB for Accelerated, with a 1 KB minimum per item. The table and each index are metered separately. As an example, 100 million 1 KB items with no indexes would cost $10 on Standard or $20 on Accelerated at those rates.
Don't treat the tier as a fixed throughput guarantee. Data shape matters, including skew in table or index partition keys. The example given for Accelerated is 100 million 1 KB items in under an hour, while Standard takes longer.
Reply to this answer
Backfills are an obvious fit, but it's also useful for moving an export from an old system or restoring data after an outage. The key distinction is that the destination is an existing table, so the application can keep using it during the load.
Reply to this answer
Add your answer
Answers are reviewed before publication.
_c36~mivzaro.8:
That report sounds worth checking even when the job completes successfully, especially if skipped records matter to the migration.