I'm transferring data from a local PostgreSQL database to a remote server with DBeaver. About 80,000 simple rows took 25 minutes, and I may need to move up to a million. What would be faster?
Is there a faster way to move local PostgreSQL data to a remote server?
Answers
For selected rows, try streaming COPY over SSH. A source command such as COPY (SELECT * FROM my_table WHERE foo = TRUE) TO STDOUT can feed a remote psql session running COPY target_table FROM STDIN.
That avoids creating an intermediate file and lets PostgreSQL handle the bulk transfer. Make sure the selected columns match the destination columns and that the destination table already exists. SSH compression may help if network bandwidth is the bottleneck.
Reply to this answer
For whole tables, pg_dump is worth a look. You can pipe a plain-text dump into psql on the destination, or use a custom-format dump with pg_restore if you want more control over the restore.
Check what you're dumping and restoring before running it against production. Schema objects and existing data can affect whether the restore succeeds or does what you expect.
Reply to this answer
Twenty-five minutes for those rows sounds slow, but check the transfer settings before blaming any one thing. Small insert batches, per-row work, network latency, indexes or constraints on the target could all contribute. Try a larger batch and compare it with a COPY load.
Reply to this answer
File-based transfer isn't necessarily a bad option, either. You can export with psql's client-side \copy, compress and transfer the file, then load it with \copy on the destination. Just remember that the file path is on the machine running psql, not necessarily the database server.
Reply to this answer
Add your answer
Answers are reviewed before publication.
h19~wrstrobe_638:
Parallel restore can help when there are multiple tables or other independent objects to restore. It doesn't automatically split one table's data load across several workers.