`get_remote_dump_and_load` is slow
Created by: qris
It takes about 10 minutes to load a reasonably-sized remote dump (1.1 MB from washwatch staging server).
Disabling the rsyncable code makes it about 10x faster (about 30 seconds). I'm not sure what the impact on the rsyncability of the file is, but this can equate to a big time saving.
This is the code in question, in dye/tasklib/database.py
:
# this option will mean that there will be one line per insert
# thus making the dump file better for rsync, but slightly bigger
if for_rsync:
dump_cmd.append('--skip-extended-insert')
It might be useful to have a command-line option to enable this manually; for example such dumps are easier to manipulate for loading into a different database engine. But I don't really see a need for it in get_remote_dump_and_load
.