Lädt...

🐧 rsync over ssh much, much faster with "--delete"


Nachrichtenbereich: 🐧 Linux Tipps
🔗 Quelle: reddit.com

Say, I've been rsyncing some pretty substantial file sets (tens of millions of files in several dozen terabytes), and my curiosity is raised a bit by behavior when rsyncing file changes.

Most of the files were already transferred (netcat's super-great for saturating them network links!), but this filesystem on the sending server doesn't have proper checkpoints set up, so I have to let it crawl through everything to transfer new and updated files since the prior rsync. Not the best way, but whatever. On this 1gbps network connection from a single RAID6 server to a glusterfs cluster, it can take two to three days to look through everything with "rsync -aP". But when I "rsync -aP --delete", the total time goes down to less than eight hours.

What exactly is going on here? My first guess was that the "--delete" meant that fewer files are held in memory concurrently, but since I've run this a few times and not many files are being deleted on each run, that wouldn't seem to hold true. Is there a known explanation for why the option causes such a tremendous boost in performance?

submitted by /u/funtastrophe
[link] [comments] ...

matomo