Portal Home > Knowledgebase > Articles Database > rsync
Posted by TonyB, 10-07-2008, 07:00 PM So I have a client wanting to sync one of their backup servers and frankly I have no solution to this one without using the same software again on a second offsite server. I figured I'd try rsync doing this: Works fine on the standard files and some of the data files that are the incrementals. The problem is there are a few big binary files that are 15GB in size. These files will change just a few bytes and now it'll transfer the entire file again as it cannot just add on the changes or anything like that. This takes close to a day to complete making it not very useful as the backup system cannot make new backups while this is going on. Confirmation on this not being possible with rsync no matter what the parameters would be great. I think I'm trying to do the impossible but I always like opinions from others.
Posted by SysAdminMan, 10-07-2008, 08:53 PM Are these the same files that you are syncing that are changing a little - not new tar's or something that are a few bytes different to the old one? Also, do you know what versions of rsync you are using? As far as I know rsync should be able to just sync the deltas (changes)
Posted by TonyB, 10-07-2008, 08:57 PM They are binary files which makes me lead to believe things like rsync are useless and I'd need a block level system in order to just back up the changes. For anyone curious this backup software is R1Soft and the archiving module will not work due to the size of the files being even larger than the rsync solution. Rsync was the solution to not using another cdp server in order to mirror the changes. I was pretty sure rsync was not going to work but not my decision on what to use to backup this onsite cdp server. Last edited by TonyB; 10-07-2008 at 09:01 PM.
Posted by SysAdminMan, 10-07-2008, 09:34 PM I must say I always assumed this worked but had never tried properly so just gave it a quick go .. I rsync'd a 280MB gzip tar using "-avc --progress" and it took 10 mins - I then edited the file using "vi -b" and tried both adding and then deleting some data and transferring it again and got this - so definitely just the changes going over. That's with version 3.04 sending and 2.68 receiving. Maybe there's a size limit - could you try with a smaller file?
Posted by zuborg, 10-08-2008, 07:41 AM add '--inplace' option
Posted by foobic, 10-08-2008, 08:08 AM rsync does work on binary files and transfers the differences only. So, are you absolutely sure that the backup software is changing "just a few bytes"? There may be only a few bytes difference in the files being backed up but these could lead to changes that spread throughout the archive. This is certainly true for gzip compressed files and the answer there is gzip --rsyncable. IDK if anything similar exists for R1soft.
Posted by zuborg, 10-08-2008, 08:25 AM by default rsync create new file, syncronize it with source, and then replace old file by new one. --inplace option cause rsync to update stale file directly
Posted by foobic, 10-08-2008, 05:41 PM It seems to me that inplace would be risky on big files - if the transfer is interrupted (as it could easily be, since it may take hours) then your original file will most likely be corrupted. And in any case, why should it have any effect on the amount of data transferred?
Posted by david-r1soft, 10-09-2008, 12:28 PM Its unclear to me exactly what you are doing. It sounds for sure that you need to store an off-site copy of your local CDP backups right? And yes you are spot on that you don't want to read the raw R1Soft Disk Safe files while they are being used. That would be bad as its like trying to use rsync to copy SQL Server or MySQL database files while the database system is running. They would be corrupt. It is also not suprising that rsync is having issues with large files. Are you using the new R1Soft Archiving add-on? It should help you overcome this. You can choose to select what folders you want to export form the Disk Safe. You can export directly to an offsite server using: CIFS (windows share) / FTP / SFTP or to a local hard disk on the cdp server. The export dumps the file sin tar or zip format (you select) so tyour long term storage can be in an open format and you can also point tape backup software at your exported archive if you need to go to tape. You shouldn't need rsync with the archiving add-on. Best Regards, -David Wartell R1Soft Founder
Posted by AdmoNet, 10-10-2008, 12:31 PM Tony, I know exactly what you are trying to do. You're trying to sync files offsite from your R1Soft server? I would highly recommend using another R1Soft license on a remote server to backup the partition(s) your R1Soft server utilizes. Using rsync to transfer the CDP datasafes is a bad idea. R1Soft performs checksums on each file and, with large files, it is often faster to re-transfer all the files every night than performing checksumming/comparisons on each binary file. We've run into the same issue and that's why we choose R1Soft to backup our R1Soft For the price of one Linux agent license you can get another server license and backup your R1Soft server to a remote location. That's what we do. We always maintain backups of our backups. Make sure to perform a backup of the partition(s) holding the R1Soft server software (/usr/r1soft) and the partition(s) that you use for your storage disks. If you have any other questions on this process please let me know. Adam Ward
Add to Favourites 
Print this Article
