I needed to be able to bidirectionally sync the same data across four machines. Two windows hosts (Home desktop, Laptop) and two Linux hosts (File Server, Backup File Server). My research lead me to Unison and that's what I'm going to give a go with. First we will configure the two Linux hosts and then “Add” the windows hosts into the mix. I needed this to run as part of an automated process nightly so I created SSH keys for passwordless access. Also keep in mind when Syncing with several hosts you need a 'primary' host that each other host will sync to. For this document I will call that the 'primary' and I'll mention this more as we go though.
These are not hard-set requirements. It's just so you know what I used for my setup. * Windows XP (For Home desktop) * Windows Vista (For Laptop) * Gentoo Linux (For both linux hosts) * Unison * OpenSSH (On linux hosts) * Putty (On Windows hosts)
For unattended nightly syncing we need a passwordless SSH connection. So.
You can have this run as root, your user account, your grandma's account. But I want it running as the user 'unison' and as such I created that user.
# adduser unison
# mkdir /home/unison
# chown unison:unison /home/unison
I used the default folder path for the file and no password (just hit enter twice)
# cd /home/unison # su unison ~ $ ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/unison/.ssh/id_dsa): Created directory '/home/unison/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/unison/.ssh/id_dsa. Your public key has been saved in /home/unison/.ssh/id_dsa.pub. The key fingerprint is: db:f7:7c:55:bd:b2:38:4a:9b:2a:e9:2b:4c:77:b7:c3 unison@impasse
Now modify the SSH config file (or create it) to contain the following
~ $ vim .ssh/.config
Only lines needing changed are the first two. Enter the IP/Name of the remote host and what user they will login as. As you see I am connecting to “primary” as “unison”.
Host primary User unison Compression yes Protocol 2 RSAAuthentication yes StrictHostKeyChecking no ForwardAgent yes ForwardX11 yes IdentityFile /home/unison/.ssh/id_remotehost_dsa
Now we need to copy the public key to the primary server so it will accept it. You can do this as root if you wish (especially if you have not created the unison user on that server yet) I am going to stick it in the temp folder for now and grab it here in a minute after setting up the unison user on that server.
~ $ scp .ssh/id_dsa.pub root@primary:/tmp/
Lets create that 'unison' user here as well. If you are using a different account just keep it in mind, and skip this.
# adduser unison # mkdir /home/unison # chown unison:unison /home/unison
While we're still root, lets move that public key over to unisons user folder and give unison control over it.
# mv /tmp/id_dsa.pub /home/unison # chown unison:unison /home/unison/id_dsa.pub
Now lets become unison.
# cd /home/unison # su unison
Now if this is a new user if probably doesn't have the SSH folders created. An easy way to do that would be to just use ssh and connect someplace.
~ $ ssh backup -l root
Just login and logout. Now .ssh should exist and backup should be added to your known_hosts file. Now lets stick that public key into the .ssh folder with a halfway useful name ~ $ mv id_dsa.pub .ssh/backup_unison.pub
Now copy that file to the authorized_keys file - this file probably does not exist so this method is safe. If you're already using this user and this file then you need to appended the contents of our new public key to the authorized_keys file. ~ $ cp .ssh/backup_unison.pub .ssh/authorized_keys
Now you can test this. Go back to the backup server logged in as unison and attempt to ssh to the primary server. It should connect without prompting for a password.
Don't forgot about file permissions. Whatever user you setup to have unison run as needs read/write access to the files. If you want to setup some connections to have read-only access then you could control that with permissions as well. For simplicity I'm going read/write with the unison user. But! I want my local user to have read/write over those files as well. So lets give them user ownership of my local account, and group ownership by the unison account with 660 permissions (770 for folders). So, below shows how to set permissions and ownership for a folder and all of it's subfolders. This will give 770 to directories and 660 to all files.
# chown -R bmarriner:unison . # find . -type d -exec chmod 770 {} \; # find . -type f -exec chmod 660 {} \;
Okay! Now back on the Backup Server. Lets test unison :) login as unison or su to unison - whichever. (I am su to unison, as I didn't give unison a password)
~ $ unison /silo/backup/media/videos ssh:primarymedia/videos -batch
This will take much longer the first time around. For reference, my primary server is Pentium D 2.8 desktop that had 'several' applications running while this took place. The backup server is a Pentium D 3.2, raid 0 server. The systems were on a GB network. The folder I synced was 41GB and contained a total of 1109 files. Unison spent 14 minutes “Waiting for changes from server” it then took a little over an hour to propagate the changes. An immediate subsequent run of unison completed in a few seconds.
But that gives you the syntax for the command. You can login to the primary server and check the running processes and you should see unison doing something. Next will be to create a crontab for the unison user that executes the above command for each folder you want kept in sync.
~ $ crontab -u unison -e
The below entry will three unison sync commands, the first at midnight, then 1am, then 2am - everyday. Notice the first line is just a command and explains the first five fields of each line.
# Minute (0-59) Hour (0-23) Day of Month (1-31) Month (1-12 or Jan-Dec) Day of Week (0-6 or Sun-Sat) Command 0 0 * * * unison /silo/backup/media/videos ssh:primarymedia/videos -batch 0 1 * * * unison /silo/backup/media/images ssh:primarymedia/images -batch 0 2 * * * unison /silo/backup/media/audio ssh:primarymedia/audio -batch
Now unless you got an error somewhere. You should be done with the two linux hosts. The backup server should every night contact the primary server and sync. You could change this to every 10 mins if you wish. Just depends on the size of the data and the importance of them being in sync. For me this part is just acting to backup data and it's about 250G of data.
I only sync from my Windows machines to linux systems. So I don't need a SSH server -running- on the Windows system. What we will do is use PuTTY and Unison. We'll create a DSA key with putty so we can make a passwordless connection. Then create a dos script that will sync specific folders..
Windows PuTTY
Good instructions on how to setup PuTTY can be found at https://www.daml.org/lab/puttycvs.html (Dead link, 2006-07-19. Temporary alternative at http://www.phileplanet.com/archives/2006/08/how-to-setup-ssh-on-windows-xp/ )
These instructions use the DSA option when generating keys, but PuTTY developers recommend to use SSH-2 RSA [1]
Generate a new key with the PuTTYgen app:
Upload the public key to your Dreamhost home directory. After you've generated the key and copied it to the server, you'll need to import and append the key to authorized_keys with the following command. Use » instead of > to append to the file instead of overwriting the contents. (If the directory ~/.ssh does not already exist use webftp to create it and change permissions chmod to 755. After creating the file authorized_keys change permissions chmod to 644.)
ssh-keygen -i -f key.pub » ~/.ssh/authorized_keys
the.. @ is.. important! connect.bat @plink.exe -i private.ppk unison@sqls.net unison -server
sync.bat unison localfolder ssh:server/remotefolder -sshcmd connect.bat == Add the Windows XP Host (Home Desktop) == == References == * http://www.davz.net/static/howto/sshkeys * http://www.endperform.org/2007/07/11/howto-backup-and-sync-with-unison/ * http://www.minezone.org/wiki/MVance/FileSyncingHowTo * http://www.buessow.com/unison/