I've used this script to synchronize 2 public directorys inside 2 mongrel clusters. but you can use it to continously sync/mirror any 2 directorys on 2 different servers. note that it any change made will be committed instantaniously (or with a delay due to connection only) on the other directory. This uses inotify tools to know on any file create/change/delete and rsync to commit the changes on the other server.
How this works?
This script uses 2 simple things: inotify and rsync. inotify is a kernel level tool to ''sense'' accessing the hard disk, we use it here to detects of anything is writing inside the directory in server1, if it happens then it initiates rsync.
rsync is a great syncronization tool, we use it here to start transfering the modification from server1 to server2. modification includes creating/modifying/deleting anything inside the mentioned directory. then rsync daemon in server2 detects the incoming connection and commits the changes inside the directory.
untill this point, everything works fine. this is a master/slave configuration. server1 is the master that commits its changes in the slave (server2). But we need to make it master-master, meaning that any modification in any of the 2 servers should be commited in the other one. So we will have to make the same configurations on the other server, creating rsync daemon on server1 and intotify to detect changes on server2.
now if any of them changes, 2 connections happens. the first connection is as we described above, commiting the changes in the other server. the second connectin that happens is in the remote server (the slave in this case) rsync server will initiate writing in the directory watched by inotify. meaning that inotify will initiate another rsync connection to the other server again, but this time rsync will find no changes to synchronize so it will just verify that directorys are synchronized and then it will be terminated without modifying anything. Fortunately rsync uses little resources in such operation so I just ignored that second connection and just considered a nice thing that verification happens after synchronizing.
For sure you can extend this to multiple servers and make either single master/multi-slave or multi-master mirroring, and I guess a loop would work too (master1 commits in slave2, and master2 commites on slave3, and master3 commits in slave1). anyway i really doubt if a structure like that would be efficient anyway, but most likely it will work. I only tried it on 2 servers and it was working great. note that the script uses the rsync daemons so make sure that both servers can talk to each other on the port 873 (rsync port).
Howto implement this:
- First, you'll need to setup rsync daemons in the 2 servers as said here.
- then, install inotify tools:
apt-get install inotify-tools
- after that, all you need is to put this startup script in ''
/etc/init.d/mirroring''. Don't forget to modify it first, I'll explain the modifications below:
## Shell Daemon For: Mirroring Cache directory
## (poorly coded, quick and dirty, example)
case $1 in
exec 3>&- # close stdin
exec 2>&- # close stdout
exec 1>&- # close stderr
$NOHUP $0 run | $CRONOLOG $LOGFILE >> /dev/null &
/bin/kill $(cat $PIDFILE)
pgrep -f "$0 $1" > $PIDFILE
while [ true ]; do
$(inotifywait -q -e close_write -e delete -e modify --format "%f" /root/cache/)
( rsync -aWuq --no-motd --inplace --del --ignore-errors --force /root/cache/ rsync://remote_server/cache/ )&
echo "$0 [ start | stop ]"
- then you need to activate this startup script
chmod 700 /etc/init.d/mirroring
update-rc.d mirroring defaults
- go ahead and do the same on the other server, then start those daemons up
- You are gonna need to change 2 lines:
- The inotify line (line 22): you need to change the name of the directory where inotify watches the changes, and you may want to change the inotify flags themselves (see ''man inotifywait'' for more details).
- the rsync line (line 23): change the directory to sync from (the same as the directory inotify watches) and where it commits the changes (the remote rsync server). note that you should take care about the trailing slashes cause rsync understands ''/root'' in a different way that ''/root/'' (the first one means the directory and the second one means what's inside the directory, you'll have to change the pathes accordingally) .you may want to change the rsync flags too, the ones used here was for highly dynamic files with small sizes, and both servers are on a gegabit lan. Mostly you'll need to modify it, the most common use for backups is ''rsync -avz'' (check ''man rsync'' for more details).
now you have active mirroring works like magic, you shouldn't worry about copying things periodically anymore :)
thanks to this page, it was really helpful