Rsync

Hi,

I have 2 servers. One is a web/mail server and the other is a backup
server. Currently, to back up the server I have a script that runs
which basically tar's the entire apache tree along with dumps of the
MySQL databases and use SCP to send them to the backup server.

Later, a script runs on the backup server which untar's the files and
imports the databases.

My question is, is this a good backup method? I see many people using
RSYNC. Would this be a better solution? If yes, how would I do it??

Thank you for your help and advice.

John

Mtek <mtek@mtekusa.com> writes:


>Hi,

>I have 2 servers. One is a web/mail server and the other is a backup
>server. Currently, to back up the server I have a script that runs
>which basically tar's the entire apache tree along with dumps of the
>MySQL databases and use SCP to send them to the backup server.

>Later, a script runs on the backup server which untar's the files and
>imports the databases.

>My question is, is this a good backup method? I see many people using
OK but cumbersome.
>RSYNC. Would this be a better solution? If yes, how would I do it??

Yes, it would be. rsync also checks the data for correctness ( hashes the
same) while transfering it.

It also transfers only what has changed, not everything every time.


The "easy" way

rsync -av /usr/local/http/ remotehost:/usr/local/http
rsync -av /usr/local/mysql/ remotehost:/usr/local/mysql


which will use sshd so you would need a passwordless login on the remote
machine.
(Assuming of couse that the apache tree is in /usr/local/http and that the
sql databases are in /usr/local/mysql on both machines. Change to conform
to what you actually have.)

Note that that last / on the first path is necessary. No slash means copy
the directory name as well as its contents, while a terminal / means
transfer just the contents.
-a means preserve the times, the owners and the permissions.

(v is for verbose. If you want it to be quiet, use q instead)

Lots more possiblities. -- excluding some directories or files, etc.
man rsync.

If you want it slightly faster, put an rsync daemon on the remote machine

exit /etc/rsync.conf on the remotehost and put in something like
[apache]
path=/usr/local/http
uid=0
gid=0
hosts allow=yourhost.yourdomain.com
hosts deny=0.0.0.0/0
readonly=false
timeout=600
[sql]
path=/usr/local/sql
uid=0
gid=0
hosts allow=yourhost.yourdomain.com
hosts deny=0.0.0.0/0
readonly=false
timeout=600

Then call it with
rsync -av /usr/local/http/ remotehost::apache
rsync -av /usr/local/sql/ remotehost::sql


Make sure that the
disable=no
is in /etc/xinetd.d/rsync
on the remotehost



>Thank you for your help and advice.

>John

On Jul 22, 9:30*pm, Unruh <unruh-s...@physics.ubc.ca> wrote:
> Mtek <m...@mtekusa.com> writes:
> >Hi,
> >I have 2 servers. *One is a web/mail server and the other is a backup
> >server. *Currently, to back up the server I have a script that runs
> >which basically tar's the entire apache tree along with dumps of the
> >MySQL databases and use SCP to send them to the backup server.
> >Later, a script runs on the backup server which untar's the files and
> >imports the databases.
> >My question is, is this a good backup method? *I see many people using
> OK but cumbersome.
> >RSYNC. *Would this be a better solution? *If yes, how would I do it??
>
> Yes, it would be. rsync also checks the data for correctness ( hashes the
> same) while transfering it.
>
> It also transfers only what has changed, not everything every time.
>
> The "easy" way
>
> rsync -av /usr/local/http/ remotehost:/usr/local/http
> rsync -av /usr/local/mysql/ remotehost:/usr/local/mysql
>
> which will use sshd so you would need a passwordless login on the remote
> machine.
> (Assuming of couse that the apache tree is in /usr/local/http and that the
> sql databases are in /usr/local/mysql on both machines. Change to conform
> to what you actually have.)
>
> Note that that last / on the first path is necessary. No slash means copy
> the directory name as well as its contents, while a terminal / means
> transfer just the contents.
> -a means preserve the times, the owners and the permissions.
>
> (v is for verbose. If you want it to be quiet, use q instead)
>
> Lots more possiblities. -- excluding some directories or files, etc.
> man rsync.
>
> If you want it slightly faster, put an rsync daemon on the remote machine
>
> exit /etc/rsync.conf on the remotehost and put in something like
> [apache]
> path=/usr/local/http
> uid=0
> gid=0
> hosts allow=yourhost.yourdomain.com
> hosts deny=0.0.0.0/0
> readonly=false
> timeout=600
> [sql]
> path=/usr/local/sql
> uid=0
> gid=0
> hosts allow=yourhost.yourdomain.com
> hosts deny=0.0.0.0/0
> readonly=false
> timeout=600
>
> Then call it with
> rsync -av /usr/local/http/ remotehost::apache
> rsync -av /usr/local/sql/ remotehost::sql
>
> Make sure that the
> disable=no
> is in /etc/xinetd.d/rsync
> on the remotehost
>
>
>
> >Thank you for your help and advice.
> >John- Hide quoted text -
>
> - Show quoted text -

Thanks Unrhu. I have 2 quick questions:

>Note that that last / on the first path is necessary. No slash means copy
>the directory name as well as its contents, while a terminal / means
>transfer just the contents.

What is the difference? Why not copy the name as well?


>If you want it slightly faster, put an rsync daemon on the remote machine

What will speed up? How is that config faster?

Thanks again! It is a lot of help.

John

Hello Mtek,






Mtek wrote:

>> >RSYNC. *Would this be a better solution? *If yes, how would I do it??
>>

Apart from the goodies described before, rsync has another nice option:
you can make several copies of your data with minimal loss of diskspace:
rsync is able to make copies with hardlinks if the data are identical.
(--link-dest=DIR)
In that case, the new version is a hardlink to the old version, so you have
two directory entries pointing to the same data. If the file is changed,
original and new file are stored separately. Of course, your database will
continuously change, if this is the major part, it does not make that much
difference.
Anyhow, if things get messed up, you have two or more generations...
You can implement schemes like, in shorthand notation:

rsync --link-dest=1 ..... 0
if success ( important!!!)
rm -rf 3
mv 2 3
mv 1 2
mv 0 1


In addition, rsync is an excellent program to transfer data via slow links,
e.g. to a remote system for backup, because only changed data are
transferred. Adding a small message to a big plain text email file means
only transfer of block checksums and the last "blocksize" bytes....

Kind Regards,
H,Janssen

Mtek <mtek@mtekusa.com> writes:


>Thanks Unrhu. I have 2 quick questions:

>>Note that that last / on the first path is necessary. No slash means copy
>>the directory name as well as its contents, while a terminal / means
>>transfer just the contents.

>What is the difference? Why not copy the name as well?

say the directory dir has a file /dir/file in it.

rsync -av /dir remote:/far
will have in /far the subdirectory called dir which contains the file
"file"
Ie /far/dir/far
rsync -av /dir/ remote:/far

has the contents of /dir in /far
Ie
/far/file





>>If you want it slightly faster, put an rsync daemon on the remote machine

>What will speed up? How is that config faster?

No ssh encryption.

So if it is sensitive data, use the first. If it is not, use the second.
slightly faster. Also control is by the host allow in /etc/rsync.conf,
rather than via ssh password or ssh key.

Unruh wrote:
> Mtek <mtek@mtekusa.com> writes:
>
>
>
>>Hi,
>
>
>>I have 2 servers. One is a web/mail server and the other is a backup
>>server. Currently, to back up the server I have a script that runs
>>which basically tar's the entire apache tree along with dumps of the
>>MySQL databases and use SCP to send them to the backup server.
>
>
>>Later, a script runs on the backup server which untar's the files and
>>imports the databases.
>
>
>>My question is, is this a good backup method? I see many people using
>
> OK but cumbersome.
>
>>RSYNC. Would this be a better solution? If yes, how would I do it??
>
>
> Yes, it would be. rsync also checks the data for correctness ( hashes the
> same) while transfering it.
>
> It also transfers only what has changed, not everything every time.

What about database file with holes, postgres comes to my mind.
Is it save to rsync them?


--Frank

On Jul 23, 6:54*am, Frank Elsner <Frank.Els...@spamfence.net> wrote:
> Unruh wrote:
> > Mtek <m...@mtekusa.com> writes:
>
> >>Hi,
>
> >>I have 2 servers. *One is a web/mail server and the other is a backup
> >>server. *Currently, to back up the server I have a script that runs
> >>which basically tar's the entire apache tree along with dumps of the
> >>MySQL databases and use SCP to send them to the backup server.
>
> >>Later, a script runs on the backup server which untar's the files and
> >>imports the databases.
>
> >>My question is, is this a good backup method? *I see many people using
>
> > OK but cumbersome.
>
> >>RSYNC. *Would this be a better solution? *If yes, how would I do it??
>
> > Yes, it would be. rsync also checks the data for correctness ( hashes the
> > same) while transfering it.
>
> > It also transfers only what has changed, not everything every time.
>
> What about database file with holes, postgres comes to my mind.
> Is it save to rsync them?
>
> --Frank- Hide quoted text -
>
> - Show quoted text -

Well, are you also saying that I do not need to export and re-import
the database tables? That it will copy the databse files and mySQL
will work fine on the backup server?

My goal obviously is if the main server goes down, I can shut it down,
change my router to point to the other server, and I'm up and running
again.

Regards,

John.

Il 23/07/2008 15.06, Mtek ha scritto:

> My goal obviously is if the main server goes down, I can shut it down,
> change my router to point to the other server, and I'm up and running
> again.

For mysql I think it is better to recreate database from mysqldump files.
Doing this, you will loose changes made after mysqldump command.
If this is not acceptable, you can set up mysql in master/slave mode and
your slave has always the same data that master has. See mysql
documentation on how to set up this, it is quite simple.

On Jul 23, 8:21*am, Alessandro Topo Galileo
<toglituttofinoalpunto.ale...@email.it> wrote:
> Il 23/07/2008 15.06, Mtek ha scritto:
>
> > My goal obviously is if the main server goes down, I can shut it down,
> > change my router to point to the other server, and I'm up and running
> > again.
>
> For mysql I think it is better to recreate database from mysqldump files.
> Doing this, you will loose changes made after mysqldump command.
> If this is not acceptable, you can set up mysql in master/slave mode and
> your slave has always the same data that master has. See mysql
> documentation on how to set up this, it is quite simple.


And going along with that, you still think RSYNC is a better idea than
ratting up specific directories and untarring them on the backup
server?

"H.Janssen" <henny@nospam.hmmsjan.dosgg.nlnospam> writes:

>Hello Mtek,

>




>Mtek wrote:

>>> >RSYNC. *Would this be a better solution? *If yes, how would I do it??
>>>

>Apart from the goodies described before, rsync has another nice option:
>you can make several copies of your data with minimal loss of diskspace:
>rsync is able to make copies with hardlinks if the data are identical.
>(--link-dest=DIR)
>In that case, the new version is a hardlink to the old version, so you have
>two directory entries pointing to the same data. If the file is changed,
>original and new file are stored separately. Of course, your database will
>continuously change, if this is the major part, it does not make that much
>difference.
>Anyhow, if things get messed up, you have two or more generations...

Depends on what things get messed up. If the backup gets messed up then all
versions get messed up since they are all the same thing. That could be a problem
However, the ability to have detailed backups from many different dates can
be a huge advantage if the backup itself is not comprimised.


>You can implement schemes like, in shorthand notation:

>rsync --link-dest=1 ..... 0
>if success ( important!!!)
> rm -rf 3
> mv 2 3
> mv 1 2
> mv 0 1


>In addition, rsync is an excellent program to transfer data via slow links,
>e.g. to a remote system for backup, because only changed data are
>transferred. Adding a small message to a big plain text email file means
>only transfer of block checksums and the last "blocksize" bytes....

Yup. Great feature.

>Kind Regards,
>H,Janssen