RE: Initialising a mirror

This is a discussion on RE: Initialising a mirror within the Rsync forums, part of the Networking and Network Related category; This is a multi-part message in MIME format. --===============1432665341== Content-Type: multipart/alternative; boundary="----=_NextPart_000_00D6_01C5D46D.5C518FA0" This ...

		Usenet Forums > Networking and Network Related > Rsync
RE: Initialising a mirror

LinkBack

Thread Tools

Search this Thread

Display Modes

#1 (permalink)

10-19-2005

Posts: n/a

RE: Initialising a mirror

This is a multi-part message in MIME format.

--===============1432665341==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_00D6_01C5D46D.5C518FA0"

This is a multi-part message in MIME format.

------=_NextPart_000_00D6_01C5D46D.5C518FA0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 8bit

"slow rsync" limited by bandwidth (or whatever) compared to....
FAST (LAN), SLOW (Internet), FAST (LAN) from "pointA" to "pointB"

100Gb of data -- how much/many of what is changing?
How many files? How big? (ranges) where changed?
How many (major) directories?
What kind of window and how big?
What are the preferable failure modes?
You do NOT have a choice of whether it fails.
You DO have a choice of which way it fails.

Looks like -p of scp preserves times and modes. ?? owner and group ??

Probably worth doing a fast look to see just what it is that you actually
have on both ends.
Intentions and reality quite often manage to do a disconnect.
You tend to get better answers if you state what you did (cut&paste)
as opposed to what you thought you did.
Generally when you have troubles they are not the same.

-----Original Message-----
From: Cyrille Bollu [mailto:Cyrille.Bollu@fedasil.be]
Sent: Wednesday, October 19, 2005 4:35 AM
To: Tony@ServaCorp.com
Cc: rsync@lists.samba.org
Subject: RE: Initialising a mirror

Hi,

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I think
this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to
the remote site :-)

Just for the record, I didn't made the standard mistake, having things one
directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary information
needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

<Tony@ServaCorp.com> a écrit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off
> from where they should be.
>
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,
> do a fast rsync from the second disk space to the legitimate target..
>
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one
> level difference between them.
>
> I know I've done it a number of times, and it is no fun over remote
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>
>
> WARNING -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so
> I'm not posting to the list.---
> ?? hosts allow/deny ignore with auth users present ??
> The major sucurity concerns are NOT permanently losing all my data
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't
like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>
>
>
> ---Timing depends on what is different. Typically this one is about
> 10 to 15 minutes (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real 192m41.906s
>
>
> real 12m18.512s
> sent 13750 bytes received 7577551 bytes 10279.35 bytes/sec
> total size is 14013850399 speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root@wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
>
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
> rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>
> ----rsync server (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
> comment = backup of sjs dwg
> path = /home/rsync-sjs-dwg
> use chroot = true
> read only = false
> list = false
> uid = root
> gid = root
> auth users = rsync-sjs-dwg
> secrets file = /etc/rsyncd.secrets
> strict modes = true
> hosts allow = 10.0.0.1 pete-162 wf
> hosts deny = *.*.*.*
> ---rsync "client" (This is a PRIMARY)
> [root@sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/OPENED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /home/dwg/$name rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
> /tmp/rsync/CLOSED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>
>
>
>
>
> -----Original Message-----
> From: rsync-bounces+tony=servacorp.com@lists.samba.org [mailto:
> rsync-bounces+tony=servacorp.com@lists.samba.org]On Behalf Of Cyrille
Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror

>
> Dears,
>
> I would like to mirror data from 2 servers connected together via
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).
>
> I'm sure rsync is one of the best tool to keep these data in sync
> but how should I use it to initialise the mirror?
>
> I'm currently testing the solution with 10Gb of data to keep in
> sync. But on my lines it would take more than 40 hours to initially
> create the mirror! And in production the data to keep in sync will
> reach 100Gb...

>
> So, I went with a laptop on the remote site and "scp -rp" the data
> on it. I then got back to my site and "scp -rp" it on the mirror to
> initialise it.
>
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.
>
> Does anybody has an idea about what I'm doing wrong?
>
> Does anybody has a solution to this problem?
>
> Regards,
>
> Cyrille

------=_NextPart_000_00D6_01C5D46D.5C518FA0
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META content=3D"MSHTML 5.00.2314.1000" name=3DGENERATOR></HEAD>
<BODY>
<DIV>"slow=20
rsync"    limited by bandwidth (or whatever) compared=20
to....</DIV>
<DIV>FAST=20
(LAN), SLOW (Internet), FAST (LAN)  from "pointA" to=20
"pointB"</DIV>
<DIV><SPAN=20
class=3D300230810-19102005> </DIV>
<DIV>100Gb=20
of data    -- how much/many of what is=20
changing?</DIV>
<DIV>How=20
many files?  How big? (ranges)  where =
changed?</DIV>
<DIV>How=20
many (major) directories?</DIV>
<DIV>What=20
kind of window and how big?</DIV>
<DIV>What=20
are the preferable failure modes?</DIV>
<DIV><SPAN=20
class=3D300230810-19102005>    You do NOT have a choice =
of whether=20
it fails.</DIV>
<DIV><SPAN=20
class=3D300230810-19102005>    You DO have a choice of =
which way it=20
fails.</DIV>
<DIV><SPAN=20
class=3D300230810-19102005> </DIV>
<DIV>Looks=20
like -p of scp preserves times and modes.  ?? owner and group=20
??</DIV>
<DIV><SPAN=20
class=3D300230810-19102005> </DIV>
<DIV><SPAN=20
class=3D300230810-19102005>Probably worth doing a fast look to see just =
what it is=20
that you actually have on both ends.</DIV>
<DIV><SPAN=20
class=3D300230810-19102005>Intentions and reality quite often manage to =
do a=20
disconnect.</DIV>
<DIV>You=20
tend to get better answers if you state what you did (cut&paste)=20
</DIV>
<DIV>as=20
opposed to what you thought you did.</DIV>
<DIV><SPAN=20
class=3D300230810-19102005>Generally when you have troubles they are not =
the=20
same.</DIV>
<DIV><SPAN=20
class=3D300230810-19102005> </DIV>
<DIV align=3Dleft class=3DOutlookMessageHeader dir=3Dltr>-----Original Message----- From: Cyrille Bollu=20
[mailto:Cyrille.Bollu@fedasil.be] Sent: Wednesday, October 19, =
2005=20
4:35 AM To: Tony@ServaCorp.com Cc:=20
rsync@lists.samba.org Subject: RE: Initialising a=20
mirror </DIV> Hi, =
 <FONT=20
face=3DArial size=3D2>thank you for your answer. But what do you mean by "slow rsync"? Do you mean limiting =
bandwith? I=20
think this will not be an option with 100Gb of data to =
synchronise...=20
 However, I like your idea with the =
LAN: I think=20
I will bring my mirror to the remote site :-) Just for the record, I didn't made the standard mistake, having =
things=20
one directory level off from where they should be. <FONT=20
face=3DArial size=3D2>I was hoping doing a "scp -rp" would preserve all =
the=20
necessary information needed by rsync but apparently not... Maybe I =
should use=20
tar? Regards, =
 <FONT=20
face=3DArial size=3D2>Cyrille <FONT=20
size=3D2><TT><Tony@ServaCorp.com> a =E9crit sur 19/10/2005 =
11:00:07=20
: > Standard mistake is to somehow get things one directory =
level off=20
 > from where they should be.</TT> <TT>>=20
 </TT> <TT>> If you have the available =
disk=20
space (both ends)</TT> <TT>> Do a fast =
rsync from the=20
primary to locally available fast disk > space (same computer or=20
LAN)</TT> <TT>> Do a slow rsync from that =
disk space=20
to another set of available > disk space over slow/unreliable=20
network.</TT> <TT>> Assuming the slow rsync =
completed=20
without problems, </TT> <TT>>     =
do a fast=20
rsync from the second disk space to the legitimate target..</TT>=20
 <TT>>  </TT> <TT>> Source=20
makes a big distinction between foo/ and foo</TT> <FONT=20
size=3D2><TT>> They both transfer the same files.</TT> =
 <FONT=20
size=3D2><TT>> They differ in where they transfer the file to =
 (BY ONE=20
LEVEL)</TT> <TT>> Both make sense and are =
quite=20
reasonable.</TT> <TT>> If you do both, the =
effects=20
are similar to untarring twice with one > level difference =
between=20
them.</TT> <TT>>  </TT> =
 <FONT=20
size=3D2><TT>> I know I've done it a number of times, and it is no =
fun over=20
remote > and often bad internet connections.</TT> =
 <FONT=20
size=3D2><TT>> With large volumes, it is not apparent when you've =
done it to=20
yourself.</TT> <TT>>  </TT> =
 <FONT=20
size=3D2><TT>>  </TT> <TT>> WARNING =
 --=20
these scripts work but are probably not the best nor the > most=20
aesthetic.</TT> <TT>> ---just a wee bit too =

informative and I'm too lazy to sanitize so > I'm not posting to =
the=20
list.---</TT> <TT>>     ?? hosts =
allow/deny=20
ignore with auth users present ?? </TT> <TT>> The=20
major sucurity concerns are NOT permanently losing all my data > =
rather=20
than eavesdroppers.</TT> <TT>> The internet =

connection(s) are ocassionally flaky to non-existent, > and =
everything=20
needs to assume</TT> <TT>> that they will =
chose the=20
most inopportune moment to do something I don't like.</TT> =
 <FONT=20
size=3D2><TT>> Essentially the stunt is that /home/rsync-foo is =
rsync'd with=20
other > such over slow and bad connections.</TT> <TT>> With bad internet I've got (not shown) multiple =
off-site backups=20
AND > the ability to bring them all upt-to-date</TT> =
 <FONT=20
size=3D2><TT>> if there is a connection (any, not necessarily the =
preferred --=20
some > connect, some don't connect)</TT> <TT>>=20
 </TT> <TT>>  </TT> =
 <FONT=20
size=3D2><TT>>  </TT> <TT>> =
---Timing depends=20
on what is different.  Typically this one is about > 10 to 15 =
minutes=20
 (about 10G) total.</TT> <TT>> If =
large MySql=20
tables are rearranged, tends to take rather longer.</TT> =
 <FONT=20
size=3D2><TT>> real    192m41.906s</TT> <FONT=20
size=3D2><TT>>  </TT> <TT>> =
 </TT>=20
 <TT>> real    12m18.512s > sent =
13750 bytes=20
 received 7577551 bytes  10279.35 bytes/sec > total size =
is=20
14013850399  speedup is 1846.04 > Tue Oct 18 07:28:00 CDT=20
2005 > rsync-sjs-dwg</TT> <TT>> =
[root@wf1=20
/root]# cat rsync-sjs-dwg > #!/bin/sh</TT> <FONT=20
size=3D2><TT>>  </TT> <TT>> rsync=20
client</TT> <TT>> # staged backup > =
time rsync=20
-a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/</TT> <FONT=20
size=3D2><TT>>  </TT> <TT>> time =
rsync -avz=20
--progress --partial --timeout=3D1750 --password- >=20
file=3D/etc/rsync.secrets/rsync-sjs-dwg \ >       =
 =20
rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/</TT> =
 <FONT=20
size=3D2><TT>> date; echo rsync-sjs-dwg</TT> <TT>>=20
 </TT> <TT>> ----rsync server   =
(actually=20
is an intermediate, not the primary)</TT> <TT>> This=20
is an extra "junk" computer with a decent hard drive.</TT> =
 <FONT=20
size=3D2><TT>> This serves to get a fast </TT> <TT>>=20
# etc/rsyncd.conf</TT> <TT>> pid file =3D=20
/etc/rsyncd.pid</TT> <TT>> =
[rsync-sjs-dwg] >=20
        comment =3D backup of sjs dwg >   =
 =20
    path =3D /home/rsync-sjs-dwg >       =
  use=20
chroot =3D true >         read only =3D =
false >=20
        list =3D false >       =
  uid=20
=3D root >         gid =3D root >   =
 =20
    auth users =3D rsync-sjs-dwg >       =
 =20
secrets file =3D /etc/rsyncd.secrets >         =
strict=20
modes =3D true >         hosts allow =3D =
10.0.0.1 pete-162=20
wf >         hosts deny =3D =
*.*.*.*</TT>=20
 <TT>> ---rsync "client"  (This is a=20
PRIMARY)</TT> <TT>> [root@sys001 root]# cat =

rsync-sjs-dwg > #!/bin/bash > # rsync-sjs-dwg =
 SJS_Drawings/=20
title/ > mkdir -p /tmp/rsync ; echo `hostname` >=20
/tmp/rsync/OPENED > rsync -a =
--password-file=3D/etc/rsync.secrets/sjs-dwg=20
 --timeout=3D750 \ >         =
/tmp/rsync/OPENED=20
      =
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/</TT>=20
 <TT>> for name in title SJS_Drawings ; do > =
rsync -a=20
--password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \ > =
   =20
    /home/dwg/$name        =20
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/ > done</TT> =
 <FONT=20
size=3D2><TT>> mkdir -p /tmp/rsync ; echo `hostname` >=20
/tmp/rsync/CLOSED > rsync -a =
--password-file=3D/etc/rsync.secrets/sjs-dwg=20
 --timeout=3D750 \ >         =
/tmp/rsync/CLOSED=20
      =
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/</TT>=20
 <TT>> echo 'rsync-sjs-dwg' > date > =
date=20
-u</TT> <TT>>  </TT> <FONT=20
size=3D2><TT>> >  </TT> <TT>>=20
 </TT> <TT>>  </TT> =
 <FONT=20
size=3D2><TT>> -----Original Message----- > From:=20
rsync-bounces+tony=3Dservacorp.com@lists.samba.org [mailto: >=20
rsync-bounces+tony=3Dservacorp.com@lists.samba.org]On Behalf Of Cyrille=20
Bollu > Sent: Wednesday, October 19, 2005 2:58 AM > To:=20
rsync@lists.samba.org > Subject: Initialising a=20
mirror </TT> <TT>> > Dears, =
 >=20
 > I would like to mirror data from 2 servers connected together =
via=20
 > VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps). > > =
I'm=20
sure rsync is one of the best tool to keep these data in sync > =
but how=20
should I use it to initialise the mirror? > > I'm =
currently=20
testing the solution with 10Gb of data to keep in > sync. But on =
my lines=20
it would take more than 40 hours to initially > create the =
mirror! And in=20
production the data to keep in sync will > reach 100Gb... :-( =
 >=20
 > So, I went with a laptop on the remote site and "scp -rp" the =
data=20
 > on it. I then got back to my site and "scp -rp" it on the =
mirror to=20
 > initialise it. > > But apparently I forgot =
something=20
because when rsync still sends all > the files from the remote =
site to my=20
mirror when I first run it. > > Does anybody has an idea =
about=20
what I'm doing wrong? > > Does anybody has a solution to =
this=20
problem? > > Regards, > >=20
Cyrille</TT></BODY></HTML>

------=_NextPart_000_00D6_01C5D46D.5C518FA0--

--===============1432665341==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--===============1432665341==--

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are Off [IMG] code is Off HTML code is Off Trackbacks are On Pingbacks are On Refbacks are On

All times are GMT +1. The time now is 08:24 PM.