RE: Initialising a mirror

This is a discussion on RE: Initialising a mirror within the Rsync forums, part of the Networking and Network Related category; Message en plusieurs parties au format MIME --===============1470178700== Content-Type: multipart/alternative; boundary="=_alternative 00349CEEC125709F_=" Message en plusieurs ...

		Usenet Forums > Networking and Network Related > Rsync
RE: Initialising a mirror

LinkBack

Thread Tools

Display Modes

#1 (permalink)

10-19-2005

Cyrille Bollu

Posts: n/a

RE: Initialising a mirror

Message en plusieurs parties au format MIME
--===============1470178700==
Content-Type: multipart/alternative;
boundary="=_alternative 00349CEEC125709F_="

Message en plusieurs parties au format MIME
--=_alternative 00349CEEC125709F_=
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,=20

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I=20
think this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to=20
the remote site :-)

Just for the record, I didn't made the standard mistake, having things one =

directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary=20
information needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

<Tony@ServaCorp.com> a =E9crit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off=20
> from where they should be.
>=20
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk=20
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available=20
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,=20
> do a fast rsync from the second disk space to the legitimate=20
target..
>=20
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one=20
> level difference between them.
>=20
> I know I've done it a number of times, and it is no fun over remote=20
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>=20
>=20
> WARNING -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so=20
> I'm not posting to the list.---
> ?? hosts allow/deny ignore with auth users present ??=20
> The major sucurity concerns are NOT permanently losing all my data=20
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,=20
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't =

like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other=20
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>=20
>=20
>=20
> ---Timing depends on what is different. Typically this one is about
> 10 to 15 minutes (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real 192m41.906s
>=20
>=20
> real 12m18.512s
> sent 13750 bytes received 7577551 bytes 10279.35 bytes/sec
> total size is 14013850399 speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root@wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
>=20
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>=20
> time rsync -avz --progress --partial --timeout=3D1750 --password-
> file=3D/etc/rsync.secrets/rsync-sjs-dwg \
> rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>=20
> ----rsync server (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast=20
> # etc/rsyncd.conf
> pid file =3D /etc/rsyncd.pid
> [rsync-sjs-dwg]
> comment =3D backup of sjs dwg
> path =3D /home/rsync-sjs-dwg
> use chroot =3D true
> read only =3D false
> list =3D false
> uid =3D root
> gid =3D root
> auth users =3D rsync-sjs-dwg
> secrets file =3D /etc/rsyncd.secrets
> strict modes =3D true
> hosts allow =3D 10.0.0.1 pete-162 wf
> hosts deny =3D *.*.*.*
> ---rsync "client" (This is a PRIMARY)
> [root@sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg SJS=5FDrawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /tmp/rsync/OPENED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> for name in title SJS=5FDrawings ; do
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /home/dwg/$name rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /tmp/rsync/CLOSED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>=20
>=20
>=20
>=20
>=20
> -----Original Message-----
> From: rsync-bounces+tony=3Dservacorp.com@lists.samba.org [mailto:
> rsync-bounces+tony=3Dservacorp.com@lists.samba.org]On Behalf Of Cyrille=20
Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror

>=20
> Dears,=20
>=20
> I would like to mirror data from 2 servers connected together via=20
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).=20
>=20
> I'm sure rsync is one of the best tool to keep these data in sync=20
> but how should I use it to initialise the mirror?=20
>=20
> I'm currently testing the solution with 10Gb of data to keep in=20
> sync. But on my lines it would take more than 40 hours to initially=20
> create the mirror! And in production the data to keep in sync will=20
> reach 100Gb... :-(=20
>=20
> So, I went with a laptop on the remote site and "scp -rp" the data=20
> on it. I then got back to my site and "scp -rp" it on the mirror to=20
> initialise it.=20
>=20
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.=20
>=20
> Does anybody has an idea about what I'm doing wrong?=20
>=20
> Does anybody has a solution to this problem?=20
>=20
> Regards,=20
>=20
> Cyrille
--=_alternative 00349CEEC125709F_=
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

 Hi, 
 
 thank you for your answer.
 
 But what do you mean by "slow rsync&=
quot;?
Do you mean limiting bandwith? I think this will not be an option with
100Gb of data to synchronise...
 
 However, I like your idea with the LAN: I
think I will bring my mirror to the remote site :-)
 
 Just for the record, I didn't made the st=
andard
mistake, having things one directory level off from where they should be.</=
font>
 
 I was hoping doing a "scp -rp"
would preserve all the necessary information needed by rsync but apparently
not... Maybe I should use tar?
 
 Regards,
 
 Cyrille
 
 <tt><Tony@ServaCorp.com> a =E9crit sur 19/10/2005
11:00:07 : 
 
> Standard mistake is to somehow get things one directory level off
 
> from where they should be.</tt>
 <tt>>  </tt>
 <tt>> If you have the available disk space (both ends=
)</tt>
 <tt>> Do a fast rsync from the primary to locally ava=
ilable
fast disk 
> space (same computer or LAN)</tt>
 <tt>> Do a slow rsync from that disk space to another
set of available 
> disk space over slow/unreliable network.</tt>
 <tt>> Assuming the slow rsync completed without probl=
ems,
</tt>
 <tt>>     do a fast rsync from the second
disk space to the legitimate target..</tt>
 <tt>>  </tt>
 <tt>> Source makes a big distinction between foo/ and
foo</tt>
 <tt>> They both transfer the same files.</tt>
 <tt>> They differ in where they transfer the file to
 (BY ONE LEVEL)</tt>
 <tt>> Both make sense and are quite reasonable.</tt><=
/font>
 <tt>> If you do both, the effects are similar to unta=
rring
twice with one 
> level difference between them.</tt>
 <tt>>  </tt>
 <tt>> I know I've done it a number of times, and it
is no fun over remote 
> and often bad internet connections.</tt>
 <tt>> With large volumes, it is not apparent when you=
've
done it to yourself.</tt>
 <tt>>  </tt>
 <tt>>  </tt>
 <tt>> WARNING  -- these scripts work but are pro=
bably
not the best nor the 
> most aesthetic.</tt>
 <tt>> ---just a wee bit too informative and I'm too
lazy to sanitize so 
> I'm not posting to the list.---</tt>
 <tt>>     ?? hosts allow/deny ignore with
auth users present ?? </tt>
 <tt>> The major sucurity concerns are NOT permanently
losing all my data 
> rather than eavesdroppers.</tt>
 <tt>> The internet connection(s) are ocassionally fla=
ky
to non-existent, 
> and everything needs to assume</tt>
 <tt>> that they will chose the most inopportune moment
to do something I don't like.</tt>
 <tt>> Essentially the stunt is that /home/rsync-foo
is rsync'd with other 
> such over slow and bad connections.</tt>
 <tt>> With bad internet I've got (not shown) multiple
off-site backups AND 
> the ability to bring them all upt-to-date</tt>
 <tt>> if there is a connection (any, not necessarily
the preferred -- some 
> connect, some don't connect)</tt>
 <tt>>  </tt>
 <tt>>  </tt>
 <tt>>  </tt>
 <tt>> ---Timing depends on what is different.  T=
ypically
this one is about 
> 10 to 15 minutes  (about 10G) total.</tt>
 <tt>> If large MySql tables are rearranged, tends to
take rather longer.</tt>
 <tt>> real    192m41.906s</tt>
 <tt>>  </tt>
 <tt>>  </tt>
 <tt>> real    12m18.512s 
> sent 13750 bytes  received 7577551 bytes  10279.35 bytes/sec=
 
> total size is 14013850399  speedup is 1846.04 
> Tue Oct 18 07:28:00 CDT 2005 
> rsync-sjs-dwg</tt>
 <tt>> [root@wf1 /root]# cat rsync-sjs-dwg 
> #!/bin/sh</tt>
 <tt>>  </tt>
 <tt>> rsync client</tt>
 <tt>> # staged backup 
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/</tt></fon=
t>
 <tt>>  </tt>
 <tt>> time rsync -avz --progress --partial --timeout=
=3D1750
--password- 
> file=3D/etc/rsync.secrets/rsync-sjs-dwg \ 
>         rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/r=
sync-sjs-dwg/</tt>
 <tt>> date; echo rsync-sjs-dwg</tt>
 <tt>>  </tt>
 <tt>> ----rsync server   (actually is an interme=
diate,
not the primary)</tt>
 <tt>> This is an extra "junk" computer with
a decent hard drive.</tt>
 <tt>> This serves to get a fast </tt>
 <tt>> # etc/rsyncd.conf</tt>
 <tt>> pid file =3D /etc/rsyncd.pid</tt>
 <tt>> [rsync-sjs-dwg] 
>         comment =3D backup of sjs dwg 
>         path =3D /home/rsync-sjs-dwg 
>         use chroot =3D true 
>         read only =3D false 
>         list =3D false 
>         uid =3D root 
>         gid =3D root 
>         auth users =3D rsync-sjs-dwg 
>         secrets file =3D /etc/rsyncd.secrets 
>         strict modes =3D true 
>         hosts allow =3D 10.0.0.1 pete-162 wf 
>         hosts deny =3D *.*.*.*</tt>
 <tt>> ---rsync "client"  (This is a
PRIMARY)</tt>
 <tt>> [root@sys001 root]# cat rsync-sjs-dwg 
> #!/bin/bash 
> # rsync-sjs-dwg  SJS=5FDrawings/ title/ 
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED 
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg  --timeout=
=3D750
\ 
>         /tmp/rsync/OPENED      
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/</tt>
 <tt>> for name in title SJS=5FDrawings ; do 
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750
\ 
>         /home/dwg/$name        
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/ 
> done</tt>
 <tt>> mkdir -p /tmp/rsync ; echo `hostname` > /tmp=
/rsync/CLOSED 
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg  --timeout=
=3D750
\ 
>         /tmp/rsync/CLOSED      
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/</tt>
 <tt>> echo 'rsync-sjs-dwg' 
> date 
> date -u</tt>
 <tt>>  </tt>
 <tt>> 
>  </tt>
 <tt>>  </tt>
 <tt>>  </tt>
 <tt>> -----Original Message----- 
> From: rsync-bounces+tony=3Dservacorp.com@lists.samba.org [mailto: 
> rsync-bounces+tony=3Dservacorp.com@lists.samba.org]On Behalf Of Cyrille
Bollu 
> Sent: Wednesday, October 19, 2005 2:58 AM 
> To: rsync@lists.samba.org 
> Subject: Initialising a mirror 
</tt>
 <tt>> 
> Dears, 
> 
> I would like to mirror data from 2 servers connected together via
 
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps). 
> 
> I'm sure rsync is one of the best tool to keep these data in sync
 
> but how should I use it to initialise the mirror? 
> 
> I'm currently testing the solution with 10Gb of data to keep in 
> sync. But on my lines it would take more than 40 hours to initially
 
> create the mirror! And in production the data to keep in sync will
 
> reach 100Gb... :-( 
> 
> So, I went with a laptop on the remote site and "scp -rp"
the data 
> on it. I then got back to my site and "scp -rp" it on the
mirror to 
> initialise it. 
> 
> But apparently I forgot something because when rsync still sends all<b=
r>
> the files from the remote site to my mirror when I first run it. 
> 
> Does anybody has an idea about what I'm doing wrong? 
> 
> Does anybody has a solution to this problem? 
> 
> Regards, 
> 
> Cyrille</tt>
--=_alternative 00349CEEC125709F_=--

--===============1470178700==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--===============1470178700==--

« Previous Thread | Next Thread »

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are Off [IMG] code is Off HTML code is Off Trackbacks are On Pingbacks are On Refbacks are On

All times are GMT +1. The time now is 03:32 PM.