Backup tape reuse strategies?

Hi all,

We have a centralized backup system of several dozen machines. My boss
is cracking down on tape costs, so we are trying to come up with a
strategy to recycle old backup tapes. The problem is that it's not
easy to do it because the backup software (Time Navigator) writes
files from different machines and from different dates onto the same
tape. Thus, if I want to keep a full backup only of a certain subset
of machines from a year ago, it's very time-consuming to track down
the tapes involved.

Obviously, the software allows to backup each host to a different set
of tapes, but managing all this becomes hard very quickly, and it
would waste tapes anyway. I can also set cut-off dates where I close
all tapes and start anew, say, each month, but this also adds to the
management time and tape wastage.

Has anyone run into this problem (ease-of-use vs precise
accountability) with backup tapes, especially regarding tape reuse?
This would probably arise with any kind of centralized backup
solution...

Thanks,
Simon Ilyushchenko

<Michael Vilain <vilain@spamcop.net>> wrote in message
news:news-F6983B.15313928072003@news.tdl.com...
> In article <3226c4d3.0307281343.42854892@posting.google.com >,
> simonf@simonf.com (Simon) wrote:
>
> \> We have a centralized backup system of several dozen machines. My boss
> > is cracking down on tape costs, so we are trying to come up with a
> > strategy to recycle old backup tapes. The problem is that it's not
> > easy to do it because the backup software (Time Navigator) writes
> > files from different machines and from different dates onto the same
> > tape. Thus, if I want to keep a full backup only of a certain subset
> > of machines from a year ago, it's very time-consuming to track down
> > the tapes involved.

>
> At my last contract, we had several rules that were luckily in place
> prior to implementing a centralized backup product:
>
> 1-no filesystem will be larger than an what can fit on an uncompressed
> backup tape (automated backups were scripts in cron and couldn't change
> tapes)
>
> 2-each day had a separate backup. backups were not spread across
> multiple tapes and days.
>
> When we implemented the backups software, we kept rule 2. So, a full
> backup of a system is the only data on that tape. The backup product
> wasn't configured to stream multiple sessions to multiple tapes, so this
> was OK.
>
> Media usage was flagged automatically by the backup software. After 10
> uses, it's discarded. If you're backing up >3TB/day, it costs money--a
> lot less than the data.


At most sites there are servers, file-systems, that have different storage
requirments.
Full-Backups for Systems a-m are kept for a year, incrementals are kept for
two months.
Oracle dumps for 2 months, with Archive dumps taken every six months.
For systems n-q, we keep only 4 months of Full backups, with decreased times
for other types of backup's.
your mileage will vary.

the trick is to put similar backups with close to the same expiration times
on the same tape. It might mean that you are making multiple runs of backups
starting
at staggered times during the night, but the target of putting all the
incrementals that are
saved for two months on the same tape, and not mixing it with full backup
runs will make
it easier to reuse the tape.
Most tape systems have a hard time figuring out that a tape is full, so it
will use tape errors to
identify that it can't write any more to the tape. The downside to this is
that the system has a hard
time detecting that the tape is worn out. The General rule suggest that one
should not reuse worn
out tapes, and that reading worn out tapes is just as hard. If you find that
you expected
to write 40G to a DLT IV and it only wrote 26? Mark it as a tape that should
not be
reused. If you only got 5G? then you don't have reliable data on the tape.
Your own site
policies will tell you what to do then.

We use the newer Level 2 tapes for the bigger filesystems and use the older
DLT's for the
much smaller incremental backups very simply to avoid wasting space. It
might take two
weeks to fill up one of the newer tape drives with incremental information,
so we try to
use the smaller tapes for those situations

most tape backup system use this concept type of "classes" to group simlar
system
backup requirments together. If you can keep only backups of a single class
with the
same retention period on the same tape then the managment overhead can be
mimimzied

Netbackup and Legato have tools, methods and dogma for dealing with this
effectivly

In comp.unix.admin Simon <simonf@simonf.com> wrote:
> Hi all,

> We have a centralized backup system of several dozen machines. My boss
> is cracking down on tape costs, so we are trying to come up with a
> strategy to recycle old backup tapes. The problem is that it's not
> easy to do it because the backup software (Time Navigator) writes
> files from different machines and from different dates onto the same
> tape. Thus, if I want to keep a full backup only of a certain subset
> of machines from a year ago, it's very time-consuming to track down
> the tapes involved.

I think you may save consideratly more money by firing the boss. Any
manager that tries to save money on backup tapes is well beyond
"best before" date.

> Obviously, the software allows to backup each host to a different set
> of tapes, but managing all this becomes hard very quickly, and it
> would waste tapes anyway. I can also set cut-off dates where I close
> all tapes and start anew, say, each month, but this also adds to the
> management time and tape wastage.

Optimizing backup's is like optimizing the unknown.

You don't know what data you will need to restore, you don't know
which tapes are usable. Failure to restore lost data might be
the last thing you do at that company.

> Has anyone run into this problem (ease-of-use vs precise
> accountability) with backup tapes, especially regarding tape reuse?
> This would probably arise with any kind of centralized backup
> solution...

Shure. We do consider minor things of less importence then major. Like
"Corporate policy for backup and archiving", a document describing
which data to preserve and for how long. This document should be
signed by the CEO and constitute the conditions for backups.

I don't see that you refer to such a document. If that's so you are
missing it. Does your CEO know that you have _NO_ policy for backup ??

> Thanks,
> Simon Ilyushchenko

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

Raven and Peter Håkanson,

Thank you for your responses.

> Can I ask if he has a valid point or is it just a case of penny pinching?
> How much value do you place on the tapes if you actually need to use them
> for a restore?

I never thought I'd regret I am not in corporate world. :) Here is a
bigger picture: we are a research institution. We *are* on a tight
budget, and doing monthly fulls for the available disk space without
recycling tapes would run into six figures (USD). Backups are
important mostly for disaster recovery and short-term data loss - I
have had virtually no requests for files over half a year old.
However, we have had cases in the past when years-old data had to be
dug up regarding some patent issues. On top of all, our attempts to
ask the backup policy on a higher level have not had any reply yet - I
guess nobody wants to be responsible for formulating it. (Looks like
this is the first thing to pursue - there are still even higher people
to consult.)

> Could it be time to reconsider your whole strategy including the choice of
> backup software?

I am not averse to the idea - one of the reasons for the post is to
find out what other people are doing in similar situations.

> How many tape units and locations? By the sounds of it you need to run
> separate jobs for each machine so as to keep groups of tapes for each.

Two locations, two tape units in each, for Windows and Unix servers
separately.
Close to a hundred servers in total with about 10 Tb of disks storage
(and growing), though, thankfully, it's only about 50% utilized so
far.

The problem with separate jobs is increased maintenance time. After
all, we are not only minimizing the tape cost, but the man-hours
spent, which can be more productively used elsewhere.

> Agreed and agreed. Perhaps it's worth re-evaluating whether the cost of
> tapes under the current system is really that excessive.

From what I understand, the budgets are *really* tight. :)

> How many tapes are we talking about in total? Surely the value of the tape
> is irrelevant if the backups are so critical to the operation?

Judging by the frequency of the requests, long-term backups are not
very critical, but as I mentioned, in the absence of the policy you
never know.

> As far as accountability, if they really are tracking the tape costs
> themselves simply serial number each tape and do a semi-regular audit. Tapes
> that go bad get officially written off, tapes that go missing get
> questioned.

I did not mean just keeping track of tapes themselves, but rather of
what exactly is on each tape. And with centralized backup software,
it's non-trivial.

Simon Ilyushchenko

BTW, I forgot to ask: is anyone checking whether a tape is good before
using it? I mean the whole tape, not just a small part in the
beginning. My backup software does not do it.

Thanks,
Simon

In comp.unix.admin Simon <simonf@simonf.com> wrote:
> Raven and Peter Håkanson,

> Thank you for your responses.

Is the tapes fully utilized ? If not ( and the contents are not compressed
on non-compressing drives) lots of space might be wasted.

You might also gain significant amount if you can separate things that
can be recovered from distributions ( OS and tools ) from the actual
data. This might include forcing data to "fileservers-only" where
regular backups are done. Creating schedules with full backups
intermixed with "levels" ( aka hanoi-tower) might spare a number
of tapes. ( I have had great success using freely avaliable amanda
in a network of machines). The perfect case is where you can say :
"We don't backup workstations, they are recreated via fresh install".

Now a good policy is essential to define the "acceptable service", without
that you are always on the loosing side whenever data is lost. If
management avoids this you could create one yourself and state "We will
use this policy until something else is decided".

This policy might stipulate that "backup is for disaster purposes and
limited to (xx ) months. Any archiving has to be done via other purposes.
( remember that backup and archiving is two totally different things,
using different tools and with differing goals)

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

On 29 Jul 2003 09:59:11 -0700, Simon wrote:
>
> I never thought I'd regret I am not in corporate world. :) Here is a
> bigger picture: we are a research institution. We *are* on a tight
> budget, and doing monthly fulls for the available disk space without
> recycling tapes would run into six figures (USD). Backups are
> important mostly for disaster recovery and short-term data loss - I
> have had virtually no requests for files over half a year old.

Yes, IRS auditors were looking into a million $ plus project
which we wrote off as development research several years ago.
I think their computer brought it up because time was about to expire
and they would not be able to look into it. :(

Our lawyer calls me up and asks for project plans, requirement/design
docs, memos, source. Glad my operations group have yearly backups
going way back.

What is even better, when the media changed from round reel to
cartridge, they copied all the data onto the new media.

We would had to have paid a bit of a penalty without the items.
Penalty would have wiped all those *savings*.

Penny wise can be pound foolish.

Hey site the risk, the example. Ask them to check with the lawyer
if Management is libel for the policy. I find when I put management
in harm's way, we can find the better solution. :)

In comp.unix.admin Bit Twister <BitTwister@localhost.localdomain> wrote:

> Our lawyer calls me up and asks for project plans, requirement/design
> docs, memos, source. Glad my operations group have yearly backups
> going way back.
> What is even better, when the media changed from round reel to
> cartridge, they copied all the data onto the new media.

Similar situation: I used to work for $VERY_BIG_COMPANY who was being
sued for a (bogus) patent infringement claim. By going back to backup
tapes, again, 2 generations older than current technology, I was able to
retrieve the weekly "report to my boss what I was working on this week"
file from the key developer who had been on the project. Using these files,
we were able to show what, when, and by whom, the work had been done.

If we had reused those tapes, it would have been a finger-pointing contest,
at which point a jury could have very well decided to give the advantage
to the "little guy" fighting the very very big company. We kept all of our
full backups (done weekly) offsite, and pay an obscene amount of money to
store them (and for the tapes that are effectively "lost"), but that one
lawsuit could have more than wiped out any savings that we would have seen
by reusing those tapes.

If nothing else, decide on a policy...full backups have *x* months of
retention ('forever' might be the right answer), but incrementals have
just a few months before they're reused. Maybe you only want to keep one
set of full backups per month, but keep those forever; it depends on
how much risk you're willing to take, compared to the cost of tapes.

One other thing - if your software can be configured so that full and
incrementals get written to different tapes, do it. That way, only full
backups will be on the tapes you need to keep around.

> Penny wise can be pound foolish.

Absolutely correct. Tape is cheaper than the work needed to generate the
data on it, almost always.

Dave Hinz

On Mon, 28 Jul 2003 19:21:51 -0500, "Kermit T Tensmeyer"
<kermit.tensmeyer@ti.com> wrote:

[ Snip ]

> Most tape systems have a hard time figuring out that a tape is full, so it
>will use tape errors to identify that it can't write any more to the tape.
> The downside to this is that the system has a hard
> time detecting that the tape is worn out.

As a general rule, in the tape drive business we call tape systems
that behave as you describe "broken".

Reaching End-of-Tape is a specific non-error exception; unfortunately,
a lot of *software* fails to distinguish the case where a write
successfully wrote only part of the requested data because of an EOT
was reached (partial completion, no error), and the case where the
same thing happened because of an unrecoverable write error (partial
completion, write error).

[ Snip ]

Malc.

In comp.unix.admin Nicholas Bachmann <usenet@not-real.org> wrote:
> davehinz@spamcop.net wrote:
>>
>> Similar situation: I used to work for $VERY_BIG_COMPANY who was being
>> sued for a (bogus) patent infringement claim. By going back to backup

> Let me guess: some company nobody's ever heard of whose sole business is
> owning patents and filing lawsuits against big companies (like GE, IBM,
> or Microsoft) who will most likely settle quietly and finance the
> purchase of more stupid patents and more bogus lawsuits. Am I right?

I think you know I worked for GE, right? So, you're kind of right...
it was a case of "David vs. Goliath". To be honest, I'm not sure how
(or if) it finally was settled.

It was an MRI scanning technology that, yes, there was a patent that
vaguely described something like what GE was also doing, and the little
company claimed "We thought of it first, see, here." It was something
like "Instead of taking scans along just the X, Y, or Z axis, let's take
them at an angle" which, to my non-legal mind, falls into the category
of "blisteringly fucking obvious application of the technology" - kind of
like patenting using the side of a crayon for coloring instead of just
the tip.

It did have one relevant (to this thread) effect, though...my backup
tape budget went _way_ up, and I was able to switch from 8mm
(boo, hiss) to DLT-7000 drives. About the same cost per gig for tape,
but much cheaper to pay for storage (they charge by the tub, not by
the byte).

It all depends on the priorities. If the OP's needs are more of a backup/
restore than an archiving, then that changes the answers somewhat. If the
decision comes down to "Do we pay for archive tapes, or continue research",
well, that seems like a pretty straightforward question to answer...making
the organization become history just to preserve the history would be silly.

Dave Hinz