Business backup strategies

[Rat]

We're looking at changing the way we do backup at work, and I thought some people here might have some suggestions. I'll start by explaining what we do now:

We have about 2 TB of data on the server, and between 3 and 30 GB usually changes or is added daily.
We use a Retrospect script to copy all new or changed data from the last couple of days to DVD. There are 2 such scripts that run on alternate days, so that one set can be kept on site and one off site.
We also have a set of removable hard drives, one of which is in the backup machine at any one time. It contains a copy of the entire server, and again a Retrospect script runs overnight to scan the server and copy any new or changed files.

This belt and braces approach means that as long as a file exists for at least one night, we can retrieve any file from any day of the past 15 years or so. Also, in the event of a disaster (fire or whatever), it would be trivial to plug in one of the removable hard drives in a new place and retrieve all the data at one time.

I realize there are some flaws in this system, but it's always worked very well for us. I further recognize that lots of people will likely tell me that we can't use optical discs because they have no lifespan, and that we should use tapes instead, but I have repeatedly found the opposite to be true, so we really don't want to use tapes again.

The reasons we're looking at it is that now so much data is changing or being added each day, the DVD backup, which kicks off at about 1900, needs a new disc at around 2100 (there are only two DVD drives) and is therefore still sitting there the following morning. It works, but it means it's wasting server time during working hours, and the number of discs will soon be unmanageable. Also, we're looking to centralize backup from all our 5 offices, so the amount of data would then increase somewhat (although not fivefold, since the other offices are smaller).

In the short term, we're looking at perhaps replacing the DVDs with BluRay, but this is clearly a workaround rather than a solution. I should mention that we're also backing up an MS SQL server and an Exchange server as well as a (2008) file server. The reason we're looking at a short-term BluRay solution is that because of "the current economic climate", we'd never get budget approval to switch to a completely new system if it meant buying all-new hardware and software, but if we knew what direction we wanted to move in, it would be helpful to decide where to go next.

I guess it's taking the piss a bit to ask someone to come up with a comprehensive backup solution for us, but I'd at least be interested to hear how others have handled it.

tl;dr:
How do you recommend doing backup?

 

[Rat]

Hmm, well I shall go to bed now, and see whether anyone wants to do my job for me in the morning.

 

[theprestige]

Not being a backup admin, all I can say is, your direction of travel seems to be more data, taking up less space, faster, cheaper. This is probably one of those two-of-four, three-of-four if you're lucky, situations.

As a systems administrator, I will say this with confidence born of long experience: Every backup system works just great, right up to the moment when you try to restore from backup. Phrases like "it would be trivial to plug in one of the removable hard drives in a new place and retrieve all the data at one time" give me the willies.

Mark my words: If you aren't testing restores on a regular basis, they will fail you when you need them most.

 

[a_unique_person]

There is no computer problem you can't fix if you've got enough money to throw at it. What is your budget? USB drives are good, just rotate them. You can buy a portable 1TB for $100 now.

Tapes are very good now. You don't want to drop them, but if you look after them I have had no problems. Very fast, fit massive amounts of data. Just rotate them. You could do a full backup nightly.

 

[JoelKatz]

What's your budget? What kind of reliability do you need? How critical is your data overall? How critical is data timeliness?

My basic approach would be two portable drive systems. On a small RAID arrays such as the four-port Drobo. 4x2TB in RAID 5 would give you 5.4TiB usable. You can upgrade to 4x3TB when needed for 8.2 TiB usable.

The other could just be a standard 1TB portable drive.

The Drobo (or whatever device you use) would remain fixed off-site. The off-site Drobo would contain a full backup and as much history as space allows. The mobile unit would carry deltas from the office back to the off-site location and used to update the Drobo.

This would mean the backup person would have to carry a portable hard drive to and from the backup location. However, that drive only has to be big enough to carry the daily deltas.

One caution -- if the loss of a delta would make you unable to produce a new delta, you must keep an extra day's delta backed up at the main site. Otherwise, the loss of the portable drive would make you unable to sync the Drobo to the production system.

 

[a_unique_person]

The easiest way to back up is to have two sans liked by high speed WAN with replication software. Then work out how you want your history stored.

 

[Ducky]

Hire an intern at minimum wage to hand write the 1's and 0's to notebook paper.

 

[Rat]

Not being a backup admin, all I can say is, your direction of travel seems to be more data, taking up less space, faster, cheaper. This is probably one of those two-of-four, three-of-four if you're lucky, situations.

As a systems administrator, I will say this with confidence born of long experience: Every backup system works just great, right up to the moment when you try to restore from backup. Phrases like "it would be trivial to plug in one of the removable hard drives in a new place and retrieve all the data at one time" give me the willies.

Mark my words: If you aren't testing restores on a regular basis, they will fail you when you need them most.

Oh, this I know. As for the DVD backups, we have to do restores of individual files quite regularly, since users are users, so I know they're all good. I've never had to do a full restore of the server from the hard drive backups, but I know they're backing up properly, since I sometimes use them for individual file restore if someone just wants something from the last couple of days.

 

[Rat]

There is no computer problem you can't fix if you've got enough money to throw at it. What is your budget? USB drives are good, just rotate them. You can buy a portable 1TB for $100 now.

Tapes are very good now. You don't want to drop them, but if you look after them I have had no problems. Very fast, fit massive amounts of data. Just rotate them. You could do a full backup nightly.

The budget we have right now is effectively zero, in that we'll never get approval right now to introduce a new system, only to maintain what we have now. What I'm really looking for, since our current system can't last for more than another year or two, is what direction we're heading in, so that we can move towards a new system over time if possible.

Of course, I'm quite happy to separate the 'backup' part (meaning the hard drives that can easily do a full restore) from the 'archive' part (the DVDs), and it's really the latter that's more important, since that's the part that's taking most of the time and becoming impractical.

 

[NewtonTrino]

I would go with something like a synology for local and an online backup service to back up the synology offsite.

 

[Zax63]

How about a drive dock and several bare SATA hard drive? You can get padded carriers for traveling with the drives.

 

[Rat]

How about a drive dock and several bare SATA hard drive? You can get padded carriers for traveling with the drives.

That's half of what we're doing at the moment. I used to take them home every day, but I'm having to do part time at the moment, so my boss takes them.

 

[Rat]

The easiest way to back up is to have two sans liked by high speed WAN with replication software. Then work out how you want your history stored.

I would go with something like a synology for local and an online backup service to back up the synology offsite.

I'd have to say, online backup is something we've looked into, and we just can't seem to find a way to do it. Partly the cost (I use Mozy at home, but it's quickly expensive for business), but mostly the speed. Even on a 10Mbps SDSL connection, we'd be struggling with that amount of data at times.

 

[rjh01]

One thing is essential to do. Set up a small team none of whom normally work were the computer is and tell them to restore the backup computer from the remote backups. They are not, under any conditions to contact anyone who works at the computer site. Give them a blank cheque. Once they have restored the computer get your users to verify they can do a full days work. If all your staff work in the one location with the computer, do not bother about remote backups and what I am saying. If a fire destroys your computer and kills your staff then your company is finished.

If you can do the above, then you can survive a major fire on your computer site that kills all your staff at the site.

Worst case scenario. You work in the WTC in New York. Then 9/11 happens. Does your business survive?

 

[Dr. Keith]

Of course, I'm quite happy to separate the 'backup' part (meaning the hard drives that can easily do a full restore) from the 'archive' part (the DVDs), and it's really the latter that's more important, since that's the part that's taking most of the time and becoming impractical.

General Disclaimer: I only understand every third acronym being used in this conversation. This is not my field at all.

Why make the Archive a separate physical media? Can't you just buy some hard drives and and do a nightly archive onto them? The prices seems to be dropping to the point where maintaining the DVD system or migrating to Blu-Ray will be more costly than the hard drives.

 

[rjh01]

General Disclaimer: I only understand every third acronym being used in this conversation. This is not my field at all.

Why make the Archive a separate physical media? Can't you just buy some hard drives and and do a nightly archive onto them? The prices seems to be dropping to the point where maintaining the DVD system or migrating to Blu-Ray will be more costly than the hard drives.

One issue here is that you need two backups.
1. Backup on site for what some idiot deletes a file.
2. Remote backup for when the site gets destroyed.

 

[Rat]

Worst case scenario. You work in the WTC in New York. Then 9/11 happens. Does your business survive?

My instinctive reaction: why would I give a ****** It's not actually my business, and I'd be dead anyway.

Realistically, though, we could be back up running within the day as long as one of the IT department survived and we had office space to move into. If the entire IT department were killed (unlikely, since one of the five is in another office 100 miles away), then it would take a little longer; the backup procedure is documented in the Disaster Recovery Plan, which is accessible by any board member (actually hosted in my web space), and which details where copies are kept. If both the office and my boss's house and my house were all destroyed, then to be honest, the continuation of the business in a timely manner is not foremost in my thoughts.

 

[Rat]

General Disclaimer: I only understand every third acronym being used in this conversation. This is not my field at all.

Why make the Archive a separate physical media? Can't you just buy some hard drives and and do a nightly archive onto them? The prices seems to be dropping to the point where maintaining the DVD system or migrating to Blu-Ray will be more costly than the hard drives.

When I first read this, it sounded like a really good idea, but the more I reflect, the less realistic it becomes. We would have to have multiple sets, both for safety and for the reason rjh01 notes, i.e. that we often need to do quick restores of individual files. And not just restores from the past couple of days. We have a legal requirement to keep all data for, usually, ten years, although 'personal' information must not be stored thanks to the DPA, which is a headache in itself. Users will often ask for a version of a given file as it existed on a particular date some years ago. And because they're users, they sometimes can't remember the file name, or where it was saved, but that's another story.

Finally, of course, while a lot of people complain that optical discs have no longevity, the failure rate of HDDs is known to most people. I really wouldn't want to rely on one in the long term. While in the top-right of my avatar, you can just see the start of a row of CD and DVD cases containing the backups going back to about 1997, and I know they're perfectly readable, since I have to climb up and grab one from time to time, brush off the dust and spiders, and retrieve a particular disc for a requested restore.

 

[NewtonTrino]

Finally, of course, while a lot of people complain that optical discs have no longevity, the failure rate of HDDs is known to most people. I really wouldn't want to rely on one in the long term.

This is why I suggest a RAID array like a synology for the local backup. It's fast and one drive puking will won't cause data loss. Then hook it up to a cloud backup service for offsite. If you wanted to be extra protective I suppose you could make a copy now and then to blu ray like you're suggesting for longer term archiving. Personally I think distributing multiple copies in the cloud would be better.

 

[Rat]

Well the server is itself raided as it is, although I can't remember what flavour at the moment. And a good job too, what with the failure rate of the drives in the last server. But that doesn't cover us in the case of a fire. And as I say, we've had no luck with looking at online backups. Aside from the speed of the line, the 'cloud' would in fact be a single supplier of server space; while I would trust them to, for example, spread and duplicate the data in case of disaster at their end, there is the possibility that they could go out of business or simply stop providing that service, which would be a big headache to sort out.