What's happening with the Forums

BorisTheSpider_3

EdInvestor wrote: »

This being my third experience of a site losing a large quantity of quality information due to backup failure, I'd like to inquire of the techies here (without seeking to ascribe any blame to anyone) just what exactly the problem is with backups?

Do technical people value the information contained on database type websites (such as this one) or do they tend to view the site content as just a pile of old threads which don't really have a value (and thus don't need to be backed up)?

I would think they do, as technical people are far more likely to be users of such sites, they would know the value of the data.

The problem is not so much the technical grunts, but at a management level. In other words, Joe Techie backs up the servers every day, but John SiteOwner hasn't given thought to what will actually happen when things go wrong, and whether his techies are backing up the right things.

Techies can back things up, but you need management to arrange Disaster Recovery exercises.

I think very few site owners really think about what would happen to their site and data when things go wrong. It is something that must be considered.

ZTD

jamesd wrote: »

ZTD, if both stop it'll usually be a master problem (damaged binary log - bug probably - or network breakage). If only one slave stops then yes it'll be something for that slave.

It's something in the slave code. Both slaves stopped - but never at the same time. To be fair, it was always under heavy load.

jamesd wrote: »

If you have two slaves the easiest way to fix significant breakage (completely out of sync and not recoverable) is usually to copy from the working one to the broken one. Or even do it just because it has predictable time to fix and is easy once you have the routine in place.

You can't do that. In order to get a consistant database from the slave you'd need to stop the database (either full lock or down it), and then do the copy. That would mean you were no longer resiliant.

The *easiest* way is to just wipe the database on the faulty slave, set the transaction number back to zero and then point it at the master again. The slave then completely rebuilds the database from the beginning from the transaction log.

Takes time, but no more time that copying it from the other slave.

jamesd wrote: »

Can't really say why it's tough for you to get the slaves started again, it's usually pretty easy. Don't really want to get into the details, though - do enough of that elsewhere.

They usually stop because the values/structure of the database is different from what the db engine was expecting, so the transaction can't go through. It's almost impossible to fix because you don't know what values where would be acceptable. The logs in that situation are less than helpful...

silvercar

Remember that the people posting on here are by definition forum users. The majority of users of MSE will be users of the main site and readers of the weekly email

I gues it is far easier and cheaper to back up the main site. For non-forum users, MSE life goes on as normal.

The forum must take up a disproportionate amount of time, money and resources. If the forum is not at full strength while this back up is being restored at least most users are unaffected and the addicted posters can help newcomers by repeating posted advice from memory for a short time.

What I'm trying to say is that addicted forum users (me included) will notice the problems far more than the vast majority of MSE users; so the catastrophe is not actually as bad as reading this thread would let us believe.

EdInvestor

Thanks guys for the answers.:) So the bottom line appears to be this:

BorisTheSpider wrote: »

Backups are the most important part of any internet site, but without testing them in a disaster recovery situation they are often useless.

and this

very few site owners really think about what would happen to their site and data when things go wrong.

Could this possibly be because most site owners (not being techies) aren't aware that the protection they think they have already got is likely to be useless in a disaster? :rolleyes:


Regarding the question of selective protection of a site - so as to make proper backups (and testing of the backups) economic, how would it work on a site using database software like V Bulletin, as here (ignoring the "editorial" part of the site which is not VB)?

Can you protect individual forums - or categories- but not others?How about individual threads?

ZTD

EdInvestor wrote: »

Could this possibly be because most site owners (not being techies) aren't aware that the protection they think they have already got is likely to be useless in a disaster? :rolleyes:

Well I think to be fair to Martin, it's not his responsibility to be aware how good the backups are. He pays for them; they should work.

The managers of the people who actually run the site are the people who should have a tested disaster recovery plan. But often/usually this testing isn't done because people grow complacent about reliability.

EdInvestor wrote: »

Regarding the question of selective protection of a site - so as to make proper backups (and testing of the backups) economic, how would it work on a site using database software like V Bulletin, as here (ignoring the "editorial" part of the site which is not VB)?

Can you protect individual forums - or categories- but not others?How about individual threads?

Depends on how you do it, but it's best not to. The most important stuff is the stuff you can't recover. It's probably a law...

caliston

Another thing to mention is that catastrophe usually happens when two or more things go wrong. For example:

Aircraft engine on fire AND pilot shuts down the wrong engine
Ship hits iceberg AND interior bulkheads poorly placed
Reactor core overheating AND thermometers not working
Dirty linen stacked next to railway sleeping car heater AND ventilation system drawing smoke into compartments
X-ray machine had poorly written software AND no hardware protection to prevent overdosing patient

(all real disasters)

The challenge is coming up with a backup system that can cope with multiple points of failure. This is difficult.

EdInvestor

ZTD wrote: »

Well I think to be fair to Martin, it's not his responsibility to be aware how good the backups are. He pays for them; they should work.

Quite so, that's exactly what I meant.

The managers of the people who actually run the site are the people who should have a tested disaster recovery plan. But often/usually this testing isn't done because people grow complacent about reliability.

Is it that - or is it that the backups are inherently not really up to the job? Or is it perhaps a combination? If the backups are on the whole unlikely to work then it would be helpful for non-techie site owners to know this, so they can make other arrangements.

Depends on how you do it, but it's best not to. The most important stuff is the stuff you can't recover. It's probably a law...

Backups do sound like a waste of time and money.

ZTD

EdInvestor wrote: »

Is it that - or is it that the backups are inherently not really up to the job?

Backups do work. However, they are a contraction of "Backup and Restore". Loads of people check the backups work, but very few check the restore.

EdInvestor wrote: »

Or is it perhaps a combination? If the backups are on the whole unlikely to work then it would be helpful for non-techie site owners to know this, so they can make other arrangements.

Backups do work, it's just you only ever hear about them not working. But think of it this way: Imagine you go to your back door every night, and throw a ball over your shoulder to land in a bucket in your neighbours garden. Every night you throw it the same way.

What could go wrong?

In a situation like that - it's best to go to the fence and check where its landed...

But that very rarely happens in the world of backups.

EdInvestor wrote: »

Backups do sound like a waste of time and money.

They are a waste of money like every insurance. Unless your house burnt down, your house insurance is money wasted. However if it did burn down, then you'll be thanking your lucky stars...

EdInvestor

ZTD wrote: »

Imagine you go to your back door every night, and throw a ball over your shoulder to land in a bucket in your neighbours garden. Every night you throw it the same way.

What could go wrong? ;)In a situation like that - it's best to go to the fence and check where its landed...

I'd have thought it would be best to go first to the fence and check if the bucket had been moved, myself.

But I'm not a techie

Thanks for the help. I think I see where the problem lies now.

libra10

Thanks for keeping us informed, much appreciated.