Scheduled Forum Downtime Fri 26 (from 7pm) – Sun 28 July 2013

meher

fermi wrote: »

Can't imagine them changing the forum permissions just so DT can be viewable during the read only period. Too much of a "car crash" in there.

you should read posts on dt in a cartoon format it can be ever so funny

Midnighter

torbrex wrote: »

We will all be 'lurkers' for the weekend, I enjoy a good lurk

Hopefully it will be just the weekend. I use another site which operates on the same OS as MSE and they went down last weekend and are still not back up almost a week later.

jamesd

MSE may already have done a practice run on a test server using Tuesday's work to ensure that they know in advance that the Friday-commencing work will happen without trouble. Smaller places may not have the expertise, server access or resources to do this sort of thing well. Bigger ones with money and serious money to lose from outages may.

For the type of database server I think they are using the standard approach for server upgrades with high uptime requirement is to take a snapshot then use the binary logs to continue grabbing data while the new server setup goes on in the background. Once that's done, replication is used to catch up and when all is ready and testing complete the switch is thrown to fail over to the new one within a second or so. But since I don't know just what they are doing and what resources they have available to them, I really don't know if this is viable.

Might be something to chat about in email sometime if they are interested.

leaphaze

Is this why Martin is taking a break?

theblagger

jamesd wrote: »

MSE may already have done a practice run on a test server using Tuesday's work to ensure that they know in advance that the Friday-commencing work will happen without trouble. Smaller places may not have the expertise, server access or resources to do this sort of thing well. Bigger ones with money and serious money to lose from outages may.

For the type of database server I think they are using the standard approach for server upgrades with high uptime requirement is to take a snapshot then use the binary logs to continue grabbing data while the new server setup goes on in the background. Once that's done, replication is used to catch up and when all is ready and testing complete the switch is thrown to fail over to the new one within a second or so. But since I don't know just what they are doing and what resources they have available to them, I really don't know if this is viable.

Might be something to chat about in email sometime if they are interested.

Sounds like you know your stuff James

pawlala

Midnighter wrote: »

Hopefully it will be just the weekend.

Enough time for the new server IPs to propagate to all visitors. The forum will be read-only because, until such point your browser points to the new site, you will be trying to post to the database on the old server which won't exist in 48-72 hours time.

jamesd

pawlala wrote: »

Enough time for the new server IPs to propagate to all visitors. The forum will be read-only because, until such point your browser points to the new site, you will be trying to post to the database on the old server which won't exist in 48-72 hours time.

It's insecure to have a database server directly connected to the internet. Best practice is to use one of the unroutable IP address ranges and firewall, then access the database server only via a gateway using a VPN connection to that gateway. In a hosted environment I would also expect MSE to have their own firewalled private network address range so that other hosting customers can't access their data on the wire or by other means. In addition, if viable, I'd expect their web servers to use one network port for connections to the database server(s) and another for web-facing traffic, to further reduce the chance of trouble if one of the web servers was somehow compromised.

More highly secure setups could use a middleware layer between the web servers and database that constrained the types of query that the web servers can send, so even a compromised web server has the damage potential limited. It's not very likely that MSE has done this.

Best practice for high load places is also not to use DNS for the database server connections if they are using the one I think they are using, because that makes DNS an avoidable point of failure that can also sometimes cause performance issues.

I don't know how the MSE web servers that we interact with, perhaps via a caching reverse proxy server, are set up but it's quite likely that they read the database server address from a configuration file or setting and will use the new one from the moment that is updated. If DNS is required, the MSE team would probably have reduced the time to live of their DNS entry so that it expires rapidly rather than staying cached for more than a few minutes. The web servers themselves may sit behind a load balancer like LVS or even in hardware.

For internal use it's often easier still just to have a service IP address for the database server and point that service address to the real one so that the web boxes continue sending to the same IP address but it now goes to the new server. It's a bit like automatic call forwarding - everyone calls your known number but the place the calls goes to changes as you need it to change.

The longer DNS-related delays are more likely to happen only when people are having to do a surprise change, not for planned ones.

When a database server is the reason, a forum would be made read only while a new server is having data loaded, to avoid any changes being made to the old one that wouldn't then be included in the new one. This is not strictly necessary but constraints within the MSE systems or budget may make it so even though it's usually easy to avoid this for planned changes if you have the resources.

I don't know the specific limitations of the systems used by MSE so there may be constraints that make some of these normal practices unworkable for them. Such constraints can always overrule normal practice when required and I'm sure the team is doing what makes sense for their own situation. It's unlikely that anything I've written is even slightly surprising to them, who always strike me as capable folks.

If anyone is curious and bored over the weekend and wants to see a bit about how popular places do things you might want to look at this not so well maintained page for an overview then the details here. Though this presentation might be more readily digestible.

jamie11

Good luck with the update, don't do what landlordzone did. Crash!

Luckyone

Huge Thanks to all concerned in getting us back up & running so quickly

:T:T:T

meher

yes fab team, thanks, i actually couldn't view dt thogh, been locked out

btw i can see something curious

about skype numbers along with thanks counts

would that be a glitch?

will post a screen shot in a mo