We've hit capacity again

alchemista

Al_Mac wrote:

You see I'm a techie and that all went over my head:D Second paragraph was OK until the red bit. I tend to work to the lowest common denominator, the business, if they can't grasp it, we end up with more meetings. Plain English = less meetings = work gets done:cool:

scalable is a measure of how much slower something gets with extra load. If I add 10x more users to your website, will it get 10x slower? It's possible it stays the same speed, or it gets 10x slower (called linear growth), or it may even get much slower like 100x slower or 1000x slower (polynomial or exponential growth).

Serialization (i.e. doing things one at a time) is not scalable because the number of people waiting for their turn increases at a much faster rate than the time it takes to get one person's job done, which results in wait times getting very bad.

Never thought I'd be discussing comp sci and computer systems eng on MSE, but it's fun!

seanyseansean

alchemista wrote:

One of the top US deal sites, fatwallet:
http://www.fatwallet.com/c/18

*warning* - don't read it or you'll become depressed at how much better deals are in the US vs. UK! :rotfl:

That's different though. They use a lot more static content than over here which is more database driven and therefore harder on the servers.

alchemista

seanyseansean wrote:

That's different though. They use a lot more static content than over here which is more database driven and therefore harder on the servers.

ummm... no, they are all database driven, it's the same type of site, but yet it has a lot more users, more frequent posts, etc

seanyseansean

alchemista wrote:

A site I use that is much larger than this handles search by archiving posts that haven't had a reply in 90 days. The default search only searches topics in the current forum, or you can search on the top banner that will scan the whole site (full text search). If you want to search archived posts, you have to go through advanced search.

The site I'm talking about is plenty fast and has much higher numbers (over 10k concurrent users). So I think there might be more needed here, and hopefully you're looking at the entire app design and not just throwing hardware on it. Optimizing the software can pay you back much more than throwing hardware at something (and value for money is what this site's about!)

You don't need to do that. Search is heavy on the servers here precisely because of how different it is to normal board usage. It's a different problem but entirely solvable, as Google have amply demonstrated. I'll try to find examples of sites which have hived off the search from the forum software to something seperate, which is the correct approach.

One of my previous jobs (software developer for a major high street retailer) did exactly the same - Decision Support Systems (essentially the systems that search the stock/sales databases and perform what if? analysis) were hived off to a seperate read only Teradata server which was built for broad searches, using hashed indices and btrees etc. It took the major read load off the main system, where the usual load was atomic writes (as in a forum).

If the devs here need any (free) help then give me a pm!

Sean

seanyseansean

alchemista wrote:

ummm... no, they are all database driven, it's the same type of site, but yet it has a lot more users, more frequent posts, etc

Well a quick look at it shows it's a less dynamic content than over here. They seem to have stripped it down, which may or may not be appropriate here.

That said i'm not registered over there, and I don't know if the page generation is more complex if you're logged in there.

alchemista

seanyseansean wrote:

Well a quick look at it shows it's a less dynamic content than over here. They seem to have stripped it down, which may or may not be appropriate here.

That said i'm not registered over there, and I don't know if the page generation is more complex if you're logged in there.

Out of curiosity, what makes you think it's less dynamic? The link I showed is to their Hot Deals forum. It has a list of posts and topics just like here. It also has deal ratings which is more data per post than here. It keeps track of the same info (last post, thread origination date, last poster, thread author, etc). It has both in-forum title search plus full search at the top of the page. It has private messages just like here.

I'm just wondering what you think is different between the sites, just because it is a cleaner interface doesn't mean it's less dynamic. Maybe if you're seeing something that I'm missing, it's a clue to why this site is performing slowly (although this morning it seems to be fine, maybe just less load, or hopefully improvements have been made!)

seanyseansean

alchemista wrote:

Out of curiosity, what makes you think it's less dynamic? The link I showed is to their Hot Deals forum. It has a list of posts and topics just like here. It also has deal ratings which is more data per post than here. It keeps track of the same info (last post, thread origination date, last poster, thread author, etc). It has both in-forum title search plus full search at the top of the page. It has private messages just like here.

I'm just wondering what you think is different between the sites, just because it is a cleaner interface doesn't mean it's less dynamic. Maybe if you're seeing something that I'm missing, it's a clue to why this site is performing slowly (although this morning it seems to be fine, maybe just less load, or hopefully improvements have been made!)

I know what you mean, but as far as I can tell from watching the pages load it seems they cache the html statically for more of the page sent, whereas here it seems to be dynamic all the time. That might be because i'm not logged on there though!

If we talk about optimising the site though we need to know where the problem is - is it cpu,disk or bandwidth bound?I doubt it's the latter, else the simple fix would be to get rid of the many pics of Martins face scattered all over the board

alchemista

seanyseansean wrote:

I know what you mean, but as far as I can tell from watching the pages load it seems they cache the html statically for more of the page sent, whereas here it seems to be dynamic all the time. That might be because i'm not logged on there though!

If we talk about optimising the site though we need to know where the problem is - is it cpu,disk or bandwidth bound?I doubt it's the latter, else the simple fix would be to get rid of the many pics of Martins face scattered all over the board

Exactly, that's the point I was making early in the thread. You need to have real evidence of where the problem is, and code profilers, database statistics, system performance logs are what will tell you that. So many people jump to conclusions (in my experience, database people are the guiltiest!) and then waste so much time chasing it down.

I've seen people spend days on a database query without looking elsewhere. It often turns out the software doesn't need to even use that query, or the query can be cached, or the software only needs a subset, etc. So the database people waste their time while it could've (and should've) been fixed in the code above.

Sorry for the rant, it's just that it's a daily aggravation for me


I CAN NO LONGER POST ON THIS SITE, CAN A MODERATOR PLEASE UNLOCK MY ACCOUNT ASAP!

seanyseansean

alchemista wrote:

Exactly, that's the point I was making early in the thread. You need to have real evidence of where the problem is, and code profilers, database statistics, system performance logs are what will tell you that. So many people jump to conclusions (in my experience, database people are the guiltiest!) and then waste so much time chasing it down.

I've seen people spend days on a database query without looking elsewhere. It often turns out the software doesn't need to even use that query, or the query can be cached, or the software only needs a subset, etc. So the database people waste their time while it could've (and should've) been fixed in the code above.

Sorry for the rant, it's just that it's a daily aggravation for me

Tell me about it. Complaining that the database is hanging when some other bit of code is holding a full write lock on a cursor on a critical table is a particular favourite!

pennylane99

digitalspy forums have a lot more members and posts