We've hit capacity again

Streeet

Have the guys at MSE got any 'finger in the air' launch dates for the technology refresh?

You've got four seconds!
I guess the database error page will start pushing users away in frustration soon.

seanyseansean

andi2 wrote:

You should be running a SQL query to do the search, something that looks like SELECT .... FROM .... WHERE .... LIKE %search%.

If you can post that query on the site, or PM me direct, i'll be able to let you know what you need to do.

No offence, but that's useless advice. LIKE will always do a table space scan (i.e. ignore the indexes and reread all the data) and will have to be queried multiple times, once for each search word.

Martins suggestion is what's really needed - a seperate search box, updated whenever a post changes, that holds b-trees of words/stem words and in which posts they are used.

EDIT: Also it would be worth it, if the servers have enough memory, to stick memcached into the system. It does a fantastic job of keeping large datasets quick.

Former_MSE_Webmaster

Hi folks,

Just to keep you updated on our progress. To answer you Streeet; I'm holding my finger in the air and it's feeling a little bit like later this week. We're pushing very hard to get the new servers integrated before Christmas.

seanyseansean: memcached has been holding us together for the last 3 weeks but even that's not enough.

Woof: Deleting threads is not what we're wanting really. We prefer to keep all the content and have a setup that can cope. Saying that, we are looking into possibly archiving some old data.

Hang in there folks, perhaps Father Christmas will bring us an early pressie!


Cheers,

Webby

Kilty_2

MSE_Webmaster wrote:

Saying that, we are looking into possibly archiving some old data.

Was just about to suggest that

pavlovs_dog

very sloooooooow tonight

alchemista

A site I use that is much larger than this handles search by archiving posts that haven't had a reply in 90 days. The default search only searches topics in the current forum, or you can search on the top banner that will scan the whole site (full text search). If you want to search archived posts, you have to go through advanced search.

The site I'm talking about is plenty fast and has much higher numbers (over 10k concurrent users). So I think there might be more needed here, and hopefully you're looking at the entire app design and not just throwing hardware on it. Optimizing the software can pay you back much more than throwing hardware at something (and value for money is what this site's about!)

Former_MSE_Neil

Hey All,

We are currently working on this - both from a software and hardware perspective. The new hardware still needs a bit of fine tuning and this is what we are looking at initially. Hopefully very soon the site will be up and running at full speed very soon.

Sorry for any problems that you are experiencing in the meantime

Thanks

MSE Neil

nelly_2

alchemista wrote:

A site I use that is much larger than this handles search by archiving posts that haven't had a reply in 90 days. The default search only searches topics in the current forum, or you can search on the top banner that will scan the whole site (full text search). If you want to search archived posts, you have to go through advanced search.

The site I'm talking about is plenty fast and has much higher numbers (over 10k concurrent users). So I think there might be more needed here, and hopefully you're looking at the entire app design and not just throwing hardware on it. Optimizing the software can pay you back much more than throwing hardware at something (and value for money is what this site's about!)

what site is this you mention?

alchemista

Al_Mac wrote:

alchemista,

Problem is these days, hardware is so cheap, modern software designers and programmers have no insentive to do a proper job.

No comments about this place, more about where I work

A software algorithm that runs in exponential time will make any hardware obsolete quickly. Changing that algorithm to a logarithmic or even polynomial will scale up on the same hardware thousands of times more than an exponential algorithm.

Sometimes, you can't even throw enough hardware at a problem to fix it. For example, one early poster recommended serializing all searches (only allowing one person at a time to search). That is a non-scalable approach (actually it is anti-scalable) because the more users trying to search, the queue depth grows worse than linearly.

Anyway, hope the MSE guys can get it working!

alchemista

nelly wrote:

what site is this you mention?

One of the top US deal sites, fatwallet:
http://www.fatwallet.com/c/18

*warning* - don't read it or you'll become depressed at how much better deals are in the US vs. UK! :rotfl: