Jump to content
superfuture

SPLITTING MEGA THREADS

Recommended Posts

supertalk has REALLY LARGE topics [like waywt] that have 80,000+ posts and 14 million+ pageviews. it has been concluded by our forum software managers that this is the core reason we have been having recent performance issues.

we have been advised that we need to archive or split these large topics. we currently have 10 topics that are considered REALLY LARGE and beyond the capacity of MySQL. we have been recommended to cut size of these threads to 15,000 and page views to 1,000,000. there are currently 137,420 topics on supertalk.

quote:

With topics in the 80,000 replies mark, you have to realise that to fetch the information to that topic, MySQL has to loop through 80,000 rows in the database, and then offset those results to get the last page, and then limit that resultset to 10, and it's doing this on every single pageload of that topic. So if 10 members are reading that topic, that's 80,000 * 10, that's 800,000 rows MySQL has to loop through simultaneously.

Now MySQL is pretty fast when it comes to this sort of thing, but it's not designed to handle this sort of stress. That's just from your ONE 80,000 posts topic, meanwhile people re doing things on your forum all the time like making new posts, making new topics, viewing other BIG topics, and before sphinx was installed, Searching. etc.

archiving means they will be put into a read-only state. no rep, no replying, reporting or quoting.

splitting them into years means they will still be in an active state as separate topics.

SPLITTING THREADS IS DEFINITELY PREFERABLE TO ARCHIVING.

HOWEVER THERE ARE NO TOOLS YET AVAILABLE TO BULK SPLIT HUGE THREADS. WE CAN EITHER DO THIS MANUALLY BY MOVING POSTS PAGE BY PAGE [NOT PRACTICAL], OR BULK MOVE DIRECT VIA THE DATABASE THOUGH WERE ADVISED THAT IT IS VERY PROBLEMATIC TO DO SO.... SO WE ARE WAITING ON NEW ADVICE.

the plan is to start new threads and split them into years... i.e. 'waywt... 2011', 'waywt... 2012' etc.

eventually when tools are available, we will archive many older posts [probably those that have not had posts for 3-5 years].

the aim is to get the server load back down and get speed optimised.

Share this post


Link to post
Share on other sites

these are mega threads that need splitting...

++++++++++++++++++++++++++++++++++++++++++++++

supershoper

++++++++++++++++++++++++++++++++++++++++++++++

[HAS BEEN SPLIT] What are you wearing today? (pics only)

++++++++++++++++++++++++++++++++++++++++++++++

[HAS BEEN SPLIT] Recent Purchases. . .

  • Started by CBM, 15 Aug 2005

++++++++++++++++++++++++++++++++++++++++++++++

superdenim

++++++++++++++++++++++++++++++++++++++++++++++

[HAS BEEN SPLIT] Shoes that look better with age...

++++++++++++++++++++++++++++++++++++++++++++++

[HAS BEEN SPLIT] What Are You Wearing Today (Denim Version)

++++++++++++++++++++++++++++++++++++++++++++++

Levi's Vintage Clothing

++++++++++++++++++++++++++++++++++++++++++++++

Samurai Jeans

++++++++++++++++++++++++++++++++++++++++++++++

Self Edge, SF/NY/LA - updates & info

  • Started by kiya, 27 Oct 2006

++++++++++++++++++++++++++++++++++++++++++++++

supertrash

++++++++++++++++++++++++++++++++++++++++++++++

[HAS BEEN SPLIT] waywt? saved my life

++++++++++++++++++++++++++++++++++++++++++++++

superconfessional

++++++++++++++++++++++++++++++++++++++++++++++

Edited by superfuture
updated

Share this post


Link to post
Share on other sites

next problem...

how to split mega threads?

--------------------------------------------

superfuture

29 August 2012 - 06:19 PM

hi alan,

ok. so we are going to accept this as the final diagnosis of 2 months of troubleshooting...

we have always been in agreement about splitting large topics and have already looked at this.

the problem is how we do it...

the plan is to start new threads and split them into years... i.e. 'waywt... 2011', 'waywt... 2012' etc.

however there are no ACP tools yet available to bulk split huge threads. i imagine we can either do this manually by moving posts page by page [not practical] or bulk move direct via MySQL though we were previously advised by IPS that this was very problematic to do so... so we need some more advice here. we posted already on IPS forums but no responses.

how are you advising other clients to split large threads?

thanks

]wayne

--------------------------------------------

Ryan

29 August 2012 - 11:34 PM

Hello,

Unfortunately, it will need to be done manually on a topic by topic, page by page, basis.

You could potentially do it using MySQL, however as stated before, that could prove to be problematic and I would not recommend testing doing so on your live site.

Regards,

Ryan

--------------------------------------------

superfuture

30 August 2012 - 12:16 AM

hi ryan,

there is no tool which lets me 'select all' posts on a single page. [i know we can do that with topics - but cannot locate this in posts]. does it exist? or are you saying we need to move every single post by clicking its checkbox one by one?

if we can 'select all posts' page by page and temporarily increase the posts per page from 50 to say a 1,000 or 10,000 then this would be doable. otherwise we would need to go through about 2 million posts one by one...

which would not be much fun at all.

can you please advise.

thanks

wayne

--------------------------------------------

Alan

30 August 2012 - 05:57 AM

Hello,

Unfortunately there is no "Select all posts" check box when viewing a topic by default but it may be able to be added with a custom modification.

I recommend posting in the forums for assistance with this or contacting one of the 3rd-party developers in the directory.

Many thanks,

Alan

Share this post


Link to post
Share on other sites

we can try and find someone to help with a custom modification but that might take time.

in the meantime...

+++++++++

1. going to just start on one of the threads above to see how we go.

+++++++++

2. rename the NEW thread xxxxxxxx [2012]

+++++++++

3. add a message to the existing OLD thread and a link to the NEW thread

--> will keep the OLD thread UNLOCKED for now --- however everyone should start posting in the NEW thread --- and mods will move anyway.

+++++++++

4. move posts.

--> we will then start manually moving all the posts from 2012 from the OLD thread to the NEW thread.

--> going from most recently posted page by page.

--> since this will take a LOT of TIME it might be realistic to only go back a month or so.

+++++++++

5. if that all works then we can do with the other threads.

+++++++++

in theory, since most users are viewing the most recent posts, this is likely where the server load issue is concentrated, so... if we take the pressure off the OLD large threads we might solve this issue without having to to go back too far... ie. maybe only 3 months or so?? not sure if this logic is correct but it's just a hunch...

Share this post


Link to post
Share on other sites

start with the smallest of the ones to be split

Share this post


Link to post
Share on other sites

it's not a big deal really, lots of forums start new threads when they hit a certain number (something like 1000)

i think you should split the larger ones, as i'd presume this would alleviate more of the problems straight away…

Share this post


Link to post
Share on other sites

obviously split the larger ones first as per the explanation of your tech support the larger the thread the more work MySQL has to do

incorporate a rule that will either split topics anually or after they hit a certain postnumber

Share this post


Link to post
Share on other sites

start in superdenim

Share this post


Link to post
Share on other sites

have started with

What Are You Wearing Today (Denim Version)

-->

WAYWT? [Denim Version] 2012

!!

UPDATE: we have moved 186 584 posts - going back to AUGUST 1st MAY 29th.

[that still took about 1 3 hours... which is slow but not as slow as we thought...]

this is the first thread we have started moving so if you have any feedback or notice anything weird let us know on THIS thread.

!!

Edited by superfuture
status update

Share this post


Link to post
Share on other sites

moving posts in that first thread seemed to go smoothly so going to try the big ones next...

What are you wearing today? (pics only)

-->

WAYWT? [pics only] 2012

!!

UPDATE: we have moved 168 posts [9 pages] from the past 3 month - going back to june 1st.

if you have any feedback or notice anything weird let us know on THIS thread.

!!

Share this post


Link to post
Share on other sites

ok. next...

waywt? saved my life

-->

WAYWT saved my life in 2102

!!

update: so far we have moved 214 633 posts going back to august june 1st.

if you have any feedback or notice anything weird let us know on THIS thread.

!!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Luisa via Roma (US)
    Brand - 125 x 125