New forum -- Indexing Progress

[martin_wynne]

Here's an indication of how the indexing is going -- until it's finished a lot of content will appear to be missing.

 

Searching for "Templot" currently produces 15 results after starting earlier today at 12 (yesterday it was 9, the day before was 6). Previously on RMweb there were several pages of results. Which gives an indication of the rate of indexing progress.

 

i.e. 15 results when searching topics using this URL:

https://www.rmweb.co.uk/community/index.php?/search/&q=Templot&type=forums_topic&sortby=newest

Searching Google's cache of the previous version using this search string:

Templot site:rmweb.co.uk/community/

produces 239 results. So progress so far after 2 days is 15/239 = 6%, or 3% per day.

 

At this rate, full indexing is going to take 30+ days. It is likely to speed up and vary widely for different content, but even so -- don't hold your breath.

 

Martin.

Edited February 4, 2019 by martin_wynne
title change

[martin_wynne]

Now up to 17.

 

Only 222 to go.

 

Martin.

[Matt C]

so is that 17 items out of 239 indexed for templot on a site with 3.5 million items to index ?  It doesn't sound so bad put like that  

 I think I've read in the other thread some members saying they cant find some content. . . . which they havent actually used for years anyway but are upset that right at this moment it isn't instantly to hand in case they did !  I totally understand the frustration at not seeing all the content that they follow in thier 'content that I follow' but mine seams to be catching up with itself and I'm now seeing threads I follow and getting notifications of new posts as these threads get active again. A  question I've been meaning to ask that I'm sure you have the knowledge to answer,

If members visit a thread they have been following and browse it and post, does that then get indexed as the site software pulls it up ? or is the indexing a completely different affair ?

Edited February 3, 2019 by Mattc6911

[martin_wynne]

3 minutes ago, Mattc6911 said:

A  question I've been meaning to ask that I'm sure you have the knowledge to answer,

If members visit a thread they have been following and browse it and post, does that then get indexed as the site software pulls it up ? or is the indexing a completely different affair ?

 

Hi Matt,

 

I don't have access to the code, but I imagine that if a topic is referenced in any way by a current user -- posting, liking, following, etc., that topic has to go to the top of the indexing queue and be indexed immediately. Otherwise it wouldn't be possible for the topic to be updated and for the page renderer to create the updated topic pages.

 

There does seem to be two different indexing tasks taking place: 1. to create the search index of all the words used in 3.5 million posts, and 2. to find and relink all the attached images into posts. I don't know to what extent those two tasks, and possibly others, are integrated. If someone can see a post from which the images are missing, try liking it and see if the images appear any sooner.

 

p.s. "Templot" now up to 19, so things are speeding up.

 

cheers,

 

Martin.

[martin_wynne]

Update, Searching for "Templot":

 

RMweb indexer: 23 results so far.

 

Google: 244 results.

 

Martin.

[martin_wynne]

Something good has happened to the indexer.

 

An hour ago: 84 results.

 

Now: 253 results.

 

We may be closer to its finishing than we thought.

 

Martin.

[martin_wynne]

And now 268. 

 

Anyone wishing to watch this race should click: https://www.rmweb.co.uk/community/index.php?/search/&q=Templot&type=forums_topic&sortby=newest

 

Martin.

[Andy Y]

As a little background as to why I decided to go live on Friday sooner than I would have ideally liked these are the indexing options presented within the software. Logical in hindsight but it just didn't feel like it at the time.

 

 

 

[martin_wynne]

24 minutes ago, Andy Y said:

As a little background as to why I decided to go live on Friday sooner than I would have ideally liked these are the indexing options presented within the software. Logical in hindsight but it just didn't feel like it at the time.

 

Thanks for that Andy. It seems odd that the Default option isn't the Recommended one.

 

p.s. "Templot" now up to 430 results.

 

cheers,

 

Martin.

[MarshLane]

14 minutes ago, martin_wynne said:

 

Thanks for that Andy. It seems odd that the Default option isn't the Recommended one.

 

p.s. "Templot" now up to 430 results.

 

cheers,

 

Martin.

 

Martin,

The recommended option would require the installer to know about CRON jobs, how to set them up on the server etc... which not everyone using the software may know, hence I suspect, why the default is site driven.

 

Rich

[martin_wynne]

Something definitely changed in the indexer today. This morning searching for "Templot" produced only a couple of dozen results on a single page, a full 3 days after the new forum went live.

 

But in the space of a day it has now jumped to 3,125 results on 125 pages, and is increasing faster than I can type.

 

I find it had to believe that there could be so many genuine results, but each page I have looked at seems to contain all valid entries. I'm wondering what the count might be if searching for something more general, such as "Peco".

 

Martin.

[AY Mod]

The blue notification bar saying that indexing was in process seems to have disappeared from the top of the Activity pages now and checking back to some old topics that haven't been touched since 2009 they all seem to have their pics so I think we're there in terms of pics and quote formatting etc. I need to have a look at post counts and reputation scores etc and see if there's any black holes.