Are reviews of new models independent or not?

[The Stationmaster]

There has been some discussion recently in another  thread about the degree of independence (from a 'manufacturer') of someone who posts reviews online.   The whole question of the independence, or allegedly otherwise, of reviewers in our hobby's magazines or via online media is in many respects a very hoary chestnut.  But it might well be worth opening a decent debate and possibly even reaching some sort of conclusion.  Hopefully some reviewers will be prepared to contribute and give us their viewpoint.  Before I go any further I should add that on various occasions I have been asked by regular reviewers for information about a particular prototype when they are reviewing a model of it.  That has been limited to information which helps them to judge the visual realism of the model and its detail against the real thing.  

 

 I try to take any review that I read - be it on paper, or pixels, or moving pixels,  in the way in which it is presented and the points it makes or doesn't make.  For example when it comes to detail there was, for example, an inaccurate review of a recent KR Models loco in a magazine and accurate reviews or comments in pixellated media (as well as inaccurate ones).   This is not, in my experience, the first time this has happened and it certainly isn't limited to a single 'manufacturer'.    By all means debate or draw attention to perceived errors and the cause of them but that doesn't mean you have to attack a reviewer or make accusations about any association you might think exists between them and the supplier of the model.

 

We can form your own opinions of both a review, and maybe by inference also of the reviewer, in whatever way we wish.  We no doubt value it accordingly against what we see for ourselves when it refers to the look or detail of a model and so on,  or even apparent performance when it comes to moving images.  But that is your value that you attach to a reviewer.  That opinion will be personal, and quite possibly subjective,  and there is no need to share it with anybody else.  Other people will form their own opinions and set their own values of what a reviewer comes up with.  And - again - there is no need to plaster it all over the 'net or - particularly  important on RMweb when it comes to fellow RMweb members - make what amount to accusations whatever your personal opinions might be.  By all means draw attention to perceived inaccuracies on. or sloppy construction of, a model ir a review of it but you don't need to comment about the reviewer in order to do that.

[john new]

Not models, but directly related. I review books for the SLS Journal, mine are totally independent. 
 

Edited January 5, 2023 by john new

[Phil Parker]

To supply a bit of background for the magazine world, to my knowledge, no manufacturer has ever threatened to pull advertising because of a poor review (@dibber25 has said the same in the past, and he's far more history in doing this than me). Even if they did, the loss would be a page an issue in most cases, they generally don't take much in the way of adverts, compared to the retailers.

 

Even a "bad" review is worth having. Landing a loco on @AY Mod's desk will cost around £100. He'll then photograph it, research it, write it up, and the resulting page space will be worth several times the cost of sending the model. Supplying items for review makes commercial sense.

 

Reviews are aimed at the average reader. Some people are fussy that all the bolt heads on a model are not rotated the same way (they are, at least one manufacturer has told us they rotate them to shut people up) but most aren't that picky. Does it look right and work? For the majority, that's what matters, and to be honest, they judge that by looking at the photos. Unless you can put numbers on something, it's largely subjective, and what matters to you might not worry others. The evidence for that is models that have been pilloried by some, going to a second production run because the first one sold out. Everyone has different standards.

 

We don't get to pick the models that arrive - so it is perfectly possible that those sent in are specially checked. But, we'd notice this on the box as it wouldn't be sealed. Sometimes duff models do arrive, and if possible, are swapped for working ones so a proper review can be carried out. We only get to review one example, and it has to be a quick process to meet deadlines, because readers expect reviews when a model appears, not several months later. Long term reviews, the sort of thing car mags do, aren't practical because we don't have time to "run" the model for miles. Even if we did, how many readers would be willing to pay for a magazine that carried them out? This is a small world, unlike the car enthusiast one.

 

Our job is not to slag everything off for the enterainment of the reader. We'll try to be as accurate, balanced and consitent as possible. Do you belive in reviews? That's up to you, but we know people read them because they tell us. As Mike says,

 

23 minutes ago, The Stationmaster said:

We can form your own opinions of both a review, and maybe by inference also of the reviewer, in whatever way we wish.  We no doubt value it accordingly against what we see for ourselves when it refers to the look or detail of a model and so on,  or even apparent performance when it comes to moving images.  But that is your value that you attach to a reviewer.  That opinion will be personal, and quite possibly subjective,

 

Which seems sensible. Ultimatly, the decision to buy, or not, should be down to the individual. If you like a model and want to run it on your railway, that decision is yours to make.

[Phil Parker]

Just now, john new said:

I review books for the SLS Journal, mine are totally independent. 

 

Do you buy the books, or are they supplied?

[john new]

6 minutes ago, Phil Parker said:

 

Do you buy the books, or are they supplied?

Supplied at random. If it is a book I have personally bought I say so, as often that is a niche subject and that either gets mentioned or listed with just an out of five star rating. 
 

Edited January 5, 2023 by john new

[Compound2632]

Mike has touched on the question of how much knowledge a review has, or can reasonably be expected to have, of the prototype of the model under review. If the purpose of the review is to answer the question "does the model look like the real thing", then ideally the reviewer needs to have access to at least as much information as the manufacturer has (one hopes) accumulated, with (one hopes) considerable time and effort. That's obviously not going to be the position a magazine's staff reviewer will be in, generally - hence the calls upon Mike's expert knowledge. So should reviews be farmed out to acknowledged experts? But (one hopes) the manufacturer will also have consulted such experts, so they will not be independent...

 

But the reviewer is also reviewing the model as a model. Does it work? Do its wheels, couplings, etc. conform to accepted standards for the scale / gauge? It its price reasonable, for the standard of construction and state of the market? For these questions, a staff reviewer, with their expert knowledge of the hobby, is probably better placed. 

[AY Mod]

2 minutes ago, Compound2632 said:

For these questions, a staff reviewer, with their expert knowledge of the hobby, is probably better placed. 

 

Also a staff reviewer will be more likely to see a cross-section of models from different manufacturers across different scales etc. to be able to make better comparative assessments for the mass market than a specialist may. If I have questions about specifics that need clarification I'll often speak to a project's researcher or an informed or knowledgeable person so it can be taken into account. Sometimes that may mean I get told "we couldn't do x because of reason y " and I try to include that where appropriate.

 

Often those that demand reviews should be something akin to bear-baiting are not the paying customer and would always be disappointed as that's the way they're wired. Occasionally someone at a manufacturer wants my head on a spike; hey ho.

[Phil Parker]

24 minutes ago, john new said:

Supplied at random.

 

Then you are in the pocket of "Big Transport Publishing", that multi-billion pound industry that controls the world of train books with an iron fist. You probably review them on a yacht while drinking copious G&T's 😆

 

(in truth, anyone who reviews books deserves a medal. It's a lot of work for very little reward)

[MarkSG]

Personally, I'd say that reviews are one of the most important features of magazines. They're being written by people who, for the most part, are doing this as a job, and therefore have a lot of experience of comparing different models and appraising them effectively. And, at least from my own reading of those reviews, I'd say that the reviewers are honest and unbiased - you don't get reviewers who, say, routinely slag off Hornby while gushing over Bachmann, or whatever. They review the model in front of them, not the reputation of the manufacturer. And because they're working as part of a team, under the direction of the editor, there is a degree of consistency there. It's the one part of magazine output that really can't be replicated by consumers or amateur website operators or solo YouTubers.

 

That's not to say that consumer reviews of models they've bought (eg, on the review sections of products on retailer websites) aren't also useful in their own way. In particular, consumer reviews can provide the one thing that @Phil Parker points out that the magazines can't, which is give a long term assessment of a model that's been running for months. Consumer reviews of the original Hattons/Heljan Garrett, for example, accurately pointed up problems with long term running that simply wouldn't - and couldn't - have been apparent to a magazine reviewer. But many people are not all that good at either making an honest assessment of a product or, even if they are, effectively conveying that to the reader. And consumer reviews can be, and often are, biased. So consumer reviews have to be treated with a degree of suspicion. Their value tends to be more in the aggregate than individual reviews - if a lot of buyers say the same thing about a model, then they're probably right, but if only one of them says it then it's probably an opinion that can be disregarded.

 

I'm not a huge fan of video reviews (eg, on YouTube). Partly, that's because they're typically solo opinions without any editorial oversight, although that's an unavoidable consequence of the system rather than a criticism, and I'm aware that some video reviewers do take their role very seriously and have put a lot of work into their assessment skills. I certainly wouldn't want to tar them all with the same brush. But it's mainly because I can read a written review in five minutes that would take me 30 minutes to watch on a video, and, frankly, life's too short! But, on the other hand, video reviews do give the opportunity to see the model in action, which can be genuinely useful. That's obviously something that a magazine can't do, at least in the printed edition, but it would be nice if they could also put short clips on their websites of their review models negotiating their test tracks. It doesn't need to be lengthy, and it doesn't even need a commentary - just a title and description linking it to the written review, and then include the URL of the video in the printed magazine.

[Phil Parker]

20 minutes ago, Compound2632 said:

So should reviews be farmed out to acknowledged experts?

 

Possibly, but then you'll need to wait for months while such an expert is located, and has time to crawl over the model with a fine tooth comb. Then the review will need to run to many pages. By which time the thing won't be in the shops. Oh, and you'll need the one expert everyone agrees with. It's not unknown for experts to disagree with each other, often quite loudly...

 

23 minutes ago, Compound2632 said:

It its price reasonable, for the standard of construction and state of the market?

 

That one is subjective. There is always someone who thinks it should be cheaper!

[john new]

1 minute ago, Phil Parker said:

 

Then you are in the pocket of "Big Transport Publishing", that multi-billion pound industry that controls the world of train books with an iron fist. You probably review them on a yacht while drinking copious G&T's 😆

 

(in truth, anyone who reviews books deserves a medal. It's a lot of work for very little reward)

Thanks. Semi close too - I do live at the seaside and close enough to the beach to be in the spray zone but no yacht! No sitting out to read books though this time of year, too many westerly gales at the moment, and we face west.

[MarkSG]

1 minute ago, Phil Parker said:

(in truth, anyone who reviews books deserves a medal. It's a lot of work for very little reward)

 

I used to review CDs for a music magazine. That varied from the delightful to the painful, and every step in between.

[The Stationmaster]

22 minutes ago, Compound2632 said:

Mike has touched on the question of how much knowledge a review has, or can reasonably be expected to have, of the prototype of the model under review. If the purpose of the review is to answer the question "does the model look like the real thing", then ideally the reviewer needs to have access to at least as much information as the manufacturer has (one hopes) accumulated, with (one hopes) considerable time and effort. That's obviously not going to be the position a magazine's staff reviewer will be in, generally - hence the calls upon Mike's expert knowledge. So should reviews be farmed out to acknowledged experts? But (one hopes) the manufacturer will also have consulted such experts, so they will not be independent...

 

But the reviewer is also reviewing the model as a model. Does it work? Do its wheels, couplings, etc. conform to accepted standards for the scale / gauge? It its price reasonable, for the standard of construction and state of the market? For these questions, a staff reviewer, with their expert knowledge of the hobby, is probably better placed. 

I'm sure that I'm not the only person who is asked for informnation when a reviewer receives a model where they are not familiar with the prototype.  But doing so can sometimes help the reviewer - in respect of detail and realism of appearance - and I, for one, am certain that nobody in this hobby knows everything about everything.   But I agree absolutely with the point made particularluy by Andy Y regarding the consistent and comparative approach that you will get from an experienced reviewer.

[newbryford]

48 minutes ago, Phil Parker said:

To supply a bit of background for the magazine world, to my knowledge, no manufacturer has ever threatened to pull advertising because of a poor review (@dibber25 has said the same in the past, and he's far more history in doing this than me). Even if they did, the loss would be a page an issue in most cases, they generally don't take much in the way of adverts, compared to the retailers.

 

Didn't Bachmann pull advertising from Rail Express (or Rail - can't remember the date) after a review of the early model of the 37?

Edited January 5, 2023 by newbryford

[jjb1970]

This is a perennial question in just about every hobby where stuff is reviewed. For model trains I've never seen shill reviews as an issue. I may not agree with a review but I don't think they are dishonest or working for model producers. That said I do think that reviewers should either maintain a separation from suppliers, as magazines generally do, or alternatively offer full transparency so people are aware of any relationships or things that might influence a review. This is different from knowing people. The world is really a collection of small bubbles and within those bubbles people tend to know each other and it's only natural people form friendships, but most people separate that from business. I don't think it's an issue in our little bubble, as others have already said the manufacturers are not the major advertisers and the effort of doing a review with professional photography and a high quality write up and the work to prepare it for publishing will be more than the cost of a free model. 

[melmerby]

20 minutes ago, MarkSG said:

I'm not a huge fan of video reviews (eg, on YouTube). Partly, that's because they're typically solo opinions without any editorial oversight, although that's an unavoidable consequence of the system rather than a criticism, and I'm aware that some video reviewers do take their role very seriously and have put a lot of work into their assessment skills. I certainly wouldn't want to tar them all with the same brush. But it's mainly because I can read a written review in five minutes that would take me 30 minutes to watch on a video, and, frankly, life's too short! But, on the other hand, video reviews do give the opportunity to see the model in action, which can be genuinely useful. That's obviously something that a magazine can't do, at least in the printed edition, but it would be nice if they could also put short clips on their websites of their review models negotiating their test tracks. It doesn't need to be lengthy, and it doesn't even need a commentary - just a title and description linking it to the written review, and then include the URL of the video in the printed magazine.

I'm the same.

Wading through a 30 minute YT review to find the relevant bits can be extremely tedious, skipping through the almost inevital 'unboxing' and various other pointless comments to find the 5 mins of the actual model being tested.

 

(It doesn't just apply to model reviews but to most "how to do" videos, where three lines of text would suffice.

[leisuresl]

16 minutes ago, newbryford said:

 

Didn't Bachmann pull advertising from Rail Express (or Rail - can't remember the date) after a review of the early model of the 37?

 

That wasn't a threat though, they just went ahead and did it.

 

Regards,

[jjb1970]

There are two extremes of the reviewing spectrum - the shill review and the outright hatchet job. Neither is positive and thankfully I think both are rare in model railway magazines (for what it's worth I think it's not that common in YT model reviews either). However, I do think Rail Express crossed the line at the hatchet job end of the spectrum at times, there was one infamous editorial where anyone buying models they didn't approve of was said to have blood on their hands. Which I found full on bonkers.

[Compound2632]

35 minutes ago, Phil Parker said:

Possibly, but then you'll need to wait for months while such an expert is located, and has time to crawl over the model with a fine tooth comb. Then the review will need to run to many pages. By which time the thing won't be in the shops. Oh, and you'll need the one expert everyone agrees with. It's not unknown for experts to disagree with each other, often quite loudly...

 

Quite. That's why one needs the staff reviewer - a journalist, a trained communicator, one such as yourself - to mediate between the expert and the reader.

 

38 minutes ago, Phil Parker said:

That one is subjective. There is always someone who thinks it should be cheaper!

 

But the professional reviewer understands the industry and so can make a better judgement as to whether the price is reasonable.

[Enterprisingwestern]

 

My 2 cents worth.

Although reviews can be "helpful" they are not, nor can they be, the be all and all of a multi year operation of a manufacturer.

At the end of the day its the reviewers opinion, formed from whatever information he has at his disposal, that colours his judgment, one only has to look at Trustpilot etc reviews and see the same shop/car/restaurant given between one and five stars, impossible IMHO.

I shall keep on caveating and emptoring based on my own thoughts.

 

Mike.

[jjb1970]

As with any product, if you like it then it's irrelevant if others don't. Just as if you don't like something then a glowing review is irrelevant. Not so much with model railways but I regular enjoy books, movies and music which are slated by reviewers and dislike stuff which wins all sorts of awards. 

[kevinlms]

One of the most interesting reviews I read, wasn't of a model, but the prototype.

 

The Royal Auto Club Victoria, did a review of the Flying Scotsman when it was in Australia in 1987. It read in the style of a review of a car, but for the Flying Scotsman!

 

I had a copy, but I've lost it! But it gave various tables of dimensions, power output and general statistics, such as mileage and pointed out IIRC, that it wasn't a 'one owner vehicle, driven to church on Sundays'!

 

The article went on to say that it's towing capacity, was where it really came into it's own and was an ideal vehicle for extended high speed running with a few hundred tons behind it!

 

[Legend]

I do believe their is an inherent bias towards manufacturers , certainly in the press and maybe some You Tube reviewers that become too close to a retailer or maybe some of the smaller manufacturers.

 

The reason is not having adverts pulled or threats . It is simply that the Model Railway Industry is not big , people all know each other manufacturers , managers , designers, editors , reviewers , there really is a very small circle, I bet most of us could name these people. Friendships are struck up acquaintances made. I've always thought of it as a Gentleman's club where help and praise is appreciated but criticism is just not the done thing .

 

So I enjoy reviews in magazines , I enjoy reviews on You Tube but in the back of my mind I'm thinking there really wont be any revelations here. You know its going to score 9.7 out of 10 or the reviewer will find "nothing of note" .  So I look at them all but the reviews that count most are from modelers or enthusiasts who have gone out and bought the model and telling you what they found. What they like , don't like . I really don't care about sprung buffers or if its diecast but do you get good value - does it run well?  As an example of a good review have a look at a channel called EuviRail , particularly his latest class 92 review.  Quite open , honest , not criticising manufacturer , but pointing out there was an issue. 

[Robin Brasher]

The Hornby R6911 LSWR Bauxite Brake van reviews seem to be examples of a bias towards manufacturers.  As far as I am aware the LSWR never painted their brake vans bauxite. I am sure the magazine editors know more about LSWR brake vans than I do yet the Railway Modeller was the only magazine that mentioned the error in the livery in passing. The only way a purchaser can correct the error is to strip down the paint and repaint the model in the correct livery and re-letter it.

 

Soon afterwards Hornby issued the same brake van in more or less the correct chocolate brown livery.

 

It looks like Hornby are finding it difficult to sell the bauxite brown models because they are still in their catalogue and probably in their Christmas sale. Anyone reading the reviews in most magazines would be none the wiser as to whether the brake vans were in the correct livery.

 

[AY Mod]

14 minutes ago, Robin Brasher said:

yet the Railway Modeller was the only magazine that mentioned the error in the livery in passing.

 

Incorrect.

 

@Tony Wright in BRM March 2020 ...

 

Quote

It’s in the matter of liveries where there might be some concern over these models. Not the BR grey one – that’s fine, but both the LSWR and SR examples raise some issues. At the Grouping, the SR chose the LSWR very dark brown/plum colour for its brake vans. Though both the LSWR and SR example have the correct red ends, they’re really nearer BR bauxite, which is a much lighter colour. I know colour perception is very subjective, but I think the ‘brown’ is much too light.

 

How much more explicit should that point be?