Surveys - how reliable?

[GnosticBrian]

I've just completed the BRM survey.

 

Question: "How many shows do you attend"

Answer: "None"

 

Question: "What is your reason for attending shows - four options offered none of which is 'I don't attend shows'"

Answer: I had to tick one of the four options; NONE of which apply to me.

 

Utility of surveys where questions don't cover the range of reasonable responses?

[gwrrob]

Yes, I said I'd been modelling for over 15 years even though I'm still a teenager....

[Enterprisingwestern]

 

When surveyed, 62% said they didn't trust them, 29% said they did and the other 14% weren't bothered one way or the other.

 

Mike.

[rab]

I've recently experienced this from the other side.

I'm part of a team which is trying to do some community work,

and we thought it would be useful to find out what people thought

were the strengths and weaknesses of their community.

 

We came up with what we thought was a suitable questionare,

but when we went through the responses we found some where

people had totally misunderstood the questions. We thought

we'd worded it in a way to get the info we wanted, but someone

else looking at it read it differently.

 

Having said all that we were a bunch of amateurs 'having a go'.

Businesses should have access to the right sources to be able

to put together meaningful surveys.

[TheQ]

As far in company surveys are concerned, i've normally found they are rigged to provide the answer they are looking for...

[ian]

 

[Oldddudders]

I believe there are companies that provide a service compiling questionnaires to help your business. Obviously there is competition among them to win your custom. It is my observation that the winning bid invariably includes the most questions. Which means I seldom complete one. 

[Huw Griffiths]

20 hours ago, GnosticBrian said:

I've just completed the BRM survey.

 

Question: "How many shows do you attend"

Answer: "None"

 

Question: "What is your reason for attending shows - four options offered none of which is 'I don't attend shows'"

Answer: I had to tick one of the four options; NONE of which apply to me.

 

Utility of surveys where questions don't cover the range of reasonable responses?

 

This reminds me of an exercise a few years back, in which a company were trying to gauge potential viewer reactions to a TV science series.

 

Before and after seeing a pilot show, people were asked some multiple choice questions about their preferences in presenters - some of whom a number hadn't heard of before.

 

Unfortunately, the only answer options available were on a scale from "amazing" to "diabolical" - with no "who's that?" option.

 

 

6 hours ago, TheQ said:

As far in company surveys are concerned, i've normally found they are rigged to provide the answer they are looking for...

 

I've encountered enough of those - including one in which an employer went through the motions of "consulting" their staff about a change in working conditions, which would prevent staff with families from taking annual leave during school holidays.

 

For some reason, this idea wasn't too popular with the staff - but management were intent on imposing this change, whilst pretending that this might somehow be by popular demand - so they came back with another survey, with no "status quo" option and the other options calculated to be even less popular.

 

Strangely enough, they didn't get many completed surveys back ... .

 

 

I believe there's a whole discipline devoted to this sort of thing - in which people are invited to choose between an officially approved option (which might not always be too popular with people given a free choice) and some "nuclear option" (which has been calculated and worded to be unpopular).

 

It's called "choice architecture" - and it appears to be favoured by some politicians - with well known examples including a certain referendum a few years back, for which both the referendum itself and what's happened since have acquired some notoriety.

 

This is neither the time nor the place to repeat arguments either way (and I continue to actively avoid expressing my own view in public).

 

However, I think it's fair to say that a number of people on both sides of this divide now seem to view the whole exercise as a textbook example of what can go wrong with surveys and the like, which haven't been adequately thought through beforehand.

 

 

Huw.

[Stubby47]

I hope you've also replied direct to BRM to express your dissatisfaction with their survey questions.

 

I often point out such stupidity in surveys at work - eventually it does get through ( not least that I'm now asked to proof read stuff )

[Keith Addenbrooke]

Prize looks nice though.

[ruggedpeak]

On 18/09/2019 at 18:55, GnosticBrian said:

I've just completed the BRM survey.

 

Question: "How many shows do you attend"

Answer: "None"

 

Question: "What is your reason for attending shows - four options offered none of which is 'I don't attend shows'"

Answer: I had to tick one of the four options; NONE of which apply to me.

 

Utility of surveys where questions don't cover the range of reasonable responses?

Joined 2015, first post above.

[F-UnitMad]

What BRM Survey, anyway??

 

[Steamport Southport]

4 minutes ago, F-UnitMad said:

What BRM Survey, anyway??

 

 

The one we all got sent. Unless you opt out of emails.

 

Check your inbox.

 

 

 

Jason

[F-UnitMad]

14 minutes ago, Steamport Southport said:

 

The one we all got sent. Unless you opt out of emails.

 

Check your inbox.

 

 

 

Jason

Ah, gotcha. Thanks.

[Huw Griffiths]

4 hours ago, F-UnitMad said:

What BRM Survey, anyway??

 

4 hours ago, Steamport Southport said:

The one we all got sent. Unless you opt out of emails.

 

Check your inbox.

 

4 hours ago, F-UnitMad said:

Ah, gotcha. Thanks.

 

I'd go along with this assessment.

 

In my case, I haven't recently had much time to check stuff like this.

 

Now that I know where to look, I've completed it myself.

 

In my case, the question I take issue with is the one about annual spend on the hobby - with the lowest option being "£1500" - and all the rest significantly more than this.

 

I'm assuming that this one was actually meant to read along the lines of "< £1500" - in which case I can actually answer it. (I've never had that much available to spend on any hobby - and I doubt if I'm alone.)

 

Otherwise, I've got no issues with this survey - and I can't see too many people objecting to the prize for one, very lucky, person who completes the survey.

 

 

Huw.

[F-UnitMad]

On 20/09/2019 at 00:35, Huw Griffiths said:

I'd go along with this assessment.

 

In my case, I haven't recently had much time to check stuff like this.

 

Now that I know where to look, I've completed it myself.

Exactly. The email title was along the lines of "Do you want to win a Western Pullman?".

No, I don't, actually, so I hadn't opened the email, so I didn't know there was a survey in it. I do like to mess with surveys though - modelling mostly outside the "4mm British" Mainstream, these things can be quite entertaining, & if by a million-to-one chance I win the Western Pullman, it'll provide useful funds for models I do want.

[TheSignalEngineer]

18 minutes ago, F-UnitMad said:

if by a million-to-one chance I win the Western Pullman, it'll provide useful funds for models I do want. 

My thoughts exactly. However one thing can lead to another

 I gained a weathered 'Super D' through winning a competition when I was designing a West Country layout. That layout never got built but I've now got a collection of 40 locos which appeared in the West Midlands in the late 1950s. 

[Darth Vader]

Typical of survey producers, they don't get the end user to try out the survey to find out if there are bugs or things that don't work.

If you put out a survey that is flawed then the results will be meaningless.

[Dungrange]

The answer to reliability also depends on the sample size.  If you get ten completed surveys, it's difficult to draw meaningful conclusions: get ten thousand and it should be easier to draw meaningful conclusions.  That said, if questions are flawed, then it doesn't really matter how many people complete the survey.

[john new]

On 20/09/2019 at 00:35, Huw Griffiths said:

 

 

 

I'd go along with this assessment.

 

In my case, I haven't recently had much time to check stuff like this.

 

Now that I know where to look, I've completed it myself.

 

In my case, the question I take issue with is the one about annual spend on the hobby - with the lowest option being "£1500" - and all the rest significantly more than this.

 

I'm assuming that this one was actually meant to read along the lines of "< £1500" - in which case I can actually answer it. (I've never had that much available to spend on any hobby - and I doubt if I'm alone.)

 

Otherwise, I've got no issues with this survey - and I can't see too many people objecting to the prize for one, very lucky, person who completes the survey.

 

 

Huw.

There were several where both, all or none/other would be correct but were missing. DC or DCC was one IIRC, I am currently trialling DCC (a yes tick) but most locomotives are still analogue (a yes tick for that). Also fully agree regarding the less than £1,500 question.

 

A key question I thought was missing, not gender, but do you model solo, with family or in a club/group setting.

Edited September 21, 2019 by john new

[Steamport Southport]

1 hour ago, john new said:

There were several where both, all or none/other would be correct but were missing. DC or DCC was one IIRC, I am currently trialling DCC (a yes tick) but most locomotives are still analogue (a yes tick for that). Also fully agree regarding the less than £1,500 question.

 

A key question I thought was missing, not gender, but do you model solo, with family or in a club/group setting.

 

"Do you belong to a club?" was in there ISTR.

 

As for the £1500 question. Far more I'm afraid. Apart from drinking and music, it's the only hobby I have, so everything spare goes on things like trains, books, heritage railways. etc. I took it to mean the whole hobby.

 

 

 

Jason

[john new]

8 minutes ago, Steamport Southport said:

 

"Do you belong to a club?" was in there ISTR.

 

As for the £1500 question. Far more I'm afraid. Apart from drinking and music, it's the only hobby I have, so everything spare goes on things like trains, books, heritage railways. etc. I took it to mean the whole hobby.

 

 

 

Jason

 

Club question - I'd forgotten that one then. apologies for the error.  

[The Stationmaster]

I shot for the moon in the question about forums when it came to 'other' where I asked for Adult Level discussion on modelling and the prototype. (although I should have said 'consistently' of course  )

[Alex TM]

Hi everyone,

 

I try to avoid surveys where at all possible.  My reasons are two-fold:

 

• I used to work for an 'organisation' that could make survey results say almost anything that was wanted; and
• In my current path of life I have been asked to complete surveys that range in function from time-wasting to voyeuristic!

 

Regards,

 

Alex.

[PeterKem]

It would appear that for the Location: Other "Spain" is not allowed.

So my views not going in then!

 

Peter