Cambrian Line Radio Signalling failure - RAIB investigating

[caradoc]

Given that all information on maximum permitted speed, whether line speed, PSR or TSR, is transmitted direct to the cab, would it not have been possible, after the TSRs had been in place for 6 months, to convert them to PSRs, which would not have been affected by the technical failure ?

[Zomboid]

15 minutes ago, caradoc said:

Given that all information on maximum permitted speed, whether line speed, PSR or TSR, is transmitted direct to the cab, would it not have been possible, after the TSRs had been in place for 6 months, to convert them to PSRs, which would not have been affected by the technical failure ?

Technically that would probably work, but I think it would constitute a "Network Change", for which NR would have to jump through a lot of hoops and gain agreement from a lot of bodies. Which they should do if it is indeed permanent, but if the plan is to remove it in due course then that effort would be better spent actually fixing the problem.

[big jim]

A couple of the crossings that had the restrictions added ended up with a speed restriction akin to the sketch in father ted ‘speed 3’ with the big bunch of boxes in the road despite there being the same sort of visibility!

 

 

[The Stationmaster]

4 hours ago, caradoc said:

Given that all information on maximum permitted speed, whether line speed, PSR or TSR, is transmitted direct to the cab, would it not have been possible, after the TSRs had been in place for 6 months, to convert them to PSRs, which would not have been affected by the technical failure ?

That is actually what should happen but as  'Zomboid' has already noted that would require it to be put through the Network Change procedure and NR don't seem to be very good at that judging. by some past examples plus of course the TOC involved is almost n bound to object due to the impact on their timetable.  On the other hand with a TROS the TOC should be compensated for any consequential delays so they might be quite happy for a temporary speed to be left there ad infinitum.

I still think however that a pertinent question should be to assess how the British system of applying temporary restrictions of speed fits into the original thinking and architecture of the ERTMS software which has been used on the Cambrian.  While there has obviously been a hole in respect of assessment of the software would the original design of the software have not left such a hole because UK practice differs from , say, SNCF practice in respect of temporary restrictions of speed and how they are advised to Drivers?   It wouldn't be the first time on Britain's railways that software design has fallen into a potential hole, with safety critical consequences, because those specifying and designing it have either not been aware of or have not understood the operational practicalities of what they thought they were dealing with.

[Grovenor]

The report is quite clear that the French version revised the software to store TSRs in non-volatile memory which solved the problem, but that change was not carried through into the Cambrian version. The oddity is that the RAIB are not recommending that change to be made. It should be relatively simple as the software already exists to be ported across.

[david.hill64]

30 minutes ago, Grovenor said:

The report is quite clear that the French version revised the software to store TSRs in non-volatile memory which solved the problem, but that change was not carried through into the Cambrian version. The oddity is that the RAIB are not recommending that change to be made. It should be relatively simple as the software already exists to be ported across.

But it’s not just a software change. The hardware would also have to change. New safety cases required so altogether a very expensive mod on a little used line. 

[DY444]

It looks like three pretty fundamental design flaws that nobody picked up on which is mind blowing.  Even worse is that the sequence of events that led to this was an unhandled exception in a data validation function.  In my opinion that is pretty basic software stuff and I just can't get my head round the fact that there wasn't a default handler to catch unanticipated exceptions.   I have written my own suite of software to control my layout and if I got an unhandled exception in that I'd be mortified.  The thought of real railway signalling software being written in a such a way that exceptions can terminate mission critical threads without anyone knowing is worrying to say the least. 

[Grovenor]

14 minutes ago, david.hill64 said:

But it’s not just a software change. The hardware would also have to change. New safety cases required so altogether a very expensive mod on a little used line. 

Why would it need a hardware change? It would not be neccessary to have a technicians terminal in the French way and there is already non-volatile memory to hold the permanent data. Granted there is not enough detail info in the report to be sure either way.

[big jim]

The last few replies go some way to prove to me that good old line side signals are a much better idea than ERTMS (certainly on a line like the Cambrian) 

 

 

[TheSignalEngineer]

4 hours ago, big jim said:

The last few replies go some way to prove to me that good old line side signals are a much better idea than ERTMS (certainly on a line like the Cambrian) 

 

 

I think the Cambrian was chosen as the RETB was getting long kin n the tooth and a failure during the testing phase would not cause major disruption to the rest of the national network. 

Regarding system development times, let's not forget that when Beardy was taking on the WCML franchise Railtrack promised him that it would be fully operational providing cab signalling for a 140mph service by 2005. Some of us in the industry at the time were shouted down and sidelined when we suggested that introduction with the 2020 timetable was somewhat optimistic. I decided that enjoyong a long and happy retirement playing with my own trains was preferably to trying to fulfil their impossible promises.

Edited December 20, 2019 by TheSignalEngineer

[Zomboid]

1 hour ago, big jim said:

The last few replies go some way to prove to me that good old line side signals are a much better idea than ERTMS (certainly on a line like the Cambrian) 

 

 

It really just shows that ERTMS as installed on the Cambrian is not a mature technology. What it's trying to achieve should offer a lot of benefits over the lights on sticks approach, it's just not there yet.

 

Whether a line like the Cambian could ever realise the benefits is another matter, but there needed to be a trial site, and it's probably a better choice than the WCML.

[The Stationmaster]

2 hours ago, Grovenor said:

The report is quite clear that the French version revised the software to store TSRs in non-volatile memory which solved the problem, but that change was not carried through into the Cambrian version. The oddity is that the RAIB are not recommending that change to be made. It should be relatively simple as the software already exists to be ported across.

Yes - but it would have been to deal with the way the French publish TROS details and not the way they are published in the UK - there are fundamental differences between the two methodologies and it is wholly logical in the SNCF situation that they should make that change.  So if somebody took onboard (without checking where it resided in the memory) the fact that SNCF could reliably use the system for TROS information that would not actually resolve the case on the British network and if you don't know the differences you could be easily misled.

44 minutes ago, TheSignalEngineer said:

I think the Cambrian was chosen as the RETB was getting long kn the tooth and a failure during the testing phase would not cause major disruption to the rest of the national network. 

Regarding system development times, let's not forget that when Beardy was taking on the WCML franchise Railtrack promised him that it would be fully operational providing cab signalling for a 140mph service by 2005. Some of us in the industry at the time were shouted down and sidelined when we suggested that introduction with the 2020 timetable was somewhat optimistic. I decided that enjoyong a long and happy retirement playing with my own trains was preferably to trying to fulfil their impossible promises.

You took the right step.  Having done some ISA work on the system being worked up for the WCML that was the first place I found a safety critical hole in what the system was being written to do because whoever had done the spec for the software developer hadn't been properly informed about something by (alleged) 'experts'.   The net result, with a whiff of similarity to this case, would have resulted in trains running at line speed when they should have reduced speed for an ROS (which wasn't signed at the lineside).  Further proof of RIRO - which is hardly acceptable when things are safety critical and a salutary message that these things are about a lot more than just certifying the software - you really need to certify the parameters being handed to the software developer.

Edited December 20, 2019 by The Stationmaster

[corneliuslundie]

This is what you need. It will still be working long after several generations of software systems have gone to their home in the sky. Easily repaired and altered, easy to get a blacksmith to make replacement components. Simple!

Where are my hat and coat?

Jonathan

[TheSignalEngineer]

1 hour ago, corneliuslundie said:

This is what you need. It will still be working long after several generations of software systems have gone to their home in the sky.

In the mid-1980s there was a reorganisation in our department of BR which proposed doing away with locking designers, fitters and testers. The then RS&TE was sure that the need for them would soon be history. He suggested that as I and my sidekick could both design and test locking and one of the otjer reps was experienced at building locking and testing frames there wasn't a problem. I pointed out that at the current rate of progress there would be many frames left long after we had retired. Fifteen years so far and counting.

Edited December 20, 2019 by TheSignalEngineer

[Zomboid]

A couple of years back the NR CEO (or someone very senior) said that there were children who were yet to be born who would retire after a full career working on mechanical signalling.

 

It's still a labour intensive and operationally more restrictive way of telling trains where and when to go.

[Oldddudders]

1 hour ago, TheSignalEngineer said:

In the mid-1980s there was a reorganisation in our department of BR which proposed doing away with locking designers, fitters and testers. The then RS&TE was sure that the need for them would soon be history. He suggested that as I and my sidekick could both design and test locking and one of the otjer reps was experienced at building locking and testing frames there wasn't a problem. I pointed out that at the current rate of progress there would be many frames left long after we had retired. Fifteen years so far and counting.

The SR RS&TE, Cliff Hale, put a paper to Regional Management Group on pretty much the same lines at about the same time. But if his scheme engineers couldn't put a proposal together there would never be the budget provision, and hence no investment in all the whizzy kit he described. And we know where his career ended. 

[The Stationmaster]

21 minutes ago, Zomboid said:

A couple of years back the NR CEO (or someone very senior) said that there were children who were yet to be born who would retire after a full career working on mechanical signalling.

 

It's still a labour intensive and operationally more restrictive way of telling trains where and when to go.

I would argue about 'operationally more restrictive' as mechanical lever frames can just as readily work signals with all the advance information they can give a Driver about the way in which a train will be going,  In fact they could equally do that with semaphore signals.  But to provide maximum flexibility you need an awful lot of signals and large lever frames and there have over the years been semaphore signalled layouts offering considerably greater operational flexibility than the 'modern' signalling which replaced them.

 

And to provide sufficient line capacity in many places you would definitely need much more labour to do the job.  But critically i can't remember a time when a minor component failure in a traditional signal box caused anything like the level of chaos we have seen in recent years when a single item has failed in an area with the most modern signalling, even with similar levels of train frequency.

[martin_wynne]

19 minutes ago, The Stationmaster said:

I would argue about 'operationally more restrictive' as mechanical lever frames can just as readily work signals with all the advance information they can give a Driver about the way in which a train will be going,  In fact they could equally do that with semaphore signals.  But to provide maximum flexibility you need an awful lot of signals and large lever frames and there have over the years been semaphore signalled layouts offering considerably greater operational flexibility than the 'modern' signalling which replaced them.

 

A radio mast looks down on the ironmongery. Which will fail first?

 

 

Martin.

[johnofwessex]

6 hours ago, big jim said:

The last few replies go some way to prove to me that good old line side signals are a much better idea than ERTMS (certainly on a line like the Cambrian) 

 

 

 

Upper or Lower Quadrant?

[johnofwessex]

24 minutes ago, The Stationmaster said:

I would argue about 'operationally more restrictive' as mechanical lever frames can just as readily work signals with all the advance information they can give a Driver about the way in which a train will be going,  In fact they could equally do that with semaphore signals.  But to provide maximum flexibility you need an awful lot of signals and large lever frames and there have over the years been semaphore signalled layouts offering considerably greater operational flexibility than the 'modern' signalling which replaced them.

 

And to provide sufficient line capacity in many places you would definitely need much more labour to do the job.  But critically i can't remember a time when a minor component failure in a traditional signal box caused anything like the level of chaos we have seen in recent years when a single item has failed in an area with the most modern signalling, even with similar levels of train frequency.

 

 

Of course of all else fails than you have signalmen every few miles who can wave a flag if needs be

[Zomboid]

41 minutes ago, The Stationmaster said:

I would argue about 'operationally more restrictive' as mechanical lever frames can just as readily work signals with all the advance information they can give a Driver about the way in which a train will be going,  In fact they could equally do that with semaphore signals.  But to provide maximum flexibility you need an awful lot of signals and large lever frames and there have over the years been semaphore signalled layouts offering considerably greater operational flexibility than the 'modern' signalling which replaced them.

 

And to provide sufficient line capacity in many places you would definitely need much more labour to do the job.  But critically i can't remember a time when a minor component failure in a traditional signal box caused anything like the level of chaos we have seen in recent years when a single item has failed in an area with the most modern signalling, even with similar levels of train frequency.

Would it actually be possible to run trains at 100 mph+ at 2-3 minute headways using mechanical signalling and absolute block?

 

Would be economically impossible if not technically.

[johnofwessex]

How exactly does the signalling at Machynlleth work, it seems as though there is a ETRMS control centre there as well as the 'proper' signals from the old mechanical box

[big jim]

There is ERTMS but also a couple of real ground signals to allow you in and out of the depot/yard that are controlled by the same signalling centre, doing that allows movements around the shed without having to be in the system at level 1 

 

the old box was demolished when ertms came in

 

under RETB it was pretty similar in so much as you got to Machynlleth, gave up the token then reverted to semaphore in station limits and then went back to RETB when you departed 

Edited December 20, 2019 by big jim

[Oldddudders]

18 minutes ago, Zomboid said:

Would it actually be possible to run trains at 100 mph+ at 2-3 minute headways using mechanical signalling and absolute block?

 

Would be economically impossible if not technically.

Not 100 mph running, which is, after all, a relatively modern innovation, but closely spaced signalboxes in metropolitan areas have for many decades delivered intensive services at close headways. Mechanical signalling, and absolute block working, remain thoroughly  sensible ways of operating a safe railway. Only the manpower cost and availability issues are against them. 

[TheSignalEngineer]

Crewe North and South didn't have 100mph running but two power frames with Block working operated a much miore complex railway and traffic pattern than the current signalling. The Trent Valley line also kept mechanical frames working station/junction areas until recently using TCB between the boxes and a theoretical headway of about 2 minutes at 100mph (although only Network Rail bosses would think that was possible for timetabling purposes in the real world).