Jump to content
 

Cambrian Line Radio Signalling failure - RAIB investigating


Recommended Posts

  • RMweb Premium

If this can go wrong, what else?

Just about anything that involves "human intervention". Will be following this one closely (I have a professional interest). Not wishing to jump to conclusions before the investigations and report have been completed, but it will be interesting to see what level of testing was done after the "routine shutdown".

 

Regards, Ian.

  • Like 3
Link to post
Share on other sites

Turn It off and turn it on again?

 

Standard 'IT' response :jester:

 

Sorry, couldn't resist. :mosking:

Unfortunately it was just that the began the issue.

 

Ironically it is no longer the defacto answer to issues, because of linked systems there is actually a methodology to restarting computer systems and if a step is missed or in the wrong order then all sorts of issues can occur - in my line of work it is very unlikely to be fatal but who knows what a chain of events can lead to.

Link to post
Share on other sites

  • RMweb Gold

Let's not start the ETCS vs Lineside Signalling Debate again, which I can see this descending into...

 

Simon

 

No need for that Simon but what does worry me about this incident (and a past one on the Cambrian) is what seems to be a divergence from long established signalling maintenance procedures where other than traditional signalling equipment is involved.  In other words Module TS 11 and the use of Form RT3187 does not appear to specifically apply to various ERTMS situations whereas in my opinion any work at all on any part of the ERTMS software or hardware ought to be covered by it complete with full testing after work has finished and before the equipment is signed back.

 

The impression I get - rightly or wrongly - that some things associated with ERTMS are regarded as 'computer stuff' rather than an essential part of the signalling system.  So for example even swapping a card in a computer or comms rack should be subject to TS 11 etc if it has anything at all to do with ERTMS, and that obviously would also include a re-boot of any part of the system.

  • Like 1
Link to post
Share on other sites

  • RMweb Premium

No need for that Simon but what does worry me about this incident (and a past one on the Cambrian) is what seems to be a divergence from long established signalling maintenance procedures where other than traditional signalling equipment is involved.  In other words Module TS 11 and the use of Form RT3187 does not appear to specifically apply to various ERTMS situations whereas in my opinion any work at all on any part of the ERTMS software or hardware ought to be covered by it complete with full testing after work has finished and before the equipment is signed back.

 

The impression I get - rightly or wrongly - that some things associated with ERTMS are regarded as 'computer stuff' rather than an essential part of the signalling system.  So for example even swapping a card in a computer or comms rack should be subject to TS 11 etc if it has anything at all to do with ERTMS, and that obviously would also include a re-boot of any part of the system.

I would imagine there is a specific section within SMTH that deals with ERTMS/ETCS "re-instatement" following maintenance and/or fault rectification. However, whether or not it is yet detailed enough to cover every eventuality remains to be seen.

 

Regards, Ian.

Link to post
Share on other sites

Unfortunately it was just that the began the issue.

 

Ironically it is no longer the defacto answer to issues, because of linked systems there is actually a methodology to restarting computer systems and if a step is missed or in the wrong order then all sorts of issues can occur - in my line of work it is very unlikely to be fatal but who knows what a chain of events can lead to.

 

There should never really be a methodology to restarting computer systems because there is always the possibility of a power cut that will not respect any methodology, most especially in linked systems.

 

Safety critical systems should be robust enough to withstand such things and recover safely and nowadays, thanks to data regulation (you don't really want to fall foul of), even trivial systems can be deemed safety critical.

 

Standard testing for large systems, hit them with an off switch then see what happens, you be amazed how often developers have neglected to consider such eventualities.

Link to post
Share on other sites

Just about anything that involves "human intervention". Will be following this one closely (I have a professional interest). Not wishing to jump to conclusions before the investigations and report have been completed, but it will be interesting to see what level of testing was done after the "routine shutdown".

 

Regards, Ian.

 

Some of these issues can be one offs and may be very difficult to replicate, then the worry is it could happen again, though I doubt Cambrian drivers need to rely on computers to remind them where the speed restrictions are.

Link to post
Share on other sites

  • RMweb Premium

No need for that Simon but what does worry me about this incident (and a past one on the Cambrian) is what seems to be a divergence from long established signalling maintenance procedures where other than traditional signalling equipment is involved.  In other words Module TS 11 and the use of Form RT3187 does not appear to specifically apply to various ERTMS situations whereas in my opinion any work at all on any part of the ERTMS software or hardware ought to be covered by it complete with full testing after work has finished and before the equipment is signed back.

 

The impression I get - rightly or wrongly - that some things associated with ERTMS are regarded as 'computer stuff' rather than an essential part of the signalling system.  So for example even swapping a card in a computer or comms rack should be subject to TS 11 etc if it has anything at all to do with ERTMS, and that obviously would also include a re-boot of any part of the system.

 

One the of main reasons for some of the early failures of the Ely to Norwich line resignalling was eventually tracked back to fibre cards being pulled at the coms centre in the midlands, without any thought for what was running over them....

 

In the telecoms world, anything of critical nature would be duplicated, with the duplicate proved to be working first, and then you do the work on it, and then prove it is working correctly on load, and then after a week or so, do the other system, so that you have half a chance of things doing what they should....

 

And we never relied on UPS systems either, we always have duplicated power units in the servers, fed from two different UPS inverter stacks, driven off properly designed DC rectifier/battery units. There has been many instances of UPS failures taking out whole rafts of critical stuff, not just on the railway either.....

 

From a signalmans point of view, we never trust anyone who says it won't affect the passage of a train, but then again, would the bobbies have even known this work was being done? and if they did, would the computer man know how to fill out the signalling disconnection form (as he would need a SMTH qualification to do so)?

 

Another load of grey areas here methinks....

 

Andy G

Link to post
Share on other sites

More worrying than whether the system was restarted correctly, is the fact that it was malfunctioning without the signallers having any knowledge of the fact. It was reported by a driver several hours later.

 

Martin.

What this does demonstrate is the need for drivers to have proper route knowledge rather than relying on technology and carrying on blindly.

Link to post
Share on other sites

What this does demonstrate is the need for drivers to have proper route knowledge rather than relying on technology and carrying on blindly.

Yet here we are in the 21st century talking about self driving taxis in parts of the world in the next 12 months - what could possibly go wrong with a driverless car in the middle of a city.

 

I know it's cars not trains but sometimes I feel that rules for planes and trains that are there for good reason somehow can never be applied to road transport which oddly has such a higher incidence of fatalities.  Almost like a country with lax gun controls bizarrely has more gun related fatalities than other similarly advanced countries with strict gun controls.

Link to post
Share on other sites

  • RMweb Gold

Does this mean then that on ERTMS-equipped routes, Temporary Speed Restrictions are not indicated by any equipment on the ground, and the only information given to the Driver is in-cab ?

Yep, you get a symbol in the DMI (in cab screen) where the restriction starts and ends and you also get counted down to it by the speed curve

 

The beauty of it is should a restriction be imposed it’s instant l, similarly once lifted its instantly removed by the signaller

 

What this does demonstrate is the need for drivers to have proper route knowledge rather than relying on technology and carrying on blindly.

From what I can make out it was reported fairly rapidly by a driver (ok several hours may have passed since the reboot) so we’ll done to him for spotting it, TBH the restrictions in question have as the report says been in place a long time so they would have been pretty conspicuous by their absence

 

Regards route knowledge, ERTMS is supposed to near as dammit do away with route knowledge/learning but incidents like this only highlight the proper need for it, when the system is down or indeed as I did 4 times last week if you are working within a possession you need to know where block markers are as if you went past the wrong one then it’s a spad but also you can’t as with TPWS override passing the authority on the move, you have to stop and do it (a block marker is the end of authority and there may be one/some within the possession you have to pass) so you need to know their locations, ok you are doing 40kph max (less in the possession) but you certainly don’t want to come screaming up to the marker and slam the brake in last second!

 

There are from a drivers point of view a number of different forms (written orders) for various situations and there is one giving permission to proceed following failure or disconnection of signalling equipment, that situation may come about after a power outage or something where the signalling centre may have lost contact/position report from multiple trains and will not pick up a location until they pass over a Balieze and axle counters

 

.......now the other week in my “down by the track thread” I said learning ERTMS was driving me mad and was struggling with it, can you see why?

Edited by big jim
  • Like 1
Link to post
Share on other sites

  • RMweb Gold

One the of main reasons for some of the early failures of the Ely to Norwich line resignalling was eventually tracked back to fibre cards being pulled at the coms centre in the midlands, without any thought for what was running over them....

 

In the telecoms world, anything of critical nature would be duplicated, with the duplicate proved to be working first, and then you do the work on it, and then prove it is working correctly on load, and then after a week or so, do the other system, so that you have half a chance of things doing what they should....

 

And we never relied on UPS systems either, we always have duplicated power units in the servers, fed from two different UPS inverter stacks, driven off properly designed DC rectifier/battery units. There has been many instances of UPS failures taking out whole rafts of critical stuff, not just on the railway either.....

 

From a signalmans point of view, we never trust anyone who says it won't affect the passage of a train, but then again, would the bobbies have even known this work was being done? and if they did, would the computer man know how to fill out the signalling disconnection form (as he would need a SMTH qualification to do so)?

 

Another load of grey areas here methinks....

 

Andy G

 

And your penultimate paragraph comes back to exactly my point.  The person doing it isn't some much a 'computer person' or a comms person as an S&T Technician when doing anything at all, even hundreds of miles away which might impact in some way or other on the signalling system.  and in ERTMS everything is part of the 'signalling system'.  Looks to me as if somebody failed to think this area through in their rush to grab clever technology but perhaps the RAIB inquiry will root out that gap (assuming it is the gap it appears to be).

 

Although oddly an ORR railway Inspectorate investigation into some broadly similar events (but not involving ERTMS) a number of years ago was cancelled just as it was getting underway.

  • Like 2
Link to post
Share on other sites

  • RMweb Gold

Why are there "temporary" speed restrictions on the approach to crossings to protect pedestrians? Is there a new race of higher-speed pedestrians currently in training to use the crossings, when the TSRs will be lifted?

 

Because the sighting committee when out and deemed that a number of crossings (north of Dovey junction) no longer have sufficient sighting time for pedestrians to see a train approach

 

Unfortunaly from a drivers point of view (more specifically a class 97 drivers point of view) these restrictions can add 10-20 mins to a ‘section’, ERTMS is set up so you input what type of train you are, standard 97, air brakes passenger 75mph, air brakes goods 60mph (there are more) which then calculates your braking curve

 

Herein lies the problem, the computer currenly can only assume you have 12 vehicles on, you can’t change that in the cab, so will give you a braking curve for 12 vehicles which as you can imagine is a fair distance, I take a 97 down there on a test train, 97+3 vehicles+97=5 vehicles

 

I approach a 10m long 25kph restriction over a foot crossing , Computer says I have to do 25kph 100 meters before said restriction And will not let me do any more, then I go across the restriction at 25kph BUT once clear the computer is then counting down 7 ‘invisible coaches’ across the restriction, however it then adds another 100m to the rear of the restriction before it opens up to allow me to accelerate, unfortunaly there are a number of these restrictions in a short space between Dovey and towyn so you quite simply just can’t get going!

 

This is the future

Link to post
Share on other sites

  • RMweb Premium

Why are there "temporary" speed restrictions on the approach to crossings to protect pedestrians? Is there a new race of higher-speed pedestrians currently in training to use the crossings, when the TSRs will be lifted?

 

I'm guessing that the crossings will be upgraded to either provide additional warnings of approaching trains, or perhaps build a bridge and close the crossing(s). If not, then the TSR might become a PSR by default? Which then could raise a whole new issue of how do you control (reliably) PSRs via ERTMS/ETCS?

 

Regards, Ian.

Link to post
Share on other sites

Why are there "temporary" speed restrictions on the approach to crossings to protect pedestrians? Is there a new race of higher-speed pedestrians currently in training to use the crossings, when the TSRs will be lifted? 

 

I'm guessing that the crossings will be upgraded to either provide additional warnings of approaching trains, or perhaps build a bridge and close the crossing(s). If not, then the TSR might become a PSR by default? Which then could raise a whole new issue of how do you control (reliably) PSRs via ERTMS/ETCS?

 

Regards, Ian.

 

I have just spent half an hour going over the ETCS reference guides for designing ETCS schemes with a colleague due to this very incident.

 

In there, it says that the term "Temporary Speed Restriction" under ETCS includes Temporary, Emergency and Permanent Speed Restrictions. The TSR is applied through a TSR system, presumably by the technician on the technicians terminal next to the interlocking, which is then is included in the Movement Authority when that is transmitted to the train.

 

So I suspect that they aren't Temporary Speed Restrictions to protect the crossings, instead they could be Permanent Speed Restrictions, but the RAIB has used the ETCS definition.

 

Also, remember, this is not an ERTMS Line, I know that rulebooks etc. state that it is, but under the correct terminology, it is only an ETCS Level 2 fitted line, ERTMS is yet to be commissioned into use anywhere in the country.

 

Simon

Link to post
Share on other sites

  • RMweb Gold

I have just spent half an hour going over the ETCS reference guides for designing ETCS schemes with a colleague due to this very incident.

 

So I suspect that they aren't Temporary Speed Restrictions to protect the crossings, instead they could be Permanent Speed Restrictions, but the RAIB has used the ETCS definition.

 

 

Nice to see someone with a passion for his work Simon!

 

Anyway I’ve asked the question of one of my drivers as to whether the restrictions come up in the DMI as temporary restriction (with associated symbol) or does the planning area just change as per PSR’s that will answer the question as to how they have been inputted, I’m he tto sign beyond Machynlleth so cant speak from experience of being in the drivers seat, I’ve only been in the 2nd mans side on a 158 recently

Knowing what little I do about the working of the system I would suggest they are ‘temporary’ as I would hope the system wouldn’t ‘lose’ PSRs after a reboot

  • Like 1
Link to post
Share on other sites

 

Herein lies the problem, the computer currenly can only assume you have 12 vehicles on, you can’t change that in the cab, so will give you a braking curve for 12 vehicles which as you can imagine is a fair distance, I take a 97 down there on a test train, 97+3 vehicles+97=5 vehicles

 

Bit silly - that would have to be changed on a more heavily used route where freight was mingling with passenger trains

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...