Serious signalling failure - RAIB

[martin_wynne]

From RAIB today:

 

"Andrew Hall, Chief Inspector of Rail Accidents said:

 

“Some of the causes of the accident at Dalwhinnie bear an alarming similarity to those found in the multi-fatal accident at Clapham Junction in 1988, and the more recent collision at Waterloo in 2017 which caused huge disruption on routes into London. These accidents share a common theme, that an undetected wiring error resulted in the failure of the signalling system. At Dalwhinnie, this meant that the signalling system did not detect that some points were in an unsafe position, resulting in the derailment of a train. Thankfully no one was injured. However, the train could have been carrying passengers and travelling at a much higher speed, and the outcome very different."

 

Press release today: https://www.gov.uk/government/news/report-102022-wrong-side-signalling-failure-and-derailment-at-dalwhinnie-badenoch-and-strathspey

 

RAIB report today: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1105718/R102022_220926_Dalwhinnie.pdf

 

Martin.

[Stevebr]

This was quite a detailed report that doesn't show Network Rail in a good light. Are things slipping with too many managers working from home?

[The Stationmaster]

1 hour ago, Stevebr said:

This was quite a detailed report that doesn't show Network Rail in a good light. Are things slipping with too many managers working from home?

More like loss of collective memory I suspect and clearly a lack of proper testing procedure while I wonder what schematics they were using for the wiring (if they were using any?).  I have a feeling that a read of the full report is going to be not only interesting but distinctly worrying.

 

I do in any case wonder, in the light of Avanti's 'traincrew mis-management problem',  if various lessons which came out off Mr Justice Hidden's inquiry into the Clapham collision have also been lost from both collective memory and standards management. 

[Edwin_m]

5 hours ago, Stevebr said:

This was quite a detailed report that doesn't show Network Rail in a good light. Are things slipping with too many managers working from home?

It quoted the Waterloo collision and another near-miss at Greenhill a few years back, so I don't think it's reasonable to claim that the rate of wrong side failures has increased since Covid.  

Edited September 26, 2022 by Edwin_m

[Jeremy Cumberland]

I do like sections 139 to 150 about interruptions and re-orientating yourself to the original task. I find I have a lot of sympathy for the people involved, even though an important part of the testing was omitted.

 

No mention is made of what awareness the renewal team and the signalliers had of each other's work, which I find a little odd. The replacement of the point motor, or rather the testing of it, does not appear to have been planned arond the signaller's actual hours of work, and the impression I get is that the maintenance tester assumed the signaller would be on duty long before 10:00, giving him enough time to complete the testing and reconnect the track circuits before 10:30, when the line was due to be handed back. As it was, the maintenance tester's asking one of his team to drive him to the signalbox at about 10:00 to see if the signaller was there, after making several attempts to contact the signaller by phone, sounds like he was getting increasingly worried about how late the time was, as well as putting him in the wrong location for correspondance testing (as I understand).

 

In the event, there was no need to hurry; the train the line was to be handed back for had been cancelled (and in any case, the report implies that the line need not have been handed back much before 10:50, even if the train had run), and the next train did not pass till 11:55. If the maintenance tester knew this, there is nothing to say so in the report; perhaps it would have made no difference to the situation, if it would count against him to hand the line back late even though the delay was of no practical consequence.

 

In practice, the signaller's start time was of vital importance. Had he been there at 9:00, when the installation work was complete, then we can reasonably assume that correspondance testing would have taken place and detected the fault, but there is nothing in the report to indicate that the signaller's start time was considered at all. The report does not mention when the signaller was due on, nor when he actually started, nor what time the renewal team expected him to be on and how long, as a result, had been scheduled for the remainder of the testing and reconnection of the track circuits.

[Grovenor]

Correspondence testing could have been done to the KRs without a signalman, but maybe that would have needed a higher grade of tester? But the whole thing was sloppy, pre-installation checks missed it, the installers wirecount missed it, the tester just forgot to do the correspondence test and apparently even ticking the relevant boxes on the form did not remind him.  And it was indeed very lucky that it was a test train that found it after several months of the fault waiting to manifest itself.  Nothing in the report to give any reason that the A end chose that move to fail to normalise.

[96701]

I'm not surprised the RMT are resisting some of the "multiskilling" that is proposed.

Edited September 26, 2022 by 96701

[phil-b259]

10 hours ago, 96701 said:

I'm not surprised the RMT are resisting some of the "multiskilling" that is proposed.

Indeed

 

The more one person can undertake the grater the chances that the been counters will use that fact to cut down on staff present.

 

If NR get away with everything they want do do (at the behest of HM Treasury) then its only a matter of time before passengers die. It might take years but as sure as eggs are eggs it WILL happen....

Edited September 27, 2022 by phil-b259

[phil-b259]

14 hours ago, Grovenor said:

 Nothing in the report to give any reason that the A end chose that move to fail to normalise.

 

Such circuitry (the interlocking triggering the point motor to run for around 5 seconds and try and regain detection) is not new and 'worked as designed.

 

The network also has huge numbers of 'self normalising' points which will automatically swing back to the normal position after a train has passed over them in the reverse position (triggered by track circuit operation and subsequent clearance).

Edited September 27, 2022 by phil-b259

[phil-b259]

On 26/09/2022 at 11:46, Stevebr said:

This was quite a detailed report that doesn't show Network Rail in a good light. Are things slipping with too many managers working from home?

 

Arguably the wrong process was used here - Really SMTH is not suitable for tasks where the alteration of wiring within point machines is potentially required. Certainly I regard point machines as 'plug and play' devices (with any 'customisation' done in the external wiring and not within the machine itself.

 

As an SMTH tester myself I admit it would not occur to me to check the internal wiring of the machine unless specifically advised this was necessary - assuming that the machine would have been supplied from the factory 'ready to go' and that a wire count would only apply to the tail cable.

 

Also the difficulties of trying to complete paper (with a carbon copy beneath) in the rain should not be underestimated - and quite frankly if NR want them done 'on site' then they need to find a way of making them weatherproof.

 

However the omission of a point correspondence check is a surprising one. After Grayrigg and Potters Bar (plus the Greenhill incident the RAIB refer too) making sure both ends have to be in correspondence at all times for detection to be obtained (or be clipped up till its done) is something I would have thought all SMTH staff would have hard wired into their brain - and all all cases trains should be delayed if its not completed. Yes it can seem like a difficult decision with control screaming down the phone at you and the potential for management to start whinging for delays - but stuff them, as SMTH testers our role is to keep people SAFE - getting them there on time or meeting performance targets has no place interfering in SMTH activities, something the high ups in NR (and their paymasters in the DfT) simply don't get.

Edited September 27, 2022 by phil-b259

[Grovenor]

35 minutes ago, phil-b259 said:

Such circuitry (the interlocking triggering the point motor to run for around 5 seconds and try and regain detection) is not new and 'worked as designed.

 

The network also has huge numbers of 'self normalising' points which will automatically swing back to the normal position after a train has passed over them in the reverse position (triggered by track circuit operation and subsequent clearance.

Yes, of course and the report explains that. That was when the points tried to normalise, what is not covered, not even to say that the investigation could not determine the cause, was why the points failed to normalise when originally called to set the route. A number of reasons are possible and had the wiring been correct would have not been an issue.

Had the redrive on detection failure not derailed the train things could have been worse, the indication in the report is that the driver noticed he had been wrongly diverted but did not appear in any hurry to apply the brakes.

[phil-b259]

35 minutes ago, Grovenor said:

Yes, of course and the report explains that. That was when the points tried to normalise, what is not covered, not even to say that the investigation could not determine the cause, was why the points failed to normalise when originally called to set the route. A number of reasons are possible and had the wiring been correct would have not been an issue.

 

 

The points didn't normalise because the interlocking circuitry thought BOTH ends were already in the Normal position.

 

If a set of points are electrically detected normal (which is not always the same thing as them physically* lying normal) then there is no need to try and move them - and the external circuitry is designed to respond on that basis.

 

Also this is not a NX panel or an IECC workstation we are talking about - its a mechanical lever frame. As such the 'route' wasn't set by calling a 'route' it was done by the signaller pulling individual levers.

 

Provided the locking saw that the point lever was in the correct position in the frame and the electric lock saw that the points were detected in the normal position the signal levers would have been free to pull

 

When the train ran through one end and broke detection THAT was when the points were commanded to move to normal in response - and where detection circuit runs through both ends it will try and move BOTH ends.

 

Hence the end of points that were lying reverse (but detected normal) moved and derailed the rear of the train.

 

* Which is why frequent routine maintenance checks need to be done to ensure the electrical point detection circuit fails when points are not safe for train movements.

[Michael Hodgson]

2 hours ago, Grovenor said:

 

Had the redrive on detection failure not derailed the train things could have been worse, the indication in the report is that the driver noticed he had been wrongly diverted but did not appear in any hurry to apply the brakes.

 

I was surprised that the report doesn't comment on his reaction speed  - unless I've missed something.

 

I would have thought he must have felt a very obvious and unexpected lurch since the speed limit over the crossover as it was (when he was expecting a smooth ride and travelling straight) was well below the speed he reached. 

 

Perhaps it would be a bit unfair to criticise him, given that even if you're used to shift work, the human body is less alert and responsive at that time of day.

[Nick C]

2 hours ago, phil-b259 said:

 

The points didn't normalise because the interlocking circuitry thought BOTH ends were already in the Normal position.

 

If a set of points are electrically detected normal (which is not always the same thing as them physically* lying normal) then there is no need to try and move them - and the external circuitry is designed to respond on that basis.

 

Also this is not a NX panel or an IECC workstation we are talking about - its a mechanical lever frame. As such the 'route' wasn't set by calling a 'route' it was done by the signaller pulling individual levers.

 

Provided the locking saw that the point lever was in the correct position in the frame and the electric lock saw that the points were detected in the normal position the signal levers would have been free to pull

 

When the train ran through one end and broke detection THAT was when the points were commanded to move to normal in response - and where detection circuit runs through both ends it will try and move BOTH ends.

 

Hence the end of points that were lying reverse (but detected normal) moved and derailed the rear of the train.

 

* Which is why frequent routine maintenance checks need to be done to ensure the electrical point detection circuit fails when points are not safe for train movements.

Having just spent a few minutes tracing round the incorrect diagram, I think that with both ends reversed, it'd give the correct detection, so it would have commanded both ends to return to normal. It would only have been once 'B' returned to normal but 'A' was still reverse that it gave the erroneous detection. 

 

It's entirely possible, though never mentioned in the report, that the northbound move was the first time the crossover had been reversed since the replacement.

[Jeremy Cumberland]

1 hour ago, phil-b259 said:

The points didn't normalise because the interlocking circuitry thought BOTH ends were already in the Normal position.

Is this stated in the report? I must have missed it. In any case, both ends were originally in the reverse position, so I would be a little surprised if the detection fault should mean that 13B normalised but 13A did not, but I claim no specialist knowledge.

 

I read the report like Grovenor did, that the original failure of 13A to normalise was not considered in the report, but that the primary cause of the derailment was the detection fault (section 71). This, of course, is entirely correct; the failure of the points to motor should not have caused an unsafe condition, merely some inconvenience. The unsafe condition was the failure of the detection system to detect the unsafe condition.

[2251]

Para 79 of the report seems pretty clear that the reason why 13A was not lying normal was not investigated:

 

"There are a variety of reasons that would explain why the switch rails at the 13A point end did not move away from the reverse position, on this occasion, when the signaller commanded 13 points to the normal position (paragraph 76). The lack of movement resulted in the switch rails at the two point ends being in opposite positions (out-of-correspondence) and, therefore, 13 points being unsafe. Such point failures are not uncommon and are generally viewed as a performance or reliability concern. Operational safety is assured by the signalling system detecting the position of the switch rails and, if incorrect, preventing the signaller from being able to clear the relevant signal. As a result, RAIB’s investigation has focussed on the failure of the detection system." (emphasis added)

[phil-b259]

2 hours ago, 2251 said:

Para 79 of the report seems pretty clear that the reason why 13A was not lying normal was not investigated:

 

"There are a variety of reasons that would explain why the switch rails at the 13A point end did not move away from the reverse position, on this occasion, when the signaller commanded 13 points to the normal position (paragraph 76). The lack of movement resulted in the switch rails at the two point ends being in opposite positions (out-of-correspondence) and, therefore, 13 points being unsafe. Such point failures are not uncommon and are generally viewed as a performance or reliability concern. Operational safety is assured by the signalling system detecting the position of the switch rails and, if incorrect, preventing the signaller from being able to clear the relevant signal. As a result, RAIB’s investigation has focussed on the failure of the detection system." (emphasis added)

 

Very true.

 

Although the detection circuit may run through several end, each point end will have its own drive relays and fuses because of the currents involved, particularly if an obstruction occurs and the clutch is slipping is high (they usually get fitted with 20A fuses).

 

As such its entirely possible for a high resistance contact / blown fuse to prevent just one end moving but the other end moves correctly.

 

The cut out which isolates the point machine from the power supply if a point handle is inserted for manual operation is another week point that can prevent an individual end moving..

 

As you say there is nothing unsafe about this because only when ALL ends are in the correct position and (the detection circuit therefore complete throughout) should detection be obtained.

 

In this and other cases the issue has come about because the detection circuit has been compromised and no longer checks every single end properly.

 

 

[rab]

7 hours ago, phil-b259 said:

Also the difficulties of trying to complete paper (with a carbonopy beneath) in the rain should not be underestimated - and quite frankly if NR want them done 'on site' then they need to find a way of making them weatherproof.

I find it surprising that relatively large organisations are still using 'paper' forms.

 

I may have shared before but I was really surprised during a stay in hospital a few years back to see paper charts at the foot of each bed, and folders made up for each patient.  The only person using an electronic device was the person taking the orders for meals.

[martin_wynne]

25 minutes ago, rab said:

find it surprising that relatively large organisations are still using 'paper' forms.

 

Paper doesn't need a battery or an internet connection.

 

[PerthBox]

The report indicates that the Installer was using his iPad to access the SMTH and SMS, and that he had great difficulty in operating it due to the wet weather at the time of installation (which also drove the Tester to complete his test records in the van and/or relay room after the event…)

 

It surprised me too that the signaller wasn’t brought out early; the only time Highland main line boxes are closed during the week is overnight on a Saturday night into Sunday morning and it is quite usual to have at least one of the boxes on the line open early on a Sunday to give up a possession or line blockage. Closing overnight on a Saturday is a money saving wheeze which was introduced in the area about 15 years ago and the requests for engineering turns (additional nightshifts or early opening) have become so frequent in recent years that it is likely that continuous coverage will be making a come back at shortly.