Solar storm railway operation disruption

[KeithMacdonald]

2 hours ago, jjb1970 said:

Common failure modes

 

Common Failure Modes (CFM)

a.k.a. Continual F***ing Muppetry?

 

I expect every industry has its own "Top Ten" of CFM's. I'm not too proud to admit my own Day Job organisation has perfected CFM to a high degree. Here's my own Top Ten off the top of my head (which is now a peculiar shape from the sheer number of times head has hit desk)

1. Alerts and warning messages by emails to single individuals (not role-based distribution groups) - even after individuals are long-term sick or have retired/resigned/been removed from the business
2. Role-based distribution groups that cannot be updated - because the Human Remains department now regards info about individuals (the long-term sick or have retired/resigned/been removed from the business) as classified personal information.
3. Alerts and warning messages by emails - that cannot reach intended recipients because their mailboxes are full.
4. Alerts and warning messages by emails to role-based distribution groups - but nobody reads them - "somebody else" will do something
5. Processes that are completely undocumented
6. Processes that are documented - but the docs are in "safe places" that can't be accessed (because "security", or 12 different places for documentation depending on department/project/team/location etc etc that are private to them)
7. Requirements, specifications, designs, etc that were written and put in personally-named folders, not project-based, then when said person leaves their folder is archived or deleted and knowledge is lost.
8. Servers that have had no service packs, patches, updates etc for years, and/or are not monitored then run-out of disc space.
9. Network infrastructure that is so slow it take more than 24 hours to replicate 24 hours of data
10. Remote site network infrastructure/comms failures, but Infrastructure Team is unaware that remote sites are off-line, because remote sites have to fill-in online forms, but cannot do so because...

Any offers on other common CFMs?

[KeithMacdonald]

On 15/12/2023 at 10:26, KingEdwardII said:

I can't decide whether it is some weirdly misplaced idea relating to impartiality (etc) or whether it's because the folk at the BBC simply don't understand how the world wide web actually works. Some of the decisions they make about (not) providing links between some of their own web pages tends to make me think the latter...

 

On 15/12/2023 at 10:19, jjb1970 said:

It's something I find annoying and really rather questionable as it begs the question why they don't make it easy and help readers do their own reading and form their own conclusions. 

 

BBC News (and many newspapers) used to have old-school proper investigative journalists that were given the time and resources to acquire and/or access expertise in a given subject. Many of us know how hollowed-out local and regional newspapers have become, but the same "budget and resources" issue affect the BBC as well.  It's no secret that a lot of the "news" we are exposed to on a daily basis now comes in the form of press releases from lobby/pressure groups that have a particular axe to grind, or slant the news in a particular direction. That often has click-bait headlines, and if it is attributed (with links) it's to the lobby/pressure group that has cherry-picked data from an original source (often not attributed or attributed but not with links).

 

As the BBC is a major player in the Trusted News Initiative (with consensus and conformity being major concerns), helping readers to do their own reading and form their own conclusions has become less of a priority than guiding readers to accept consensus positions and not ask awkward questions.

[The Stationmaster]

3 hours ago, Michael Hodgson said:

 

All very well in normal operation, but I would hope that they still have charts and somebody on board who knows what a sextant is for?

Many operators have gone chartless, aka 'paper free navigation.   The problem with paper charts is that they are only any good of they are kept up to date.  Making sure that is done  and then doing it puts updating railway operating publications well into deep shade - it can be a very fiddly process relying on 100% accuracy (yes  - I  have watched it being done and was shown how it is done).

 

In theory (and pretty much the case in practice so I understand) the amendment and updating system for electronic charts is good - certainly so for Admiralty Charts.  As far as the latter is concerned when i was at an evening 'do' on. Trinity House vessel (moored alongside HMS Belfast) a few months back I got talking to a chap about charts and it turned out that he was an IT person working on the development of Admiralty electronic charts and his view was that their updating  is about as quick as you can get - and it is automatic once you've downloaded a chart.  But it is of course then resident in a vessel's computer system so if that goes pop you don't have a chart (apart from any local backed up stuff.

 

And the course plotted on teh electronic chart can then drive the vessel's 'auto-pilot' if a ship is so equipped.   Even quite small craft can be found using electronic nav systems - we went eagle spotting on a Norwegian boat of no more than about 40ft and the electrononic navigation fit on that was incredible - and no paper charts in sight.

 

I like paper charts and do wonder about reliance on electronics but at sea they are definitely becoming things of the past.

[johnofwessex]

24 minutes ago, The Stationmaster said:

.........................

 

I like paper charts and do wonder about reliance on electronics but at sea they are definitely becoming things of the past.

 

Indeed, what happens when it goes wrong

 

Now in the Good Old Days even the Bristol Queen didnt carry radar until her last few years in service, ditto the  1955 Manxman with Decca = largley a result of the demands of film makers meaning ending the day with 'I wonder where we have ended up?'

[Jeremy Cumberland]

Electronic charts and SatNav are two different things. I can't imagine an electronic chart plotter not having SatNav, but they can be used with traditional navigation as well; you just use mouse clicks rather than drawing instruments, a bit like creating engineering drawings with CAD rather than using a pencil.

[KeithMacdonald]

21 minutes ago, The Stationmaster said:

I like paper charts and do wonder about reliance on electronics but at sea they are definitely becoming things of the past.

 

I too like paper charts, and similarly I'm concerned that an over-reliance on electronics can mean essential skills and knowledge can be lost.  As it's getting on for 30 years since I got my own Certificates of Competence, I was curious-enough to have a look at the current requirements for Certificates of Competence.  e.g. Yachtmaster Offshore.  A couple of items caught my eye.

 

Quote

The boat must be equipped with a full up to date set of charts and navigational publications along with working instruments and either plotter or GPS.

and

Quote

 

Additionally if not on the boat, you will need to bring to the exam:

1. laminated or waterproof charts
2. GPS set (may be hand held)
3. tide tables
4. pilotage information for the local area, eg pilot books, port information etc
5. plotting instruments.

 

Ref: https://www.rya.org.uk/training/certificates-of-competence/yachtmaster-offshore

 

So candidates still need to demonstrate a level of competence with the olde-worlde analogue and paper methods. Even if they abandon them immediately after the course in favour of smart phones and tablets and other assorted digital devices, there's a good chance that enough gets ingrained to give them (a) an instinct for when the electronics has gone unreliable and/or (b) a better chance of coping when it's total tits-up e.g. the kinds of system failures than can and do happen, like battery/power failure and dead electronics.

 

Seeing tide tables on the list reminds of another skill, working out the tidal vectors. So when the boat goes in a different direction to the one you were steering, you know one of the reasons why, instead of feeling confused and/or useless.

 

With our own children, on coastal passages, M'Lady and I used to make it a regular part of keeping our children entertained and busy. At about 10 years old, they had the VIP job of taking bearings with a compass off landmarks and keeping track of where we were on a spare paper map. From that, we had them working-out what course corrections we needed to make to get round the next headland.    A few years later, they both breezed through their DofH Gold Expedition section. That's the kind of hands-on skills that also encourages an active "can-do" attitude instead of a passive "WTF do we do?" or "phone for help" attitude.

[newbryford]

On 14/12/2023 at 12:48, adb968008 said:

 

 

if a solar flare happens, do what the wireless operators of 1859 did… unplug everything, and carry on, using the free electric provided…

your model locomotives should start running themselves..

 

 

Only if they have stay alives fitted.

[david.hill64]

7 hours ago, MarkC said:

This was the big concern when the centralized signalling centers were first proposed - what happens if, for example, somebody accidentally puts a *JCB scoop through a cable run? I don't recall any proper answer to this ever being offered...

 

* other excavating machinery is available...

 

Mark

Not sure about the big railway as my recent experience is with metros and Taiwan HSR. These have diverse redundant comms routes with the option for local control if, for example, the route is completely severed in the middle.

[adb968008]

55 minutes ago, newbryford said:

 

Only if they have stay alives fitted.

Every model loco should have the Beegees fitted :-)

 

oh you mean a capacitor.. that may go pop, just like the beegees on a saturday night.

Edited December 17, 2023 by adb968008

[Oldddudders]

1 hour ago, david.hill64 said:

Not sure about the big railway as my recent experience is with metros and Taiwan HSR. These have diverse redundant comms routes with the option for local control if, for example, the route is completely severed in the middle.

They should, but didn't always. In August 1983, something caused a cable fire in the South Bermondsey area, which cut off London Bridge box from all signalling south of North Kent East Junction, roughly to Woolwich, Eltham, Sidcup, Grove Park and Catford Bridge. It stayed that way for a couple of days, crippling operations on the intensive services into London Bridge (E), Waterloo (E), Charing Cross and Cannon Street. Black hole signalling and time-interval became the only mode of moving trains at all. Scary stuff. Never did the cable-jointers earn their corn so well afterwards!

 

Hopefully in a more digital age resilience is indeed being designed-in. 

[jjb1970]

One of the odd side effects of the war on terror period was that some companies looked at things like resilience more seriously. The electricity sector identified some howlers in terms of potential vulnerabilities. I was never that worried about the terror angle but upgrade works addressed a few issues that could have been embarrassing in the backhoe loader bucket type scenario or severe storms.

[Reorte]

11 hours ago, Oldddudders said:

Hopefully in a more digital age resilience is indeed being designed-in. 

Could go either way. More concern for the things going wrong seems to be in fashion, but another fashion is that any form of redundancy is seen as wasteful and inefficient.

 

Personally speaking, whilst I understand why it's all moving in the direction it is (and it's not as if I don't understand the technology either, at least as a generalisation and basic concepts level), I just find the end result a more depressing world. Give me a more local, hands-on, human world any day.

[KeithMacdonald]

14 hours ago, jjb1970 said:

One of the odd side effects of the war on terror period was that some companies looked at things like resilience more seriously.

 

5 hours ago, Reorte said:

More concern for the things going wrong seems to be in fashion, but another fashion is that any form of redundancy is seen as wasteful and inefficient.

 

Maybe it depends a lot on the specific industry and context? I'm more than happy for transport systems (planes, trains and boats especially) to have as much resilience and backup systems as they want.

 

Sadly, in my Day Job, it's still the penny-pinching accountants that have to sign-off every major project, and "things that might never happen" are often chopped out of requirements and design specs. Like multiple-paths comms links, failover servers, contingency disc storage, system performance monitoring, etc etc. Until "the thing that might never happen" does sooner or later happen, and the executive team goes into full headless chicken blame-game mode, desperately thrashing around for a scapegoat. Then it's "rob Peter to pay Paul" as existing budgets get mugged to pay for the repairs and system changes, which invariably cost more than it would have done to have included it in the first place.

[frobisher]

On 16/12/2023 at 11:11, billbedford said:

Has anyone else noticed how odd the numbers quoted in that article are?

 

This time -- X2.8

2017 -- X8.2

2003 -- X28

 

No info about the scale either. 

 

A quick google finds this which gives some explanation; The X28 is because the maximum the sensors could record was X28 and was probably X45, the recurrences of 8 and 2 are just coincidence.

 

What are X flares? Can they harm us? (earthsky.org)

 

solar-center.stanford.edu/SID/activities/flare.html Gives the actual scale...

 

 

 

[Reorte]

1 hour ago, KeithMacdonald said:

 

 

Maybe it depends a lot on the specific industry and context? I'm more than happy for transport systems (planes, trains and boats especially) to have as much resilience and backup systems as they want.

 

Sadly, in my Day Job, it's still the penny-pinching accountants that have to sign-off every major project, and "things that might never happen" are often chopped out of requirements and design specs. Like multiple-paths comms links, failover servers, contingency disc storage, system performance monitoring, etc etc. Until "the thing that might never happen" does sooner or later happen, and the executive team goes into full headless chicken blame-game mode, desperately thrashing around for a scapegoat. Then it's "rob Peter to pay Paul" as existing budgets get mugged to pay for the repairs and system changes, which invariably cost more than it would have done to have included it in the first place.

The problem with that thing that'll never happen is that there are so many of them, each individually extremely unlikely but there are enough of them that every now and then one will crop up (well, assuming you've not got completely useless types in control of everything who seem to actively invite problems).

[Mallard60022]

Talking of Comms and losing control...did anyone see this series on that TV thing, of VIGIL.

Comm's/Control interference, but nothing to do with Solar flares, juts plain old naughty Humans and really very scary. To me the 'human intervention' as dramatised in VIGIL, is far more of a concern, along with the development of AI and the way the World seems to be going just now.

I'm not a Conspiracy clot, or living in terror of life, but I sure am already a 'victim' of Technology and it's place in the control of so many things.

I know this is rather OT, but to me the natural happenings of Nature are inevitable and often unpredictable, but often 'expected', but sometimes ignored (e.g. a pandemic).

Human behaviour is thus, but the associated manipulation is so very unpredictable as well. 

I'll shut up now.

Phil

[adb968008]

1 hour ago, Mallard60022 said:

Talking of Comms and losing control...did anyone see this series on that TV thing, of VIGIL.

Comm's/Control interference, but nothing to do with Solar flares, juts plain old naughty Humans and really very scary. To me the 'human intervention' as dramatised in VIGIL, is far more of a concern, along with the development of AI and the way the World seems to be going just now.

I'm not a Conspiracy clot, or living in terror of life, but I sure am already a 'victim' of Technology and it's place in the control of so many things.

I know this is rather OT, but to me the natural happenings of Nature are inevitable and often unpredictable, but often 'expected', but sometimes ignored (e.g. a pandemic).

Human behaviour is thus, but the associated manipulation is so very unpredictable as well. 

I'll shut up now.

Phil

Ive worked in IT 25 years, in Data Centres globally.

i know enough that if something happened to about a select dozen such facilities in this country ( in perspective theres hundreds in the UK)  it would be enough to cause chaos here.

 

I know one major uk company nearly stopped its business for 3 days costing millions and affecting thousands of customers, just because a contractor shut a door to an IT cabinet… it was wedged open because it had bad airflow/too much heat… closing it tripped the fans, which started a chain reaction of tripping breakers upto the main pdu.. and because of cut backs in IT / over reliance on contractors, took down the facility, which wasnt securely redundant because of the same cut backs… and with 45 minutes their business was crippled, costing them far more than had they done the job properly.

 

That said, my whole career has been based on identifying and convincing people of these issues.. they only listen when recession approaches, or after the major failure… take a horse to water etc. That said the “told you so” moment usually means we can now safely jack the price up.

 

I know the weak points of hundreds of global IT facilities around the world.. i predict one major “oh ….t” moment will come in South London.. they arent listening either, they have a major piece of IT without labelled cables, or a door on the cabinet..with a traffic cone for protection, the quote is ready…

 

Fortunately when these places go pop its usually isolated incidents, so worst case is an outage for a day or two affecting one company, and if they are unlucky it reaches the news.

 

Its not solar storms that are the real threat to our infrastructure, its CFOs… they only see cost, not risk.

 

 

 

 

 

 

 

 

 

 

 

 

 

Edited December 18, 2023 by adb968008

[KeithMacdonald]

2 hours ago, Reorte said:

well, assuming you've not got completely useless types in control of everything who seem to actively invite problems

 

[mezzoman253]

BIG SUNSPOT ALERT: Sunspot AR3529 is still growing, quadrupling in size since Tuesday. This 48-hour movie from NASA's Solar Dynamics Observatory shows its rapid development: The animation is here https://www.spaceweather.com/

 

With two primary dark cores twice the size of Earth, AR3529 is an easy target for solar filtered telescopes. You can even see it through eclipse glasses--no magnification required.

 

Of greater interest is the sunspot's 'delta-class' magnetic field. Within the sunspot + and - magnetic polarities are pressing together. Magnetic re-connection could produce an X-class solar flare. Any eruptions today will be Earth-directed, as the sunspot is almost directly facing Earth.

 

 

 

 

 

 

Edited December 21, 2023 by mezzoman253
Unwanted copy and paste.

[KeithMacdonald]

On 18/12/2023 at 19:53, Mallard60022 said:

did anyone see this series on that TV thing, of VIGIL. Comm's/Control interference, but nothing to do with Solar flares, juts plain old naughty Humans and really very scary.

 

Yes, as a piece of fiction it was quite entertaining. As with any good piece of dramatic fiction, the plot hinged on various naughty folks doing malevolent things. Whereas, in the real world, muppetry and mistakes are far more prevalent than malevolence.

 

[ess1uk]

On 18/12/2023 at 19:53, Mallard60022 said:

Talking of Comms and losing control...did anyone see this series on that TV thing, of VIGIL.

 

Watched 2 episodes 

couldn’t take it seriously