Cambrian ERTMS - Loss of TSR Data (20/10/2017)

[whole diary]

[SandTEngineer]

To a  signal engineer this looks extremely serious and I'm not supprised RAIB^▸ (Rail Accident Investigation Branch) has decided in the circumstances to undertake an investigation.

Welcome to the Digital Railway.....

Cambrian line
Investigation into the loss of speed restriction data to trains on the Cambrian line, 20 October 2017.

Published 21 February 2018
From:
Rail Accident Investigation Branch
S300 Cambrian TSRs^▸ (Temporary Speed Restriction)

During the morning of Friday 20 October 2017, a train driver travelling on the Cambrian coast line in North Wales reported that long standing temporary speed restrictions were not indicated on their in-cab display. As signalling staff at the control centre in Machynlleth investigated this report, they became aware that this failure applied to several trains under their control. The temporary speed restrictions were required on the approach to level crossings so that people crossing the line had sufficient warning of an approaching train.

The Cambrian lines were equipped in 2011 with a pilot installation of the European Rail Traffic Management System (ERTMS^▸ (European Rail Traffic Management System.)), a form of railway signalling. ERTMS removes the need for signals along the track by transmitting data directly to the train. This data is used to display movement authorities and other information such as temporary and permanent speed restrictions, on a screen in front of the driver.

Subsequent investigation found that the signalling system stopped transmitting temporary speed restriction data after a routine shutdown and restart at around 23:10 hrs the previous evening. The signallers had no indication of an abnormal condition and signalling control centre displays showed these restrictions as being applied correctly.

The RAIB has decided to undertake an independent investigation because to date, the signalling system supplier has not identified the cause of the failure. It is possible that finding the cause would have been assisted by downloading of suitable data from the signalling system before it was restarted during correction of the failure.

An additional procedure, since introduced at the control centre, is intended to identify and avoid any recurrence of the failure.

The RAIB investigation will consider:

the geographic extent of the failure and the effect it had on the safety of railway operations
why trains were permitted to operate without information about temporary speed restrictions
practices for the gathering of data needed for investigation before restarting computer based signalling systems after a potentially unsafe failure.

[stuving]

To a  signal engineer this looks extremely serious and I'm not supprised RAIB^▸ (Rail Accident Investigation Branch) has decided in the circumstances to undertake an investigation.

Welcome to the Digital Railway.....

Indeed. We all know software has design faults, and I've observed corrosion, but it's not meant to suffer from gales, vandalism, attacks by livestock, or whatever else removes lineside signs.

Now, who are these "suppliers"? RAIB should tell us when they report, but all I can find is that Systra claim to have done systems integration (on- and off-board). That might or might not involve writing "glue" software, but I'm sure all the boxes were made by other companies.

[grahame]

The RAIB^▸ (Rail Accident Investigation Branch) has decided to undertake an independent investigation because to date, the signalling system supplier has not identified the cause of the failure. It is possible that finding the cause would have been assisted by downloading of suitable data from the signalling system before it was restarted during correction of the failure.

I wonder when the RAIB were informed and whether "to date" means up to the date of the new report - 4 months.

It may sound like a not very dangerous circumstance - a long standing alert that all the drivers must have received multiple times in a day for months and all knew about - but it was a wrong side failure of the signalling system ...

[ellendune]

It may sound like a not very dangerous circumstance - a long standing alert that all the drivers must have received multiple times in a day for months and all knew about - but it was a wrong side failure of the signalling system ...

The driver noticed the longstanding one.  But what about the recent ones the driver did not know about? - If there were any? 

[SandTEngineer]

....from the careful wording by RAIB^▸ (Rail Accident Investigation Branch) (as always, not making any assumptions) I have a hunch that more than TSR^▸ (Temporary Speed Restriction) data may have been involved.  I hope it wasn't signalling data....

[Trowres]

Why would ERTMS^▸ (European Rail Traffic Management System.) have required a "routine shutdown and restart"?

[welshman]

Windows Update obviously.   

[GBM]

Why would ERTMS^▸ (European Rail Traffic Management System.) have required a "routine shutdown and restart"?

Also frequently used on First Kernow busses!

[Trowres]

RAIB^▸ (Rail Accident Investigation Branch) has now published an interim report:
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/749430/IR012018_181018_Cambrian_TSRs.pdf

Just after 23:00 hrs and near the end of passenger service on 19 October 2017,
a software reset occurred in the computer based signalling system controlling
the Cambrian lines and located at Machynlleth signalling control centre.  This
automatic reset, known as a rollover, was triggered when the equipment on-board
a train at Machynlleth station, automatically requested a movement authority
already allocated to another train.

Such software rollovers occur between 10 and 12 times each year, and the
signalling staff at the control centre followed their established processes for
returning to normal service.  It was necessary to stop movement authorities being
given to the three trains within the area controlled by the signalling system during
the rollover but normal working resumed around 23:19 hrs.  The three trains
continued to their respective destinations after a short delay. These were the last
trains of the day.

During a rollover, the signalling system at Machynlleth takes information from
a support computer system (the GEST server described at paragraph 25). 
This should include information about temporary speed restrictions.  However,
during the rollover on the 19 October 2017, the data relating to temporary speed
restrictions between Dovey Junction and Pwllheli failed to reload from the support
system to the signalling control system.  Staff working in the signalling control
centre were unaware of this when they subsequently permitted trains to begin
operating again.

The remainder of the report discusses at some length the ongoing attempts to discover why the GEST server repeatedly failed to provide details of the TSRs^▸ (Temporary Speed Restriction) to the signalling system.
What it doesn't discuss is the so-called "rollover". Why would a train's on-board equipment automatically request "a movement authority already allocated to another train" ? Is this part of the ERTMS^▸ (European Rail Traffic Management System.) spec? And why, if this causes a system reboot that halts trains under the control of the Machynlleth system for about 15 minutes, has it been tolerated "between 10 and 12 times each year"?

[eightf48544]

Agree with you it seems very odd that a train can request an already issued movement order.

Also your point that the RAIB^▸ (Rail Accident Investigation Branch) don't seem to have queried why this happens at around once a moth and requires a rollover which stops the trains for 15 minutes.  Do they just come to halt anywhere on the line, not necessarily a good idea given some of the gradients?

Can the driver override it (on instruction)?

[stuving]

Before asking those questions, I'd want to know more about how ERTMS^▸ (European Rail Traffic Management System.) (aka ETCS^▸ (European Train Control System)) works - such as what constitutes the same movement authority? If a train haasn't been issued with it, how can it request it, rather than a different one for the same track section? I wonder if this is because the MA is already pre-allocated as part of a route.

But of course ERTMS is complicated stuff - no, more complicated than that!

I can see that this is a cobbled-together experimental system only fit for a lightly-used line, so its shortcomings may have been acceptable. But I am puzzled that it (the GEST) was based on an existing system in use is Spain, yet Ansaldo have design data for it.

Remember, this is an interim report.

[grahame]

Report published yesterday ... summary from the BBC^» (British Broadcasting Corporation - home page) - not sure if this is the same incident, related but different, or disconnected and at about the same time

A passenger train approached a level crossing at nearly three times the speed limit because of computer problems, a report said.

The train was travelling at 80km/h (50mph) in Gwynedd, breaking the limit of 30km/h (19mph), which aims to give people time to cross safely.

No accidents were caused due to the speed between Barmouth and Llanaber.

But the chief inspector of railways said the industry had to learn "important lessons".

Simon French said nobody running the railway at the time had any idea anything was "amiss".

The Rail Accident Investigation Branch (RAIB^▸ (Rail Accident Investigation Branch)) report into the incident made five recommendations, including improving computer safety measures.

It found information about temporary speed limits failed to be transmitted to four trains using the line on the morning of 20 October 2017.

Fortunate there was no accident.   This serves to stress the importance of signalling data being complete and correct - and how the completeness and correctness of that data must be in a different league to the correctness (sometimes lack thereof) of data we see on passenger information systems.

[SandTEngineer]

Its the same incident, Grahame.  We will never know the full truth of what happened as it seems the data logger was reset as part of the incident, and all recorded data was lost.  Interesting that RAIB^▸ (Rail Accident Investigation Branch) had to call in the Norwegian equivalent to provide technical expertise.

Full report here: https://www.gov.uk/government/news/report-172019-loss-of-safety-critical-signalling-data-on-the-cambrian-coast-line?utm_source=9bee70f2-baf6-40de-80ac-7578bc4febdb&utm_medium=email&utm_campaign=govuk-notifications&utm_content=immediate

As I said in my original post that started this topic way back in February 2018, 'Welcome to the Digital Railway'

[Celestial]

One thing I find slightly surprising is that the temporary speed restriction had been in place for 3 years because of level crossing sighting. I wonder at what point a TSR^▸ (Temporary Speed Restriction) becomes a PSR^▸ (Permanent Speed Restriction), and is there any plan to address the issue.

On that note, it also struck me that there were several of these reducing the speed to 19mph. I would imagine the total effect of these is considerable in terms of increasing journey times, and thus making the overall journey less attractive, and probably less reliable. 

[SandTEngineer]

One thing I find slightly surprising is that the temporary speed restriction had been in place for 3 years because of level crossing sighting. I wonder at what point a TSR^▸ (Temporary Speed Restriction) becomes a PSR^▸ (Permanent Speed Restriction), and is there any plan to address the issue.

Thats a very good question.  Probably the problem here is that the Cambrian ERTMS^▸ (European Rail Traffic Management System.) system is quite bespoke and there is little, if any, technical support for getting software changes made.  Just my personal thoughts.