Major Signalling Failure Swindon to Didcot 29/04/2017

[whole diary]

[SandTEngineer]

A major failure has occured between Swindon and Didcot resulting in the loss of all signalling.  Several trains trapped and some turning back at Swindon and Didcot.  Unfortunately the available diversionary route via the B&H^▸ (Berks and Hants - railway line from Reading to Taunton via Westbury) is closed due to engineering work......

Must be quite serious as NR^» (Network Rail - home page) are quoting that specialist technicians won't be on site until 1830....

[Zoe]

A major failure has occured between Swindon and Didcot resulting in the loss of all signalling.  Several trains trapped and some turning back at Swindon and Didcot.  Unfortunately the available diversionary route via the B&H^▸ (Berks and Hants - railway line from Reading to Taunton via Westbury) is closed due to engineering work......

Must be quite serious as NR^» (Network Rail - home page) are quoting that specialist technicians won't be on site until 1830....

Also no replacement road transport available, current advice on the GWR^▸ (Great Western Railway) website is not to travel.  I can't remember the last time this happened for reasons other than weather but I expect that at this time of year a lot of coaches will be in use on a Saturday and so not available at short notice.

[Rob on the hill]

Currently 2 trains stranded between Didcot and Swindon, and 2 between Chippenham and Swindon.
http://www.opentraintimes.com/maps/signalling/swindon

[John R]

Having seen some trains which were caught in the area have reversed to Didcot and have arrived back there three hours after they left, I've a bit of a dumb question.  Given the route in question is a relatively simple double track section (apart from a couple of loops). why could not the drivers be instructed to drive on sight until they get through the affected area?  It's a nice straight line too, so 30mph should be comfortably possible. I'm not suggesting that trains should enter the affected area - just that those caught in-between Swindon and Didcot could be brought out somewhat quicker than appears to be the case.   

[ChrisB]

Especially as they were allowed to reverse to DID^» (Didcot Parkway - next trains) under the same arrangements?!

[Electric train]

Having seen some trains which were caught in the area have reversed to Didcot and have arrived back there three hours after they left, I've a bit of a dumb question.  Given the route in question is a relatively simple double track section (apart from a couple of loops). why could not the drivers be instructed to drive on sight until they get through the affected area?  It's a nice straight line too, so 30mph should be comfortably possible. I'm not suggesting that trains should enter the affected area - just that those caught in-between Swindon and Didcot could be brought out somewhat quicker than appears to be the case.   

Especially as they were allowed to reverse to DID^» (Didcot Parkway - next trains) under the same arrangements?!

At Swindon several sets of points would need to be set to allow the trains into a platform, hand cranking then clipping and scotching is a high risk activity compared to reverse working in the wrong direction.

There would be a senior NR^» (Network Rail - home page) manager and a senior TOC^▸ (Train Operating Company) manager on a conference call with the senior engineer on duty / call the duty ops managers where decisions would be made and plans put in place with constant reviews, in this incident its likely NR would be the lead (Gold Command).

It may there is a lack of common sense to an outsider, however there are set protocols, Rule Book limitations that will steer actions.   

[SandTEngineer]

It can take a considerable amount of time to set up special working and it requires quite a few number of ground staff to implement.  Its often easier to reverse trains back to the unaffected area but when you are held in the middle of a completely failed section that can still be very difficult.

Noted that NR^» (Network Rail - home page) are now stating that it won't be fixed until the end of the day.  They didn't say which day though.....

[didcotdean]

Now fixed apparently although it will still be a long grim night for some.

[JayMac]

Network Rail incident timeline (in reverse):

   
Momentary red outs between Uffington and Swindon.

Telecoms Tech support Report:

Fault has been passed back to Swindon S&T^▸ (Signalling and Telegraph) with full update.

The bearer circuit to Bourton for the SSIs^▸ (Solid State Interlocking) has tested clear.

The fault appears to be beyond this, where the SSI route is split to the LDTs^▸ (Long Distance Terminals) over signalling equipment.(Signalling local end).

Swindon techs en-route to Bourton RR to reset the LDT's x 4, ETA 1845 Hrs

Once the LDT's have been reset BOX TO will be able to assess the signalling system
 
Incident Controller

**********

All points clipped and scotched for TBW^▸ (Temporary Block Working) Swindon to Challow on both lines.

Swindon MOM^▸ (Mobile Operations Manager) will issue tickets at SW1190 signal platform 1 Swindon, p/way will collect tickets at SB1010 for up trains
Oxford MOM will issue tickets SB1011 signal Challow, p/way will collect tickets at SW1205 signal platform 3 Swindon Station

Incident Controller

************

16:52 Box To reports that he has directed S&T to LOC MLN066U096 at Uffington Yard where more Long Distance Terminals (LDTs) are dropping out. This may be a symptom of the fault at Bourton or it could be an additional problem. Meanwhile the LDTs at Bourton remain powered down. Box TO still liaising with Stoke TEC.


-------------------------

16:46 Bristol Parkway on site at Swindon has clipped and scotched 8626A/B in the normal position he has now been joined by 2 Swindon P Way members of staff to assist with clipping and scotching points.

16:41 TVSC^» (Thames Valley Signalling Centre - about) box TO advises the temporary repair is not holding up and has again failed.

-----------------------------------


16:15 TFC Doncaster advises the ETA for their engineer to site is 18:30.

------------------------
15:49 Stoke TEC remote analysis has identified a failed communications card (MAXM01) at site ID 0024 (Bourton) that is intermittently dropping out. Stoke TEC will instruct telecoms technicians to attend. ETA to be confirmed.

-----------------------------------

15:36 Box TO reports that S&T investigated the LOC at Bourton, where the four data links originate, where they witnessed a synchronized pattern of failure and recovery. In addition there are multiple LED failure indications suggesting a possible telecoms FTN fault. Swindon S&T are en route to the neighboring LOC to examine the indications there. Box TO still awaiting Stoke TEC analysis.

--------------------------

Swindon MOM and Oxford MOM are clipping up Uffington and Bourton area.

15:35 P Way staff avaialble for the Swindon area enroute no ETA .

1 x Didcot
2 X Swindon.

---------------------------
15:31 train service has been suspended


15:25 TVSC SSM advises two trapped trains 1B40 and 1C85 will be brought through to Swindon by clearing signals on approach.

---------------------------------

14:48 TVSC SSM advises the red outs are occurring a regular basis every 4 or 5 minutes, further COA^▸ (Change Of Aspect)'s reported by 1L62 SW1112 green to red CAT B SPAD^▸ (Signal Passed At Danger), 1C16 SW1129 green to red.

14:44 TVSC Box TO advises that the affected Signal circuits are UFF.UP.01 A&B and UFF.DN.02 A&B the FTN node is the TVSC-Bourton route.
Stoke TEC FTN is checking for telecoms issues in the area but cannot do an in depth search until the Box TO comes back with the telecoms FTN circuit number which he is looking up now.


14:43 Swindon MOM requested to go to Bourton and Oxford MOM to Uffington to clip all points.
---------------------------

14:30 TVSC SSM advises the first red out occurred at 14:13 affecting down main line only, the second red out occurred at 14:20 affecting the up and down main lines. Also there has been a third red out at 14:25 affecting the down main line only.

There has been one COA reported by the driver working 1A19 (13:30 Bristol Temple Meads to Paddington) on SW1112 Yellow -red - green, driver is okay to continue.


14:30 TVSC SSM advises the first red out occurred at 14:13 affecting down main line only, the second red out occurred at 14:20 affecting the up and down main lines. Also there has been a third red out at 14:25 affecting the down main line only.

There has been one COA reported by the driver working 1A19 (13:30 Bristol Temple Meads to Paddington) on SW1112 Yellow -red - green, driver is okay to continue.

-----------------------------------

14:18 TVSC Swindon signaller advises he has had a momentary red out affecting the down main line between Uffington and Swindon Bourton area. There has been no COA reported and everything has fully recovered with the TVSC box TO investigating.

14:19 TVSC SSM advises there appears to be a second momentary red out affecting all lines between Swindon Bourton and Uffington, waiting further details for the SSM or TVSC box TO.
 
TV IC^▸ (Inter City)

[Electric train]

15:49 Stoke TEC remote analysis has identified a failed communications card (MAXM01) at site ID 0024 (Bourton) that is intermittently dropping out. Stoke TEC will instruct telecoms technicians to attend. ETA to be confirmed.
 

This always fascinates me in the Signalling system there does not seem to be a independent dual telecoms routing between the controlling Box and the remote interlocking; in electrification the SCADA^▸ (System Control And Data Acquisition,) telecoms has two coms lines diversely routed from the ECR to the remote traction power location; its very rare that SCADA ever looses both coms lines 

[ChrisB]

So a) no trains reported as having revered to DID^» (Didcot Parkway - next trains); indeed, states 'brought through to Swindon;

And b) Doncaster? Really? NR^» (Network Rail - home page) had no one closer?.....strewth

[grahame]

And b) Doncaster? Really? NR^» (Network Rail - home page) had no one closer?.....strewth

I read that and thought the same.  However, reading back carefully it may very well be a reference to a control centre / specialist remote support resources in Doncaster, with the engineer(s) from there being on call from their bases / homes elsewhere.

Estimate was 2.5 hours to be on site ... would be hard pushed to do that from Doncaster to what's probably a fairly obscure spot in the countryside somewhere on the Oxfordshire / Wiltshire border.

[IndustryInsider]

It doesn't mention whether Emergency Special Working was introduced (before Temporary Block Working was eventually set up) which is authorised between Ladbroke Grove and Wootton Bassett as part of a trial and is designed to get things moving quicker in such a situation.

https://www.rssb.co.uk/rgs/deviations/15-088-DEV.pdf

[stuving]

So a) no trains reported as having revered to DID^» (Didcot Parkway - next trains); indeed, states 'brought through to Swindon;

And b) Doncaster? Really? NR^» (Network Rail - home page) had no one closer?.....strewth

That'll be NRT^▸ (Network Rail Telecoms), not NR. Network Rail Telecoms runs the fibre trunk network (FTN - though apparently that stands for Fixed Telecom Network). They have two network management centres, Crewe and Doncaster.

I had been wondering for some time whether the "recontrol" of signalling areas to the ROCs^▸ (Rail Operating Centre - a centralised location for railway signalling and train control operations for a specific route or region) was by now using IP for its networking, and if so down to what level. The point being that you then could - and would - provide at least two links to everywhere, so no single failure would cut off a node. There's a technical article here from RailEngineer that says the IP-based successor to the FTN (FTNx) exists on the core network, but suggesting that it does not go much further out into the countryside. Germany is cited as having gone much further, with IP to the signal head, which I presume is the ambition here too.

That reminds me of a question I had about the resignalling of the GWML^▸ (Great Western Main Line), resulting in it all being connected to ROCs, and whether that will or will not reduce the excessive number of failures seen recently. Will all the signals across the four-track sections rely on the same grey boxes and links to their ROCs, one one side of the line (at any point)? Following which you could add: will here be at least two IP routes to trackside, at grey box if not lightbulb level?

[SandTEngineer]

Now then, this is a very complex subject but here goes.......

In simple terms you have two diverse end fed baseband communication links (maximum length of 8km) feeding all Trackside Functional Modules (TFM^▸ (Trackside Functional Modules)) that control the actual signal lamps and through relays, the points.  These are housed in the trackside grey boxes (known as Location Cases).  In this area Train Detection is still by track circuits (but due to be replaced by axle counters in the future) and these also feed into the TFM.

So any loss of both baseband links means total loss of signalling.  Beyond the baseband links there are the Fixed Telecommunication Network (FTN) nodes that provide the link between the trackside and the signalling control centre, Thames Valley Signalling Centre (TVSC^» (Thames Valley Signalling Centre - about)) in this case.  The interlocking is located there and this is triplicated for redundancy in a 2 out of 3 configuration (i.e. any 2 out of 3 modules need to be working and in agreement before any output is processed, so very robust).

The FTN network is very diverse so I am very supprised about the nature of this failure.  I'm sure we will hear more about it.

Hope that makes sense and is not too technical (and apologies to CfN who might have to add a few things to the acronyms and abbreviations page )