Crossrail/Elizabeth Line. From construction to operation - ongoing discussion

[whole diary]

[stuving]

The Evening Standard report (of 31/1/18) had a couple of other things in it too:

Engineers last night failed in their third attempt to switch on the power. The first test trains will now not start running on the new tracks until next month at the earliest.

Crossrail chiefs are set to switch to “Plan B” in May, when Heathrow Connect services are incorporated. The Connect trains will have to remain in service as signalling problems are likely to prevent Crossrail trains getting all the way to the airport.

Sir Terry said he was “very confident” the line, on which construction began in 2009, would be able to open by the end of December. He said: “We are very close on the funding envelope, and we’re certainly going to have to continue to work together to make sure we get this railway running this year.

“I don’t know what more we can do. When we’ve had problems we’ve scratched our head and tried to find the best people in the world. We have either got them, or, if we haven’t got them, we have gone and got them.

“It was always going to be a very difficult time but the team is still very confident they can get there.”

For some of that, and especially the signalling, Tuesday's TfL^» (Transport for London - about) board agenda is a bit more factual:

4.5 In May 2018, TfL Rail will commence operation to Heathrow, taking over from the current Heathrow Connect service and operating to Terminals 2/3 and Terminal 4.

4.6 For Heathrow services Bombardier is fitting the Class 345 trains with European Train Control System (ETCS^▸ (European Train Control System)) signalling equipment, which is required for operation on the Heathrow branch. Testing continues at Melton test track in Leicestershire, but progress is behind plan due to immaturity and instability of train software. The first train testing in the Heathrow tunnels took place on 12 January 2018 and, in addition, the wayside and train elements are due to be tested in a laboratory environment at Charleroi in Belgium.

4.7 The development and assurance of the train signalling and software remains the most significant single risk to timely commencement of Stage 2 services. In case Bombardier is unable to complete testing in time for May 2018, a contingency plan has been developed with TfL operating the existing Class 360 units to provide a two train an hour service to Heathrow, augmented with a two train per hour service using the Class 345 trains (without using ETCS) between Paddington and Hayes and Harlington. Should this contingency be required, it is expected that the planned service would be in place by autumn 2018.

4.8 At Christmas 2017, Network Rail made significant progress in updating the signalling system on the lines out of Paddington. It also successfully installed the Driver Only Operation CCTV^▸ (Closed Circuit Tele Vision) system at stations on the Heathrow route and will undertake the necessary work to extend the bay platform at Hayes and Harlington Station with a planned completion of April 2018.

4.9 At Heathrow, work took place at Christmas 2017 to facilitate Oyster^▸ (Smartcard system used by passengers on Transport for London services) and Contactless ticketing facilities as well as to roll out new ticket vending machines ready for May 2018. Work is also being delivered to update and amend signage and customer information.

4.10 The 14 hectare Old Oak Common depot will be the main base for the Elizabeth line train fleet. The first Class 345 was hauled into the depot on 7 November 2017 and the first stage of the depot’s signalling system was commissioned in December 2017. The depot was energised on 9 January 2018.

4.11 From December 2018, passenger services will operate in the central tunnel, between Paddington (new underground platforms) and Abbey Wood. From this date services will be referred to as the Elizabeth line.

4.12 The first test train was brought into the tunnel on 28 October 2017 and hauled to Abbey Wood. The next stage is energisation of the south east section of the infrastructure. This has been delayed by the failure of high voltage transformer equipment at Pudding Mill Lane. Once power is on, ‘dynamic testing’ (operating one, and subsequently more, trains in the tunnel to sequentially test the infrastructure) will get underway. Dynamic testing will start in the section between Abbey Wood and Canary Wharf and will then be extended to encompass the rest of the central section. Significant work is underway with installation of power, communications and signalling systems critical to extending dynamic testing.

4.13 The internal Master Operational Handover Schedule setting out the very detailed sequence of testing, commissioning handover and operational testing through to public opening is currently being updated.

4.14 For Stage 3, the Class 345 train requires further development to incorporate Communications Based Train Control (CBTC^▸ (Communications-based train control)) signalling, added to the two other systems that will already be fitted to the train. The software for this stage is also under test on the Melton test track. Given the critical importance of train signalling, regular reviews continue with Bombardier, the project team and, ultimately, with the Managing Director London Underground (LU) and the TfL Commissioner.

The emboldened bit is TfL's "plan B" for not having ETCS working. Apparently it's all going to be sorted by December, with all those nasty bugs guaranteed to be fixed. So, if you plan going to Heathrow that way, are you reassured?

[stuving]

This report from the BBC^» (British Broadcasting Corporation - home page) has a couple of new (or different) bits in it:

Crossrail: Concerns over project as completion draws nearer
Tom Edwards Transport correspondent, London
    2 February 2018
...
There are serious concerns about the power or "energisation" (turning on high voltage power) of the scheme after a design error caused equipment at Pudding Mill Lane sub-station to blow up on 11 November 2017, when two voltage transformers failed.

Crossrail has since confirmed the second attempt at "energisation" was successful in the early hours of Thursday morning.

Despite the progress, testing will now be postponed until the end of February.
...

Having first noted that Tom Edwards may not be the most reliable of sources, if there were two transformers that blew up, but not knowing which two, that still doesn't really tell us how major the issue is. But at least, if it was a case of fourth time lucky at turning on the power, that is something to start from. 

[broadgage]

The term "voltage transformer" may mean different things to different people, especially to reporters. However the accepted usage in electrical engineering is a transformer that is of very limited capacity and that is provided to power low voltage instruments, relays, pilot lamps and the like from a high voltage system.
The output can be any safer and easy to handle voltage, very often 110 volts.

As a simplified example, one might need to know the actual voltage on a nominal 25kv or 50kv system. Voltmeters for such voltages are not common, nor is it desirable to bring such high voltages into control rooms, so it would be usual to provide a voltage transformer that provides a 110 volt output in order to operate a standard type of voltmeter, that therefore indirectly indicates the actual voltage in the high voltage equipment.

Similar arguments apply to watt meters, KWH meters, pilot lamps, and relays, standard types operating at say 110 volts are employed via a voltage transformer.

Voltage transformers are not intended to supply significant power, but are sometimes used to supply limited emergency lighting in high voltage substations.

The failure of anything connected to a high voltage system is a matter of some concern, but voltage transformers are arguably at the lower end of the scale of degree of concern.
Causes of failure include defective manufacture, connection to the wrong voltage on the HV side, short circuit or gross overload on the LV side, or in the case of oil filled units omitting to fill them with oil !

However knowing the standard of reporting about technical matters, it could be almost anything that was "blown up"

[paul7575]

Perhaps they just wrongly edited the word “high” out, from an original statement that read “high voltage” transformers.  it could also of course be Crossrail’s PR^▸ (Public Relations) department who lost the detail...

Paul

[eightf48544]

Appears from another Forum that all 345s have been taken out of service.

https://www.railforums.co.uk/threads/class-345-progress.120221/page-59

Also lack of toilets has reach Reading one of its MPs^▸ (Member of Parliament) is taking up the issue.

Interesting that C2Cs version will have 3 toilets

[W5tRailfinder]

In May, the 4tph TfL^» (Transport for London - about) Rail service Paddington to Heathrow T4 was also taking over the role of the 4tph T2/3 to T4 shuttle. If TfL Rail is only going to run a 2tph service with class 360s from Pad to T4, are they going to run a 2tph shuttle or will Hex provide it?

[paul7575]

Also lack of toilets has reach Reading one of its MPs^▸ (Member of Parliament) is taking up the issue...

Has he really just realised? 
What can he do now?

Paul

[broadgage]

Also lack of toilets has reach Reading one of its MPs^▸ (Member of Parliament) is taking up the issue...

Has he really just realised? 
What can he do now?

Paul

I rather doubt that much will be done, but the downgrade to "no toilet, metro style" stock is regrettable in view of the length of some journeys.
I suppose that toilets MIGHT be retrofitted, but would not count on this.

The future of outer suburban stock is "metro style" i.e. designed primarily for standing and with limited seating.
The future of long distance stock is like old suburban multiple units, with number of seats maximised over all else.

[didcotdean]

It seems that TfL^» (Transport for London - about) treats the Elizabeth Line as a tube service. With capacity on steroids maybe, but a tube service nonetheless. I have seen statements that ultimately it will be corporately responsible through LUL^▸ (London Underground Ltd). In that regard, Reading is its Amersham.

[TonyK]

Also lack of toilets has reach Reading one of its MPs^▸ (Member of Parliament) is taking up the issue...

Has he really just realised? 
What can he do now?

Paul

Very little. He's got nothing to go on.

[stuving]

The first mention of the "big bang" was in the Elizabeth Line Operational Readiness and Integration report to TfL^» (Transport for London - about)'s Programmes and Investment Committee of 12 December 2017:

5.12 The next stage is energisation of the south east section of the infrastructure. This is now scheduled to commence in January 2018. This delay was caused by the failure of high voltage transformer equipment at Pudding Mill Lane during initial testing...

Now that was noted on this thread, with this comment:

On time and on budget then still.  But lots still to do...

Somehow we all missed the implications of that "failure".

The word "exploded" was first used by Terry Morgan at the TfL Board meeting of 30 January 2018, as reported in this thread via Rail Technology Magazine, Evening Standard, the BBC^» (British Broadcasting Corporation - home page), and in the relevant agenda item report. The minutes of this meeting are also now available (or were, but have now vanished!), but so brief as to be quite unilluminating.

So, having been puzzling over the extreme vagueness of the reported words defining just what exploded,  I looked out Terry Morgan's actual words. The board meeting video is still available on line, and it does contain a few more comments of his I've not yet seen elsewhere. See next post for a transcript.

[stuving]

Here, as a public service, is what I heard of the TfL^» (Transport for London - about) Board meeting as recorded. Note that while Terry Morgan obviously started with a prepared statement, most of this was unscripted, so has the fluffs and dodgy grammar you'd expect. I've left out false starts immediately reworded, and some repetition.

The Crossrail team are three people:

Mark Wild     Managing Director, London Underground
Sir Terry Morgan CBE     Chairman, Crossrail Limited
Howard Smith     Director Operations, Crossrail Limited

So TM^‡(resolve) was reporting the construction and testing, rather than the implications for TfL and service start-up. I've only covered Pudding Mill Lane here, but obviously there are other problems - mainly signalling, signalling, and signalling.

First, the report of progress to the Board (2:39):

I can’t think of anything more that we can do together to resolve the issue that exists, which is that we, as I reported last month*, we opened up the energisation of the east side of our railway, which was always going to be our platform for testing this train, in November. It’s a relatively standard piece of kit, it had to interface though between our power needs and Network Rail.

It got switched on, and exploded, and Mark is quite right is that first of all why did it happen, and to be frank we also had to go through how did it explode and are we satisfied that that couldn’t repeat itself in the way that it did. So that took a lot more work to be frank than we anticipated, but the current plan is we energise tonight. If that works, then we start testing towards the end of February.

*To the Programmes and Investment Committee meeting on 12 December 2017.

Later, he fielded a question about the explosion in particular (2:50):

I’m very unhappy with what happened; this is a factory-built unit, fully tested, there’s nothing more we could have done, it was almost like a plug-in type facility. It has some peculiarities because of the connections [-] have to make. And its mode of failure was a surprise to us, and when we actually eventually found out it had been designed incorrectly. And so this is a very complicated bit of kit built in a factory that we basically had to rebuild on site and it’s taken longer than I wanted it to.

Tonight we’ve all got fingers crossed it’s going to work.

We’ve got a similar facility to the west, at Old Oak Common, we think we’ve taken all the precautions to make sure it can’t be replicated at that site. It looks OK, but the reasons for all the emphasis are on the east is to reflect that fact that’s where we want to test our trains.

We have a free test track, basically, out to Abbey Wood, and we want to start getting some testing done on the trains.

He (with Mark Wild) was questioned further about the suppliers, and safety issues (2:52):

These are quite international companies that have let us down on the energisation issue and there is nowhere else we can go in terms of making sure that they are aware, of both their reputations and the consequences of what they are doing to us right now.
…
We’ve come to the conclusion it can’t fail like that again because it was designed incorrectly.
…
We will not switch it on until we are absolutely satisfied that we are confident it will work.

After this the subject was continued in private, since he was asked "what's our comeback on the factory", so names would have to be named. Comments follow in next post.

[stuving]

So, what to make of those TfL^» (Transport for London - about) reports? here's what I think.

At no point is it said or written that a transformer exploded. It was "high voltage transformer equipment", and that's not going to be a grid transformer since (as ET noted earlier) they belong to NG^▸ (Natural Gas) and are not in the ATFS. In any case, they will be protected against a truly massive fault current on their outputs, and are of a standard design.

You might think it would be an autotransformer, but there are none of those in this site. That may seem odd, but I can see why it might be. While at least one of the grid supplies is present, there is no need for an AT - the local supply is balanced already. Since the grid transformers are attached directly to the grid, not a branch off it, they are "always on" - loss of one of those would take out a big chunk of east London. At other ATFSs, like Didcot, there are ATs to be used when a line away from there is not fed from the Didcot end (perhaps related to the length of the 2x25kV feed).

When Sir Terry (TM^‡(resolve)) said "we’ve got a similar facility to the west, at Old Oak Common" he must have meant Kensal Green ATFS - and there's no AT there either. The intended standard operation is for the tunnel and the GEML^▸ (Great Eastern Main Line) to be fed from PML, as two separate (out of phase) sections, while the GWML^▸ (Great Western Main Line) is fed from KG. So one of KG's transformers is just a spare.

Note that Crossrail procured the substations at PML ATFS, Stepney Green SATS, Plumstead ATS^▸ (Automatic Train Supervision), and Westbourne Park ATS (the planned Custom House ATS seem so have vanished). Kensal Green was Network Rail's responsibility, and (like the other GW^▸ (Great Western) sites) uses switchgear in ground-mounted frameworks. TfL's outdoor sites use SMOS (switchgear on posts), so the design commonality is pretty low. AIUI^▸ (as I understand it), despite being a bunch of separate items, it would have been assembled and tested at the factory and then brought to site and reassembled, hence TM's comments about "a factory-built unit, fully tested".

So my conclusion (albeit still tentative) is that it was the ATFS - i.e. the switchgear - that blew up. As to the implications of "its mode of failure was a surprise to us", presumably it was a current path that by-passed the protection of the switchgear itself. Or maybe just that at least one team member was seen raiding the emergency clean underwear cabinet afterwards. 

[Electric train]

From my experience as a traction power distribution commissioning engineer, I have first energised quite a number of 25kV bits of switchgear and transformers also quite a bit on the DC^▸ (Direct Current) 33kV network there are a number of devices that can fail when first charged.

Voltage Transformers being one of them, these physically small devices have to manage the electrical stresses of the system voltage, its not unknown for them to pop.

I cannot help feeling there are other pressures going on within or being applied to the project team delivering the distribution equipment

[SandTEngineer]

An interesting article in 'London Reconections' giving an update on the current position (too long to quote here):
https://www.londonreconnections.com/2018/crossrail-cutting-fine/