Great Western Coffee Shop

Nice touch by LNER following yesterday's problems (the fallout from which is ongoing)

All caused by a single rogue flight plan apparently!

Depends what you means by "caused". I'd expect a system like that to be 100% protected against bad input data, not to declare a serious internal error nor for it to take hours for its minders to work out that's what happened. That's what caused the damage.

Sounds like an error trapping fault in the software. 

It seems astonishing, especially in a system which should have been tested virtually out o existence - however, I can look back at a life working with software and tell some stories.

I was responsible for a batch / record keeping system where we updated our main tape "file" every few days - a pile of punched card updates grew n our office, and once it reached a line on the wall behind the table, we submitted a job.  Occasionally we ran it "short" for an urgent update, or delayed it if the computers were too busy with customer production work - and the update pile grew even to several times the normal height.   Worked perfectly, no problems.  For years.

Then one day the data "corrupted" - the output printout showed that the tape file had not been updated  It had been initially tested, running for years, trusted to work every time ...  what the *** had happened?

It turned out that if the number of input records was a multiple of 427, then the last 427 records were ignored.  The line on the wall was at around 350 and usually there wasn't even one "set" of 427.  At very busy times, two updates tended to be combined and it ran with around 700 updates. Perfect.   Then one day ...

The bug was a ".GE." rather that a ".GT." (or was it the other way?) deep in the code ... records were sorted in batched prior to the update. There was a buffer of 50000 bytes and a record size of 117 bytes and if it got full but didn't trigger a new buffer we had a problem. The number "427" was no-where in the code.

I don't know what happened at NATS - but I have an element of sympathy.   Of course, it has got more complex these days and it sounds like the system and feedback loops (lacking in our system 50 years ago) rang alarm bells ....

If the safety of my flight depended on a pile of punched cards against a wall I think I’d walk.

However joking aside I suspect going back there were a number of what we now call safety critical systems that did operate on a bit of a wing and a prayer in the early days.

Lots of similar examples over the years.  I’d rather have them fail safe than carry on regardless as in the case of the Therac 25 machines of the 80s.

True, but it must be said that AFAIK, not a single aircraft crashed as a result of this failure, certainly no lives were lost (unless you include heart attacks and the like resulting from stress at the delays). Every single aircraft that was already in the air landed safely.
I remain opposed to air travel in all but exceptional cases, but that is due to climate change, not fear of accidents.

I too am not worried about aircraft crashing when I fly, but I am ever more inclined to avoid air travel due to the tedious waiting, queuing and herding that's involved nowadays before and after a flight, including the faff of getting to and from the airport.  A 2-hour flight often has a door-to-door time of 10 hours for me - that's two daysworth of tedious overhead when taking a holiday.  Then add in the environmental impact of flying, and I'm becoming increasingly inclined to stay in Britain - there's so much to enjoy here.

I have seen a comment that the system is designed that if there is a significant data input problem then it will fail safe by reverting to manual.

In this case fail safe prevented a lot of flights.

Sums it up quite well!!!

I presume that aircraft are exempt from the ULEZ rules and can fly over the relevant areas without restriction.

Willie Walsh quoted by almost any news source:

“… questioned whether the firm should continue to hold responsibility for handling the UK's flight traffic.”

Is there a spare complete ATC system somewhere, (with suitable staff just sitting around waiting), ready to be switched on?   ???

Paul

A two-hour flight will take you about 1,000 miles, which isn't bad for 10 hours. It all depends on where you want to go.

A fast and direct train could take you 1000 miles in 10 hours, in greater comfort and with less stress than air travel.
Even better if it was a sleeper train.

Mmmmmm ! I flew to Denver and back in May this year, I went over 5000 miles in 10 hours. I went Business Class (out my own pocket !), I was in better comfort that ANY train I have ever travelled on; food, wine and service were top notch. My little "suite" on the aircraft converted into a very comfortable bed as well.  The only stress was worrying that GWR would fail to get me to Heathrow on time - they did manage that OK, but I did claim delay repay on the return trip.

It may also be worth saying that my trip to Denver cost me 21p per mile, a recent trip to Temple Meads on a train that the RSPCA would have shown interest in, if cattle had been in it instead of mere humans, cost me 27p a mile.

(I used to travel 1st Class on the train, but the appalling "trains" that we now have are just not worth the extra money - I'll put the savings toward another very comfortable trip with BA)

A London to Barcelona Sleeper then ... 9:30 p.m. in St Pancras, 8:30 a.m. in Barcelona.   Return 10:30 p.m. from Barcelona, 7:30 a.m. in St Pancras.  Just under 950 miles?  Daytime train with the same hours, swapping p.m. to a.m. with my timings?    You could do London to Munich in similar timings.

............or you could fly there in 2 hours, almost certainly a great deal more cheaply -  and the comments about comfort and "less stress" are entirely subjective, especially if you were relying on the railways to get you to St Pancras!!!

Somewhere in the first chapter of every textbook on how to fly a plane, you will read the most indisputable piece of wisdom on the subject of whether or not to take off, repeated at regular intervals throughout the course:

"It is better to be down here wishing you were up there, than up there wishing you were down here."

I might add that it is better as a passenger to be inconvenienced at a full stop on the ground than at 500 mph at 35,000'. They did the right thing.

Grahame's piece on the unimagined consequences of a precise number of data cards was fascinating, and reminded me of my teenage years. My parents ran a guest house between Blackpool and Fleetwood, and our phone number was Cleveleys 2100. At that time, local calls didn't need you to dial the area code. From time to time, usually in the evenings, we would get a call from someone who was after the operator, and had somehow got the number 2 in front of the number. I had fun asking them to help with a frequency test by whistling the national anthem, but it remained a mystery until a friend left school to start an apprenticeship with the then GPO telephone people. He launched his own small investigation at the local exchange, which was then very much analogue, with row upon row of what were called "uniselctors", basically electrically operated switched with contacts matching each digit dialled. If one dialled a number, the first in the bank would click into life and connect the call. If that was in use, the next call passed to the second, and so on. He found a fault in number 596 (or whatever) in the queue that meant that the number 2 was always present. To get through to me at home, the caller had to the 596th (or whatever) making a call at the same time, and had to be trying to call the operator. Given the chances of both conditions being met, we had a surprisingly large number of calls. It is another example of how an unknown fault in a system can cause an unexpected outcome, no matter how tiny the chances of it happening.

I rather missed it when he fixed the fault, although there was a second string to my mischief bow. Fleetwood football club, then non-league was Fleetwood 2100, and we got quite a few calls on Saturdays from people who forgot to add the area code asking "What time is kick-off?" to which I would ask "Are you playing or watching?" We always listened for the full-time score though, so they wouldn't have to make a second call. Again, it surprised me how many people cared, but were a bit useless with a phone.


I'm thinking of heading back to Madeira for some of the cold bits of the winter at home. Nothing in the Trainline so far as I can see...

Relishing a challenge, I tried to find out how quickly I could get from Reading to Sants Station, Barcelona, which is the closest to La Rambla, heart of tourist Barcelona, using the Rio2Rome website. OK - I assumed that I would be travelling by train to Heathrow, but starting out at 8 am tomorrow.

By air, the problem was that by the time I arrived at Heathrow, using the first available (08-20) train from Reading, the next available flight allowing for check-in was not until 12-55, arriving 16-00 local time, and at Sants (assuming I could recover any hold baggage in time to catch a metro train by 16-38) was 16-58. It would be 30 minutes later if not. The problem is the check-in time of 2 hours before departure, with the earliest arrival at Heathrow being 09-10 for a 10-35 previous departure. I guess it would work, but do I want the stress? I could also arrive a couple of minutes later using the Rail Air coach from Reading, but the same considerations apply, albeit if I did get the 10-35 flight it would be a 13-50 arrival at El Prat, and a likely 14-58 arrival at Sants.

For a train trip, I could delay departure from Reading until 09-30, and with changes across Paris, I can pick up an InOui TGV direct, arriving at Sants at 21-25 locally.

So - yes, it's quicker, but I find airports much more stressful than railway stations. And while there's a choice of fiddling around to get to Heathrow for the air alternative and the nuisance of crossing Paris by train, at the other end it's straight into the heart of Barcelona if you go by rail. 7 hours 38 minutes versus 10 hours 55 minutes - and I would enjoy watching the French and Spanish countryside roll past, much nicer than hoping I get a window seat and straining my neck to see what's underneath - assuming clear skies!

Finally prices - the train alone (with transfers/Underground/Paris Metro) - £343. By air (ignoring Rail Air bus option) - if the 10-35 flight - £587-94; if the 12-55 flight, £320-94.

Or, given that you're starting from Reading, you could chuck your toothbrush and a pair of budgie smugglers in your bag, nip down to Gatwick, and be there later today with change from £100 for a pint of San Miguel and an ice cream! :)

I'd come to think the different reactions to this event were due to how "data" was understood. I thought (and still do) that flight path data are what this safety-critical always-on data processing system processes, and  nothing in that should compromise system safety. There should be a one-way barrier between the system and the data it processes, and if faulty data are put in they should be rejected. So I'd concluded that in this event "data" must be something on the system side of that barrier, not flight plan data at all.

But I was wrong: it was a flight plan wot dunnit. NATS gave the CAA an interim report on Monday, which was published yesterday (https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf). And in this it says:


I don't follow that. It's true that this plan came from Eurocontrol, where it was accepted despite containing contradictory data (distinct waypoints with the same ID likely to be interpreted as an impossible path). But I can't see that a flight plan is safety critical in any sense that prevents it being rejected at least four hours before the aircraft in question reaches UK airspace. Isn't it safety critical in the sense that it must be rejected?

Incidentally, NATS have at least managed to name the supplier of this delinquent software (FPRSA-R) as the Austrian company Frequentis AG.

It should have validated it on receipt & kicked it back/rejected acceptance at that time, rather than simply shut everything down & gone to manual input.

The CAA's independent Review Panel for this incident has issued an interim report (https://www.caa.co.uk/publication/download/21478/). This concentrates on how NATS and other organisations coped with the loss of the National Airspace System service (NAS). Obviously that aspect will get most of the media interest, including the bit where the panel note that "some relationships between aviation sector stakeholders appear to be adversarial."

However, it does tell us a little more about what caused the system to fail. There was an earlier preliminary report from NATS (https://www.caa.co.uk/publication/download/20648) on this (and there is also a final report, seen by the panel but not yet signed off by NATS and issued). That explained that a duplicate waypoint ID in the flight plan caused the processing failure, but his new report illuminates that further.

The relevant system is FPRSA-R, which takes flight plans from the AMS-UK (Aeronautical Messaging Switch) in the European standard format (ADEXP), identifies and marks the entry and exit points to UK airspace, and converts them into a domestic format and transfers them to NAS. Both AMS-UK and NAS are trusted to keep flight plans safely and not lose them; FPRSA-R is not - it's purely processing.

That means FPRSA-R is allowed to just shut down if it's unhappy, provided the processing is suitably interlocked. Thus when a flight plan is read from the output queue of AMS-UK, it is not removed - just copied. Only after it has been processed and handed over to NAS, and it has been confirmed as accepted, is the AMS-UK queue allowed to delete that entry and offer the next flight plan to FPRSA-R.

Despite all this talk about safety-critical errors, this behaviour of FPRSA-R looks to me like an unhandled exception. The initial recovery action was to restart it, but it tried to process the same flight plan and so failed again. The AMS-UK output queue was stuck with the invalid plan as its leading item. When the makers' expert from Frequentis AG was eventually called in, their advice was to transfer this flight plan into a new, unconnected, queue; in effect to quarantine it. It could then be given to a human operator to see if it could be entered manually, or if not, why not. Restarting FPRSA-R then succeeded, though it was another hour before the system was running again.

We are told NATS have a fix to stop this happening again, but not what this is. I can think to two obvious fixes. The shutdown can be allowed to happen as before, but the operators are trained to recognise its dying message and do this quarantining manually and restart. Alternatively some of this can be automatic: the quarantine action quite easily, avoiding the shutdown of both FPRSA-R systems with more difficulty.

This new report gives an explanation of how FPRSA-R works, and how the duplicate waypoint IDs arise and are handled (though no doubt this is simplified). From that, it seems clear it's just not clever enough. There is enough information available resolve the duplicates, but it isn't used. And behind that, I think there is work going on internationally to get rid of the short IDs that cause the duplications, but it's exactly the kind of non-urgent task for which progress in the international aeronautical community is at best very, very, slow.

I was caught up in this last august, delayed in Milan by 25 hours.

EasyJet paid me all my expenses within 5 working days, but no compensation eligible as it was outside of their control.

Do we know which airline made the faulty flight plan?

It wasn't "faulty" when written. The processing in Brussels adds extra interpolated waypoints to flight plans, and in this case added one at Deauville, which has the ID code DVL. It already had a waypoint at Devil's Lake (North Dakota), which also has ID code DVL. That was probably a mistake - there's loads of possible waypoints to pick. And, taken in sequence, they can be located 4000 miles apart and the ambiguity resolved. It was the rather simplistic processing in FPRSA-R that led to it being unusable. And that should not have been such a drama.

Interesting angle on it here too;

https://www.ft.com/content/8a90caa6-bf37-4de3-b0b2-e6d6e257e434

Unfortunately behind a paywall