Probability

[whole diary]

[TerminalJunkie]

A little quiz for you:

90% of the trains at a particular station (Melksham, say  ) run as timetabled: the other 10% are cancelled.

Some bright spark creates a website (www.ismytraingoingtorun.co.uk), which can tell you whether a train will run or has been cancelled. There is a bug in the program however, which means it can only give you accurate information 90% of the time.

You log onto the website, and it says your train is cancelled. What is the probability that train will turn up?

[Worcester_Passenger]

A lot depends on why the website gives accurate/inaccurate information.

Let's assume that there's no linkage between the workings of the website and the running of the trains.

The website will give accurate information 90% of the time, which means that it will report the train accurately as running 81% of the time, and accurately as not running 9% of the time.

For the 'other' 10% of the time, it will give inaccurate information, by which I assume "it gets it wrong". So for 90% of the 10% (aka 9% of the whole time) it will report the train inaccurately as not running (when it actually is). For the other 10% of the 10% (aka 1%) it will report the train inaccurately as running when it's not.

Adding this lot together, it'll report the train as not running for 18% of the time - 9% being accurate information and and equal-and-opposite 9% being wrong.

So I think that when the website says the train is not running, it actually does run for 50% of the time.

[TerminalJunkie]

A lot depends on why the website gives accurate/inaccurate information.

Not in this hypothetical situation!

So I think that when the website says the train is not running, it actually does run for 50% of the time.

Spot on!

[Worcester_Passenger]

There is a bug in the program however, which means it can only give you accurate information 90% of the time.

Suppose that this bug is such that the website is fabulously optimistic - and simply reports that all the trains run all of the time. It'll only be 90% accurate - and it will never report a cancelled train, so the issue doesn't arise!

[grahame]

So I think that when the website says the train is not running, it actually does run for 50% of the time.

Ah yes ...

Looking first at the situation "on the ground" rather than away from the Web, (and picking up the gaunlet of the Melksham example) ... at times that the information point is working, there are occasions that the system reports a train as cancelled and it turns up anyway, and there are times it disappears off the system and it's implied that it has left (or been cancelled) but never the less it appears 15 minutes late!   You won't be surprised to learn that the number of passengers getting on has usually been reduced to around zero by the time it appears as people have phoned for lifts, or made other alternative plans. From Trowbridge (pop 30,000) to Bath, people would just wait 30 minutes for the next service to come along and be happy to see the cancelled one in a few minutes, but from Melksham (pop 24,000) to Swindon, a wait from 07:17 to 19:51 wouldn't be realistic so they would be gone.  So although the train ran, as far as travellers from Melksham were concerned it is as good as cancelled.

I have a lot of sympathy with FGW^▸ (First Great Western) and other information providers - they have a tough job knowing at what point to declare a train cancelled. I note that they very rarely - if ever - say "MAY not run", but in practise trains sometimes go into that state ... there's a 25% chance that the brakes can be unlocked, or a 75% chance that we'll find a spare driver when we phone round and offer an extra shift, and in that interim the service, strickly, should show up as a POSSIBLE non-runner.  [[Question - what IS the policy on when a service is stated as being cancelled?]]

Add the web on top of this and you have a whole extra level of issues - on already shaky data, you add things like the need to cache records, rather than going right back through all the stages of the system to the original source every time someone pulls up the enquiry page. And, yes, it would be so easy to take a cancelled record and assume "once cancelled, always cancelled" when in fact re-instated messages do come up from time to time. TerminalJunkie - if you have a particular URL in mind that makes this sort of error, can you please let me know - especially if it's one of mine;  we do have appropriate disclaimers and explanations around, but I attempt to make everything as accurate as possible and to explain shortcomings of which - baring a very expensive and official system - there are bound to be some.

[TerminalJunkie]

TerminalJunkie - if you have a particular URL in mind that makes this sort of error, can you please let me know

The question I posed is normally put in the form of 'There is a disease that afflicts x% of the population, and a test for the disease is x% accurate', blah blah blah.... It's a nice question for demonstrating the way most people misinterpret statistics and probability (most people will look at the question and answer 90%, a bit like people assuming that using two trains doubles the chance of being late)

I merely reworded it to try to turn it into an on-topic question; there was no intention to slur anyone, and any resemblance to any webmaster, living, dead, or on Facebook, is purely co-incidental.

As it happens, in my first draft there was a notice board at the station that gave that information about cancellations, but I changed it as I thought no-one would have believed it

[lympstone_commuter]

Hi,

Thanks for posting the problem.

I think this can be clarified with a contingency table for the fate of all timetabled trains:

                      actually                 actually
                      runs                      cancelled

reported            A                          B
running     

reported            C                          D
cancelled

If A, B, C, D are expressed as percentages then clearly A + B + C + D = 100.

"90% of trains actually run"       =>   A + C = 90
so 10% are actually cancelled   =>   B + D = 10

"90% of reports are correct"     =>   A + D = 90

Plugging all this in, the contingency table must take the form


                      actually                 actually
                      runs                      cancelled

reported            A                          A-80
running     

reported            90-A                     90-A
cancelled


By my reckoning, "A" (the percentage of timetabled trains that are reported running and do actually run) cannot be explicitly determined from the given information, although it must lie somewhere between 80% and 90% since all probabilities must be greater than or equal to 0.

I agree with the conclusion that half of the trains which are reported cancelled do actually run. ( (90-A)% of all timetabled trains are reported cancelled but actually run. Equally (90-A)% of all timetabled trains are reported cancelled and are in fact cancelled.)

I know - I should get out more 

[grahame]

I merely reworded it to try to turn it into an on-topic question; there was no intention to slur anyone, and any resemblance to any webmaster, living, dead, or on Facebook, is purely co-incidental.

I certainly didn't take it as a slur (though you did get me thinking "how are OUR sites with regard to  false negatives / false positives" which is exactly what any web site developer should do!) - but I am relieved it's not something you've seen on my stuff - which *is* going to have inaccuracies at times due to source inaccurancing and imprecise feeds.

[eightf48544]

Whilst I find mathematical probability fascinating I think the the real question here is not the accurracy of the information but why are 1 in 10 trains cancelled and not run?

I can only assummed FGW^▸ (First Great Western) are trying to run too many trains with too little stock and staff. Now that is another whole batch of statistics to do with productivity and reliability. Accountants expect stock to be available and running 100% of the time. Which in the real world we know is not possible. So the question becomes what percentage of stock will be out of service at anyone time.

When Hull trains had four sets for 3 diagrams they were achieving some of the best miles per failure of any diesel unit but with realtively low utilisation as a whole. When one was dropped and badly damaged they had to use 3 sets every day their failure rate escalated but their utilisation increased.   Bean counters always go for maximum utilisation which more often as not may appear efficient but never gives the best service in practice.

I remember doing calculations of computer specs and disk drives in particular. The manufactures will give you a maximum number of bytes you can read in a second. However if you ever try to achieve that level the response time increases dramatically.

In most systems if you plot utilisation as x axis and result (adverse) on y you get a gently rising line as utilisation increases until at a tipping point it increase sharply. The tipping point is usually between 60% and 75% of theorectical utilisation, depending on the system. Finding that tipping point is the art. In some cases it's mathmatical but in others more empirical methods have to be used.

Empirically I would suggest FGW are over utilising their stock.