Railway staff do have some authority in law. Section 5(2) of the Regulation of Railways Act 1889 allows them to detain a passenger who fails to show or pay for a valid ticket and fails to give their name and address. They can be detained "
until he can be conveniently brought before some justice or otherwise discharged by due course of law."
https://www.legislation.gov.uk/ukpga/Vict/52-53/57/section/5In practice this power is rarely used by railway staff unless the person is compliant.
I saw this in action once when a most belligerent passenger told the
TM‡ on my ride towards t'north exactly what he could do with his ticket machine, and what he would do to the poor man if he didn't stop pestering him for a ticket, because he didn't need one but didn't feel the need to explain why not. The doors remained locked at the platform as our man enjoyed his victory in the queue to get off, the TM having sensibly withdrawn from view. When he reappeared, it was with the two
BTP▸ officers he had let board further down the train. The man had a brief go at them too, brief because they had him face down on the floor, and handcuffed in very short order after the first abusive threat, then frogmarched him off the train to applause. He had been detained by the TM, but subtly.
For general anti-social behaviour and breaches of the Railway Byelaws all railway staff can practicably do is call the police. They wouldn't even be able to make a (much misunderstood) 'Citizens Arrest'. That power only applies when someone has committed, is committing, or is suspected of committing, an indictable offence. An offence that can be tried by a jury. Railway Byelaws only contain summary offences, tried by magistrates.
Correct, although there is often another indictable offence you could hold someone for, like an assault rather than public order. In practice, it's best to leave these things to the experts, a point which will have been part of the training for railway staff.