Catching law breakers (or those who have made a mistake) should never be incentivised.
Agree with your general point. But penalty fares sit in a strange legal limbo. They are not a criminal fine but rather a civil penalty. But they are accompanied by the scare tactics etc to make people think that they are a fine.
I would prefer a system where all on-board staff were incentivised to sell tickets to passengers (not with targets but with a percentage margin on the ticket price), and (with appropriate exceptions for unstaffed stations, broken ticket machines etc) tickets were more expensive to buy on board because they include an "onboard purchase supplement" of say £5 each way or £15 on an intercity service" The incentive would be there to "buy before you board" but the phycology would be different and the inspectors would have a far easier and calmer time of it because the person buying on board would see it as simply and foolishly having to buy a poor value ticket rather than being "caught and punished".
Deliberate fare evasion (with intent proven to a criminal standard of evidence) would continue to be prosecuted as now, but not by private companies but rather public servants.
I dislike the current system because it incentivises the ToCs to de-staff train. My proposal would involve an on-board purchase supplement which would be enough to fund the ticketing staff continued presence on the train.