Largest ever miscarriage of justice?

A secret room full of secret terminals running secret, unaudited applications to fix data because the system didn't work AND THEY STILL TOOK PEOPLE TO COURT, SENT THEM TO PRISON, AND DROVE SOME OF THOSE PEOPLE TO COMMIT SUICIDE!

It's no wonder they are wriggling on the witness stand. I doubt any will do time, but I hope most will be sued for every penny.
 
Jenkins was the expert witness used by the PO to dispute defences that something was wrong with Horizon.

Expert witnesses are used by prosecutors and the defence to provide detailed and supposedly reliable evidence on a topic they have expert knowledge of. A common example of expert witness evidence, are doctors who make comment on injuries, or the lack of such, in cases of assault or murder.

Yes, I understand all that. What I don't understand is that Jenkins - a man who helped build Horizon - was called as an expert witness in any case (or inquiry) that involved Horizon. That's a huge conflict of interest.
 
CPS - Expert evidence

The duty of an expert witness is to help the court to achieve the overriding objective by giving opinion which is objective and unbiased, in relation to matters within their expertise.

I don't see how Jenkins could have been seen to be "objective and unbiased," he'd been at ICL/Fujitsu for over 30 years
 
A question.

Firstly, there are about 23,000 sub-postmasters in the UK. About 900 were prosecuted... that is less than 4%. I'm not sure how many more were having issues with money going missing, but it seems Horizon worked fine for about 95% of the SPMs
Only 4% of cases got as far as prosecution? I'd say that is a lot. There are probably many more instances where there was a glitch and the sub postmaster covered the shortfall themselves and it never went further.
Secondly, fibre Optic cables didn't start being laid and used for internet in the UK until abput 2008, so all of these SPM systems must have been connected with copper landlines and dial-up or VDSL. If I recall my VDSL protocols correctly, you had to be within a certain distance of the cabinet/exchange (a half mile?) for VDSL to work properly.

So my question is, what was it that was peculiar about those who were having trouble? Did they have something in common, like, were they rural? Or using dial-up, or VDSL a long way from their nearest cabinet?

It may even be the case that, originally, they were not using the Internet and the terminals communicated using dial up lines. In any case, even with reliable networking, distributed systems like Horizon should be designed to be robust. It doesn't matter how reliable the network connection is, the cleaner pulling the plug on the router to plug in the vacuum is a possibility that must be accounted for, especially where money is concerned. From what I have heard about Horizon's architecture, it doesn't seem that it was so reliably designed.
 
It may even be the case that, originally, they were not using the Internet and the terminals communicated using dial up lines. In any case, even with reliable networking, distributed systems like Horizon should be designed to be robust. It doesn't matter how reliable the network connection is, the cleaner pulling the plug on the router to plug in the vacuum is a possibility that must be accounted for, especially where money is concerned. From what I have heard about Horizon's architecture, it doesn't seem that it was so reliably designed.

I got the impression the overall design was good and specifically allowed for a post office to be offline for a while. The real problems were in the handling of individual transactions. For example, I recall there was a bug that would let a transaction be silently posted twice. So, for example, a deposit of £100 could get posted twice and at the end of the day there would be a cash shortfall of £100.

Because the Post Office believed Horizon was reliable, they refused to believe it could have lost track of that £100 and instead charged the subpostmaster of pocketing it.

Every report of a shortfall should have been investigated to determine if the fault was with Horizon. In the case of £100 going missing, that investigation should have turned up the double-posted transaction. That in turn should have been investigated to determine why it had been posted twice, the fault documented and fixed, and an update rolled out. If there were multiple issues like this, for every one found any still open tickets should have been checked to see if the newly discovered fault was responsible for the problem.

That's how the team I was on did things back in the 1980s with the country elevator system.
 
I got the impression the overall design was good and specifically allowed for a post office to be offline for a while.

That's not necessarily good design. Such a distributed system has to be designed very robustly to account for communications failures.

The real problems were in the handling of individual transactions. For example, I recall there was a bug that would let a transaction be silently posted twice. So, for example, a deposit of £100 could get posted twice and at the end of the day there would be a cash shortfall of £100.

By robust design I mean that, when the transactions are sent from the terminal to the central server, there needs to be a protocol in place so that the terminal knows when a transaction has been successfully recorded by the central server and also a means to identify transactions that have not been successfully sent.

You could have a sequence like this:

  1. terminal sends transaction
  2. server records transaction
  3. server sends acknowledgement of recording
  4. terminal marks transaction as sent

If communications fail after the server has recorded the transaction but before the terminal has received acknowledgement, the two records at terminal and server will be in an inconsistent state. There has to be a way to bring them back into consistency and you can't just send the transaction again because that could result in a double post, if the server can't identify it as a transaction it has already got.

You may say it is relatively trivial to fix that and it is, but back in the late 90's, the world was not as connected as it is now and it is quite possible that the engineers didn't think of it.

Because the Post Office believed Horizon was reliable, they refused to believe it could have lost track of that £100 and instead charged the subpostmaster of pocketing it.
I think you can assume we all know the basic facts of the case and you don't need to repeat them.
Every report of a shortfall should have been investigated to determine if the fault was with Horizon. In the case of £100 going missing, that investigation should have turned up the double-posted transaction. That in turn should have been investigated to determine why it had been posted twice, the fault documented and fixed, and an update rolled out. If there were multiple issues like this, for every one found any still open tickets should have been checked to see if the newly discovered fault was responsible for the problem.

That's how the team I was on did things back in the 1980s with the country elevator system.

I don't necessarily agree. There are many reasons why there might be a shortfall including dishonesty or genuine mistakes, but as soon as it was clear that a pattern was emerging - this might be as simple as several sub postmasters reporting similar problems - Horizon should have been investigated.
 
Last edited:
... it was clear that a pattern was emerging - this might be as simple as several sub postmasters reporting similar problems - Horizon should have been investigated.

this is causing me anguish. Spotting patterns is data, following them up to see if they're significant, and then trying to find the cause of the problem. They knew there were problems, and they spent, what tens of millions, and, worse, ruined hundreds of lives covering it up.
 
That's not necessarily good design. Such a distributed system has to be designed very robustly to account for communications failures.



By robust design I mean that, when the transactions are sent from the terminal to the central server, there needs to be a protocol in place so that the terminal knows when a transaction has been successfully recorded by the central server and also a means to identify transactions that have not been successfully sent.

You could have a sequence like this:

  1. terminal sends transaction
  2. server records transaction
  3. server sends acknowledgement of recording
  4. terminal marks transaction as sent
Which is not that different from the example given in an introductory class I did in 1983. You want to transfer £20 from your bank current (everyday) account to a savings account. So the program has to extract £20 from one account, add £20 to another account, save the changes. When should the changes be saved? What if the program crashes or the compuer loses power etc?
If communications fail after the server has recorded the transaction but before the terminal has received acknowledgement, the two records at terminal and server will be in an inconsistent state. There has to be a way to bring them back into consistency and you can't just send the transaction again because that could result in a double post, if the server can't identify it as a transaction it has already got.
This was a well known problem at the time Horizon was designed. IBM's LU62 communication protocol was designed to manage this and allow a 2 phase commit.
At the very least every transaction (or attempted transaction) should have a unique key. Off the top of my head, the terminals unique id, microsecond timestamp, direction (eg terminal to server) etc. Any loss of comms should include a resolution process to ensure all both ends agree on the transaction state.
You may say it is relatively trivial to fix that and it is, but back in the late 90's, the world was not as connected as it is now and it is quite possible that the engineers didn't think of it.
In around 2008 I was asked to be a general IT guy on a project to sort out longstanding issues with a bank. One problem was one customer (an insurance company) complained about the bank's secure mailbox. It was a Sterling product whose name escapes me and we had a live and contingency instance. The problem was they didn't talk to each other so if this customer read a package from the live server it was still marked as unread on the contingency server. This gave them problems. My answer of "tell them to sanitize their inputs" as the packages could clearly be seen to be duplicates was (cough) "unwelcome". So I had to dig out my C book (Banahan and Rutter from 1983) and use the products user exit points to write code that could tell the other instances a package had been read. In 2008!
don't necessarily agree. There are many reasons why there might be a shortfall including dishonesty or genuine mistakes, but as soon as it was clear that a pattern was emerging - this might be as simple as several sub postmasters reporting similar problems - Horizon should have been investigated.

Yes ITIL was a thing in 1996 and the PO's ties to UK government should have seen it an early adopter. That should have included some root cause analysis and maybe some tie up with known Horizon bugs
 
Last edited:
That's not necessarily good design. Such a distributed system has to be designed very robustly to account for communications failures.



By robust design I mean that, when the transactions are sent from the terminal to the central server, there needs to be a protocol in place so that the terminal knows when a transaction has been successfully recorded by the central server and also a means to identify transactions that have not been successfully sent.

You could have a sequence like this:

  1. terminal sends transaction
  2. server records transaction
  3. server sends acknowledgement of recording
  4. terminal marks transaction as sent

If communications fail after the server has recorded the transaction but before the terminal has received acknowledgement, the two records at terminal and server will be in an inconsistent state. There has to be a way to bring them back into consistency and you can't just send the transaction again because that could result in a double post, if the server can't identify it as a transaction it has already got.

You may say it is relatively trivial to fix that and it is, but back in the late 90's, the world was not as connected as it is now and it is quite possible that the engineers didn't think of it.


I think you can assume we all know the basic facts of the case and you don't need to repeat them.


I don't necessarily agree. There are many reasons why there might be a shortfall including dishonesty or genuine mistakes, but as soon as it was clear that a pattern was emerging - this might be as simple as several sub postmasters reporting similar problems - Horizon should have been investigated.


Speaking from the other side of the screen, as were the SPM's, a system showing a shortfall is unlikely to point to dishonesty because dishonest people makes sure the system balances. For example, 'Rogue Trader', Nick Leeson, a dealer at Barings in Singapore, simply traded with company money and bunged all of the losses into a 'suspense account' - which was labelled '9999' on the Balance Sheet - so of course, all his stuff balanced. With the SPM's AIUI Horizon was deliberately designed so that users couldn't just balance shortfalls out by making a counter entry in a 'suspense account', they had to have a proper voucher for each transaction. It is the honest ones who ring up HQ or write detailed memos explaining the problems encountered. The other problem is they had no access to an audit trail, so if their figures were out - and some spent all weekend trying to balance their books, to no avail - eventually, they either had to 'get around the problem' (which the PO labelled 'false accounting') by using a voucher from another period or using their own savings to make it add up. Of course, the ones who were honest enough to report their problems, were the ones who got a visit from the PO auditors, who, on discovering the SPM's were getting the books to balance via receipts belonging to another day or an imaginary figure, they'd be immediately marched out and locked out of their own sup Post Office, one or two arrested in front of everybody.

I have come across workplace fraud in serious embezzlement on at least four occasions. The crooks made sure everything balanced. So the PO assuming fraud because Horizon would not balance seems strikingly perverse to me. The latest testimonies from Gareth 'not that I recall, it was a long time ago' Jenkins and now Tim Parker evading Jason Beer's straightforward questions fills me with scepticism of their 'I knew nothing ATT' line of defence. Looking forward to Mark Davis' testimony or has he been?* as he seems to be hated by everybody, even by Vennells, as her right hand henchman.

*ETA: Yes, of course, he appeared end of May. We discussed him at the time - a complete spin doctor type.

https://www.youtube.com/live/tiPs9pBJdJA?si=8_aheDx1tY0L-4aH
 
Last edited:
I finished reading The Great Post Office Scandal by Nick Wallis.

It was first published in 2021, and this edition was updated in 2022 when the statutory enquiry was beginning.

Now I need to get my head around what’s happened since then.
 
I finished reading The Great Post Office Scandal by Nick Wallis.

It was first published in 2021, and this edition was updated in 2022 when the statutory enquiry was beginning.

Now I need to get my head around what’s happened since then.


I sometimes listen to something I think I'll find uninteresting to fall asleep to. The videos from the hearings were trending on YouTube and I figured, "Parliamentary Inquiry into the Post Office", that's the ticket; I'll be out like a light. Wow.

I listened to one of the videos and then bought the book. This is so horrifying. The problem isn't that the people at POL were stupid or that they were evil. They were stupid and evil. They wrote down the evil down on purpose. They edited witness statements with track changes left on. They sent e-mails about not leaving paper trails. They knew they were breaking the law and ruining peoples' lives and sent each other e-mails about it. They even set up a fake union for the sub postmasters.

Horizon wasn't really the problem. The problem was that people who ran the Post Office were mentally subpar cartoon villains.
 
I sometimes listen to something I think I'll find uninteresting to fall asleep to. The videos from the hearings were trending on YouTube and I figured, "Parliamentary Inquiry into the Post Office", that's the ticket; I'll be out like a light. Wow.

I listened to one of the videos and then bought the book. This is so horrifying. The problem isn't that the people at POL were stupid or that they were evil. They were stupid and evil. They wrote down the evil down on purpose. They edited witness statements with track changes left on. They sent e-mails about not leaving paper trails. They knew they were breaking the law and ruining peoples' lives and sent each other e-mails about it. They even set up a fake union for the sub postmasters.

Horizon wasn't really the problem. The problem was that people who ran the Post Office were mentally subpar cartoon villains.

Well said!

They ignored their own independent inquiry. Then when their evil was found out, they still set up a fake mediation scheme, and when successfully sued and brought to an inquiry, they still deny their evil, while apologising for it. Stupid and evil.
 
That's not necessarily good design. Such a distributed system has to be designed very robustly to account for communications failures.

By robust design I mean that, when the transactions are sent from the terminal to the central server, there needs to be a protocol in place so that the terminal knows when a transaction has been successfully recorded by the central server and also a means to identify transactions that have not been successfully sent.
For sure. In fact, it doesn't matter if the two computers are right beside each other on the desk or a few thousand kilometres apart: this sort of checking should be in place for any sort of critical data.

You could have a sequence like this:

  1. terminal sends transaction
  2. server records transaction
  3. server sends acknowledgement of recording
  4. terminal marks transaction as sent
If communications fail after the server has recorded the transaction but before the terminal has received acknowledgement, the two records at terminal and server will be in an inconsistent state. There has to be a way to bring them back into consistency and you can't just send the transaction again because that could result in a double post, if the server can't identify it as a transaction it has already got.

You may say it is relatively trivial to fix that and it is, but back in the late 90's, the world was not as connected as it is now and it is quite possible that the engineers didn't think of it.
Here's a recent thread on Reddit about how Horizon was initially programmed: The Horizon IT system - how buggy software written in an ancient programming language caused disaster [Reddit] (The "ancient" programming language was Visual BASIC, which wasn't quite so ancient in the mid to late 1990s.)

The initial design of Horizon worked in a fashion very similar to what you described. Each post office had one or more counter computers and a Windows NT Server computer (in your description above, this is the "central" server.) As business was done throughout the day, the counter computer(s) generated one or more XML "messages." Using a set of software called Riposte, which appears to be a message queuing system, the messages were sent to the server. Riposte would have the responsibility for ensuring any message generated at the counter was indeed sent to the server with acknowledgement of receipt, and for ensuring a single message wasn't sent twice.

Once a day the server would run a program that would read the message log forward from its previous checkpoint, create an batch update set for head office, and write a new checkpoint.

According to the Reddit thread, there was no consistent data format for the XML messages, nor was there a proper XML schema against which messages could be validated. That meant a slip-up by the programmer that generated a malformed message wouldn't necessarily be caught by the server. And depending on how the server was programmed, when it found a message it couldn't understand, it might simply ignore it instead of generating an error and stopping.

The Reddit thread also says the Submit function on many of the subsystems was non-blocking. If someone pressed "Submit" twice, a second (and new) message would be generated. Sometimes people would click "Submit" several times if the computer was taking an unusually long time to complete a transaction, unaware the system was duplicating the action every time.

I don't necessarily agree. There are many reasons why there might be a shortfall including dishonesty or genuine mistakes, but as soon as it was clear that a pattern was emerging - this might be as simple as several sub postmasters reporting similar problems - Horizon should have been investigated.

Absolutely! Hence the scandal.
 
My eyebrows shot up so far I may need to shave them off and grow new ones.

They'll be on your arse after this:

The problem was also technical. The Horizon system was never fit for purpose on inception. One case that came out was of a subpostmaster receiving £8k from central and registering it back with the system. He pressed the button 4 times because the system was slow. The central system recorded 4 separate deficits of £8k while the local terminal only saw 1. He was sued for stealing £24k.
 
And core problems were known at a high level:

Our source said Fujitsu’s unwillingness to deal with this central problem became a bone of contention with him. He said he was offered more responsibility on the project but refused unless the cash account was rebuilt from scratch. ...snip.....

He concluded: “It was a prototype that had been bloated and hacked together afterwards for several years, and then pushed screaming and kicking out of the door. It should never have seen the light of day. Never.”
 
Let's not forget the corporate culture of 'brown-nosing', with people encouraged to congratulate the bosses on their various awards and achievements, even though they were rubbish - a form of social coercion. Staff were told they were learning how to be team players and fitting in well when they did this stuff.

One was even commended for her arse-licking by Angela van den Bogerd, the business improvement director who would later be labelled untrustworthy by a judge and who had a torrid time at the inquiry last month. After the official had emailed "truly inspirational woman" Vennells, van den Bogerd wrote: "Taking the initiative to send a note to Paula is a clear indicator of just how much she has grown in recent years."
Private Eye 1623

The sycophancy as encouraged by Vennells and her Communications Director, Mark 'Spin' Davis is truly vomit inducing.

As the honours were unveiled on 28 December 2018, Vennells wrote gushingly to Davies, who had vehemently defended the Horizon IT system when BBC's Panorama picked up on it in 2015, with legal threats and complaints to BBC bosses. "I wanted to thank you personally for your huge contribution towards this honour," Vennells wrote. "I have lost count of the number of times I have relied on your judgement, listened to your wisdom and then taken your advice over the last few years. Your call was always the right one: guiding us through stormy waters of all kinds. I count myself very fortunate to have worked with you!"
ibid
 
The sycophancy as encouraged by Vennells and her Communications Director, Mark 'Spin' Davis is truly vomit inducing.
As the honours were unveiled on 28 December 2018, Vennells wrote gushingly to Davies, who had vehemently defended the Horizon IT system when BBC's Panorama picked up on it in 2015, with legal threats and complaints to BBC bosses. "I wanted to thank you personally for your huge contribution towards this honour," Vennells wrote. "I have lost count of the number of times I have relied on your judgement, listened to your wisdom and then taken your advice over the last few years. Your call was always the right one: guiding us through stormy waters of all kinds. I count myself very fortunate to have worked with you!"

Ugh! What sicking sycophancy!
 
The stupid was in evidence on the stand yesterday, evil remains to be proven. The buck passing strongly suggests it

Editing a witness statement to take out exculpatory information is evil. Stupid is leaving the track changes on so you can hover over the edit and see who made the changes and when. Extra stupid points for explaining in an e-mail that you made the edit to avoid providing said exculpatory information.
 
Yes, I understand all that. What I don't understand is that Jenkins - a man who helped build Horizon - was called as an expert witness in any case (or inquiry) that involved Horizon. That's a huge conflict of interest.

That is where the courts failed. The lawyers and judge knew he worked for Fujitsu and was invested in Horizon. He was the expert in how it functioned, but was he going to reveal potential faults? The answer is a resounding no.

He was used by the prosecution as an expert, which is often countered by defence experts. But the sub-postmasters could not find or afford experts to counter the PO.
 
And core problems were known at a high level:
Our source said Fujitsu’s unwillingness to deal with this central problem became a bone of contention with him. He said he was offered more responsibility on the project but refused unless the cash account was rebuilt from scratch. ...snip.....

He concluded: “It was a prototype that had been bloated and hacked together afterwards for several years, and then pushed screaming and kicking out of the door. It should never have seen the light of day. Never.”

It certainly wouldn't have been the first time that management failed to understand the meaning of 'prototype', and insisted on going with that rather than actually developing a properly designed version.
 
I just finished, "The Great Post Office Scandal". What I don't understand is how a collection of people this stupid did so much damage for so long. They routinely demonstrated consciousness of guilt, in writing, for years. The oversight in government was in collusion, dumber than the idiots in the Post Office or some combination thereof. I simply cannot grasp how it didn't occur to anyone in the government oversight role that maybe they should have a look at private prosecutions undertaken by the aggrieved party.
 
I just finished, "The Great Post Office Scandal". What I don't understand is how a collection of people this stupid did so much damage for so long. They routinely demonstrated consciousness of guilt, in writing, for years. The oversight in government was in collusion, dumber than the idiots in the Post Office or some combination thereof. I simply cannot grasp how it didn't occur to anyone in the government oversight role that maybe they should have a look at private prosecutions undertaken by the aggrieved party.

I think the answer is that there was no way respected businesses like Fujitsu and the PO could be so wrong, so the wrong became a right and it took a lot for those involved to finally understand that they were wrong. Some still refuse to accept they were wrong, as seen in the evidence to the Inquiry.
 
I got the impression the overall design was good and specifically allowed for a post office to be offline for a while. The real problems were in the handling of individual transactions. For example, I recall there was a bug that would let a transaction be silently posted twice. So, for example, a deposit of £100 could get posted twice and at the end of the day there would be a cash shortfall of £100.
Because the Post Office believed Horizon was reliable, they refused to believe it could have lost track of that £100 and instead charged the subpostmaster of pocketing it.

Every report of a shortfall should have been investigated to determine if the fault was with Horizon. In the case of £100 going missing, that investigation should have turned up the double-posted transaction. That in turn should have been investigated to determine why it had been posted twice, the fault documented and fixed, and an update rolled out. If there were multiple issues like this, for every one found any still open tickets should have been checked to see if the newly discovered fault was responsible for the problem.

That's how the team I was on did things back in the 1980s with the country elevator system.

This bit should read "Despite the post office knowing horizon to be unreliable". There's plenty of evidence out there that shows senior bods at the post office knew horizon was wonky but swept the problems under the carpet due to wanting the company to look profitable for privatisation.
 
"Former sub-postmaster Noel Thomas will be honoured for his role in campaigning for justice against the Post Office.
He will be awarded an honorary degree by Bangor University on Tuesday for his public service in raising awareness of the Horizon scandal."

https://www.bbc.com/news/articles/c4ngrd7wyrgo

The article is badly worded, but maybe Sir Alan Bates is getting one too.
 
I think the answer is that there was no way respected businesses like Fujitsu and the PO could be so wrong, so the wrong became a right and it took a lot for those involved to finally understand that they were wrong. Some still refuse to accept they were wrong, as seen in the evidence to the Inquiry.

George Thomson!
- Still believes there was nothing wrong with Horizon.
- Still thinks the sub-postmasters were all crooks.
- Still thinks the PO acted correctly.
- Still doesn't understand that he shafted his own people.
 
This bit should read "Despite the post office knowing horizon to be unreliable". There's plenty of evidence out there that shows senior bods at the post office knew horizon was wonky but swept the problems under the carpet due to wanting the company to look profitable for privatisation.

That's a more accurate representation of the facts. Everything the Post Office said, even when defending itself against the 555 subpostmasters' lawsuit, repeated the idea it believed Horizon was for all intents and purposes infallible. Except they knew it wasn't.

I've been programming computers for over fifty years. Every non-trivial program has bugs, and large, complex systems can have bugs that are very difficult to track down. Claiming a system is as accurate as the Post Office made Horizon out to be is extremely far-fetched.

The Space Shuttle had almost perfect software, but the resources needed to make it that way were enormous. Part of that was different computers running software written by different teams, and they constantly checked each other.
 
Is a bug a bit of programming that was faulty, causing an unexpected result? Basically, human error causes computer error.
 
Is a bug a bit of programming that was faulty, causing an unexpected result? Basically, human error causes computer error.


Not always directly.

A piece of software can have a "bug" show up as a result of an update... i.e. it wasn't a bug when the original programmer wrote the code, but an update to an operating system can cause the bug to show up. Hardware drivers are an example of this - a piece of hardware can stop working because the driver stops working after an OS update.

I have run into this myself. I have a film scanner that worked in Windows 2000, XP, Vista and Windows 7 & 8 but does not work on Windows 10 (which is why I have a Win 7 VM just to drive the scanner). It worked in the original version of Windows 10, until 2021, when update 21H1 borked it.
 
Not always directly.

A piece of software can have a "bug" show up as a result of an update... i.e. it wasn't a bug when the original programmer wrote the code, but an update to an operating system can cause the bug to show up. Hardware drivers are an example of this - a piece of hardware can stop working because the driver stops working after an OS update.

I have run into this myself. I have a film scanner that worked in Windows 2000, XP, Vista and Windows 7 & 8 but does not work on Windows 10 (which is why I have a Win 7 VM just to drive the scanner). It worked in the original version of Windows 10, until 2021, when update 21H1 borked it.

So the notion of a bug free system is widely understood to be nonsense.
 

Back
Top Bottom