Regular readers will remember my writing an article about voice risk analysis software employed by Harrow Council in particular to detect benefits claimants who may be lying. At the time I pointed out that there was no evidence, other than anecdotal, that the product worked as claimed. And that:
“Harrow Council, and soon the Department of Work and Pensions, could well be detecting and prosecuting fraudsters using software whose only claim to be efficacious is that the company that makes it says it is.”
Today I find that this is not just my opinion, but also that of Paul Lewis, presenter of Radio 4’s Moneybox programme. He popped up on Breakfast on BBC1 this morning to put the arguments against the credibility of these devices much more succinctly than I have done.
In brief, Voice Risk Analysis (VRA) software claims to detect stress in a person’s voice, not lying specifically. In day to day use, the software flags up calls from benefit claimants whose voices betray signs of stress and the call can then be followed up with further checks.
The presenters told Lewis that Harrow Council had saved £110,000 by the use of Voice Stress Analysis: Lewis pointed out that the problem was that they had nothing to compare the method to, how did Harrow Council know for instance, that picking people at random and following up their claims would not generate an equivalent saving? Also, simply knowing that Harrow Council was piloting such a scheme might be enough to put off fraudulent claimants (as well as discouraging genuine claimants wary of harassment).
Superbly put. Excellent piece. Also in Lewis’s favour was his mention of Niels Bohr in the context of making predictions about the future. Anyone who can mention a famous physicist at 8:45 in the morning on national television gets my vote.
A piece on VRA also appeared later in the day on the 29th December issue of Moneybox. Lewis was presenting and went into a lot more detail.
It turns out that the supplier of the software is Digilog as predicted on this blog back in September. Digilog, it seems, will not tell anyone how it is able to detect stress in voices, nor why they assume that stress and lying are linked. This should start alarm bells ringing in any reasonable person’s head. Contrary to what your intuition might tell you, systems whose workings are secret are far more vulnerable to exploitation than those which are not; this is because the more people who know how the system works, the more chance there is of them spotting errors, mistakes or other problems in the software. It’s obviously very important that systems designed to catch people lying work properly: no one wants criminals getting away with it on the one hand, or innocent people being harassed on the other.
Unbelievably, James Plaskitt, the Parliamentary Under-Secretary with responsibility for Housing and Council tax benefit in the Department of Work and Pensions chose to beg the question when interviewed for Moneybox. What follows is a paraphrase of the exchange with the interviewer:
Interviewer: What scientific research is there to show that this technology works?
Minister: That’s why we’re running the pilot schemes, to see if it works.
Interviewer: But that isn’t scientific research, is it?
Minister: No, but it’s not up to us to review the science.
Interviewer: Surely it’s important before you implement this system that you know it works. How will you know if you don’t look at the science?
Minister: Well that’s why we’re running the pilot schemes – to test this thing out. What’s more, the operators I’ve talked to who use this system are convinced it’s solid!
With the minister responsible for national rollout talking in circles and presenting anecdote as evidence, it’s no wonder that a device that has been shown to be no more accurate than flipping a coin can gain such a grip on the minds of politicians and civil servants, all of whom are seeing it as a magic wand to cut fraud.
How can intelligent people not spot that there’s no evidence for something, or think that a pilot scheme will establish its efficacy without anything to compare it with? I hope the answer isn’t “because they don’t understand how science works”.
Remember the recent biometrics scandal, where a person wearing a gelatin overlay over their fingers was able to fool a commercial fingerprint detector 80% of the time? The UK government is committed to launching a national ID card scheme based on just such biometric equipment starting in 2009, with about as much evidence that it will work as Digilog have for their VRA software.
The fact is we don’t know that VRA technology works, and have no right to believe that it will work based on the evidence seen so far. The government should not be implementing it.
