Seriously?
If there was one tenuous application of machine learned algorithms where EXPLAIN-ABILITY was paramount, this is it.
When a decision can send a man or woman to jail, or perhaps even to the gallows, false positives are not tolerable! And a hidden magic algorithm is absolutely out of question. Accuracy of the decision does not matter as much as having an open understanding and explanation of how the decision was arrived at. It is the cornerstone of modern jurisprudence that it is better to allow ten guilty people to go free rather than punish a single innocent person.
Would you like to be judged by machines in court? This is not a matter of recommending merchandise or predicting clicks – this can be a matter of life and death. Humans, however imperfect, are the only agents who should be allowed to judge other human – always a jury of peers. We should not allow algorithms and ML anywhere near the courtroom. This is different from using ML in law enforcement – predicting locations where crimes can happen or such – which may be acceptable with proper controls. But the courts are our last resort, and their sanctity and impartiality must be guarded zealously.
It is not simply a matter of doing justice, it is equally important to be seen to be doing justice.
Anand's musings 