We Humans and the Intelligent Machines. Jörg Dräger

Чтение книги онлайн.

СКАЧАТЬ pick up her stepsister.⁵ She sees an unlocked child’s bicycle at the roadside. The 18-year-old takes it, rides a few yards on the bike, which is far too small for her, and leaves it lying on the ground. Someone yells after her: “That belongs to my son!” Too late. Neighbors recognize Brisha and call the police. She is charged with stealing. At the preliminary hearing, the judge sets bail at $1,000 although the prosecutor has not requested any bail at all. Brisha has to spend two nights in prison before her family can find the money to free her.

      Also in Florida, the same county, a few months earlier: Vernon P. is caught at the hardware store taking tools worth $86.35 – a similar amount the bicycle stolen by Brisha was valued at. Vernon’s criminal past, however, reads more dramatically than Brisha’s. The 41-year-old has been convicted several times and has already spent five years behind bars for armed robbery. Unlike Brisha, who in the past had only committed a few minor offenses, he initially remains at large.

      In both cases, the COMPAS algorithmic system described in Chapter 1 assisted the judges in deciding on detention, bail or freedom. The software calculates and quantifies the probability of a suspect’s recidivism and has been used in many US states for several years in preliminary hearings and pre-trial motions. Brisha’s risk of recidivism in the near future was estimated by COMPAS at 8 on a scale from 1 to 10 – a fairly high score. Vernon’s likelihood of reoffending, on the other hand, was rated only 3 by the algorithm.

      Two similar minor offenses. A young woman who has previously only come under scrutiny for misdemeanors. A grown man, convicted multiple times of robbery. She is classified as a high-risk criminal. His risk of reoffending is considered low. She is black. He is white. COMPAS was introduced to ensure that skin color no longer plays a role in determining penalties. That, in any case, was one of the hopes when the software was first adopted. While judges formerly made inconsistent decisions or unconsciously discriminated against individuals, law enforcement authorities expected the algorithm would assess every person neutrally and without bias, regardless of background or skin color. As a result, the software was not allowed to use these parameters when recommending a sentence.

      Even if New York City managed to reduce the total number of inmates with the help of COMPAS because more defendants were released on bail or probation, the hope that there would be no more discrimination has not yet been fulfilled. While Brisha did not attract attention again after stealing the bicycle, a short time later Vernon ended up in prison for eight years after committing another serious theft. Brisha and Vernon are not isolated cases. The ill-judged results follow a clear pattern. The software systematically overestimates the risk of black people relapsing just as it underestimates the probability that whites will do so.

      Proof of this imbalance was provided by the US non-profit newsroom ProPublica.⁶ The organization, which is dedicated to investigative journalism in the public interest and was awarded the Pulitzer Prize in 2010, examined 7,000 uses of COMPAS in Florida. The investigation revealed that skin color plays a decisive role in calculating the probability of recidivism (see Chapter 12), even though the software is not allowed to include this characteristic at all and was introduced precisely with the aim of preventing discrimination.

      Obviously, a suspect’s race finds its way into the algorithmic prognosis through the back door. One reason for this is that many of the criteria that COMPAS considers correlate with skin color in the US. Social environment, for example, or housing. In addition, the software compares each case with a control group of over 7,000 imprisoned criminals. The greater the similarity between the personal circumstances of a suspect and those of the convicted criminals, the higher COMPAS’s assessment of the risk of recidivism. The problem, however, is that the control group does not represent a cross-section of the American population, but that of its prison inmates. And blacks are disproportionately represented there. For black suspects, this increases the probability of matches with the inmates’ profiles – which causes the algorithm to forecast a higher risk of a repeat offense. As a result, the computer program reinforces existing inequalities. This was also criticized in 2016 by Eric Holder, then Attorney General in the Obama administration. “Although these measures were crafted with the best of intentions, I am concerned that they inadvertently undermine our efforts to ensure individualized and equal justice,” he said. “They may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society.”⁷

      The original goal behind the COMPAS software is based on a broad social consensus: the desire to reduce discrimination and arrive at a just judgment regardless of a person’s background. It follows the principles underlying the rule of law in a liberal democracy. Unfortunately, COMPAS does not achieve these goals. Well meant is by no means well done. The algorithmic system cannot perceive its own failure since it does not evaluate itself. This task falls to those responsible for its use, in this case the American government and its judiciary, or the commercial producer of the software. They all failed to make the program accessible for review. Uncovering the algorithm’s shortcomings thus required a non-profit organization to carry out a time-consuming investigation.

       One-sided learning: Algorithms as self-fulfilling prophecies

      Unlike in Florida, in the state of Wisconsin the COMPAS software is not only used to assess pretrial risk. The judges there also use the recidivism probabilities calculated by its algorithms to determine whether a defendant should go to jail or be put on probation⁸ – a decision of enormous significance for the person in question, and for the public and its feeling of security. Therefore, it is all the more important that COMPAS is able to learn from the results of its forecasts, i.e. when was the algorithm right and when was it wrong?

      The problem is the one-sidedness of this learning process. Parolees can confirm or disprove the system’s prognosis, depending on how they behave during the parole period. If, on the other hand, they go to jail because of the COMPAS recommendation, they have no chance of proving that the software was wrong. This is not an isolated example: People who do not receive a loan can never prove that they would have repaid it. And anyone who is rejected as an applicant by a computer program cannot prove that he or she would have done an excellent job if hired.

      In such situations, the algorithmic system has what is in effect a learning disability, since verifying whether its prognosis was correct is only possible for one of the two resulting groups. For the other, the question remains hypothetical. Freedom or prison, creditworthy or not, job offer or rejection: Algorithms used in the areas of law enforcement, banking and human resources are fed with one-sided feedback and do not improve as much as they should.

      Users must be aware of this and create comparison groups from which the algorithm can nevertheless learn. A financial institution could, for example, also grant a loan to some of the applicants who were initially rejected and use the experience gained with those individuals to further develop its software. HR departments and courts could also form comparison groups by making some of their decisions without machine support, generating feedback data to assess their algorithmic predictions: Was an applicant successful at the company even though he or she would have been rejected by the software? Did someone not reoffend even though the system, as in the case of Brisha, had made a different prediction?

      To reduce the problem of one-sided feedback, users must be willing to examine the issue and be adept at addressing it. Both qualities are urgently needed. After all, one-sided feedback not only creates a learning problem for algorithmic decision-making, it can also reinforce and even exacerbate discrimination and social disadvantages. A longer stay in prison increases the risk of a new crime being committed afterwards; a loan taken out in desperation at exorbitant interest rates increases the risk of default. This threatens to turn the algorithmic system into a generator of self-fulfilling prophecies.

СКАЧАТЬ