'The Algorithm Made Me Do It': How Racist-Tech led an African-American man sleeping in a filthy cell

We've heard of Fintech, maybe even Regtech, but have we heard of Racist-Tech?

In the past few weeks, the US sees the largest protests in its history. I am not referring to the protests where armed protestors show up to state-capitals without much reaction. Rather, these are the protests that were in response to the death of George Floyd. George Floyd who died after a police officer kneeled on his neck (with his hands in his pocket) for eight minutes and forty-six seconds. These protests, in contrast, have been met with a strong reaction.

A related incident occurred a few months before Mr. Floyd lost his life.

As reported in NPR, Robert Julian-Borchak Williams was picked up by police by January 2020 and when he got to the station, he was surprised to the lack of resemblance between him and the pictures of the suspect.

The officer's response? "So I guess the computer got it wrong, too." 

Regardless, "Williams was detained for 30 hours and then released on bail until a court hearing on the case, his lawyers say."

(For more on the story, check out this video)

The story is chilling, to say the least.  The knee jerk reaction is to think of Skynet and dark AI. But is that really what's happening here?

The social unrest speaks to how the desegregation struggles of the 1960s have not totally succeeded. The challenge is that racism is systemic. Within the institutions that hold society together, the gothic systems that existed in the 1950s somehow still exist until today. Sure, it's illegal for prosecutors, judges and cops to be racist. But then how do we explain the treatment of George Floyd and Robert Williams? Is there is no overall monitoring provisioning to ensure that the desired equality is achieved? For example, good monitoring controls over a system would assess the outcomes to see if the desired outcomes are achieved. There was a case that tested this idea. In McClesky v Kemp, where the defence team provided Dr. Baldus's study that statistically proved that the African American is 4.3 times more likely to get the death penalty, the "big data" analysis was rejected and Warren McClesky was put to death by the state. (And yes it controlled for 35 non-race variables).

In other words, data analysis shows there actually is a problem. However, the courts essentially denied this reality and pretended everything is okay.

What does this have to do with Racist Tech?

It means that the systems and the data are biased. Racist Tech will naturally grow out of such systems. AI and predictive policing models that use data from the court system - also pretending everything is okay - will inevitably lead to people like Mr. Williams getting caught up in the criminal justice system. Compared to George Floyd he only had to spend 30 hours in a filthy cell. But during that time he would have no idea whether it was going to be 30 hours or 30 months, given how long it takes to exonerate the innocent.

I was once asked at a conference whether we can look forward to a future where AI takes over. My response was to point out the real issues is with the human that run the technology.  If I had to answer that question today, I would simply ask them to call Mr. William who knows that the nightmare scenario is here already.

Author: Malik Datardina, CPA, CA, CISA. Malik works at Auvenir as a GRC Strategist that is working to transform the engagement experience for accounting firms and their clients. The opinions expressed here do not necessarily represent UWCISA, UW, Auvenir (or its affiliates), CPA Canada or anyone else.

Big Data and Predictive Policing: Can algorithms become racists?

Interesting article on Forbes by Thomas Davenport on Big Data. The articles discusses how various government, including Canadian Public Safety Operations Organization (CanOps), have used big data tools for "situational awareness". These systems draw on myriad sources of data to give users (e.g. law enforcement) the information they need to deal with a particular situation.

Here are a few points that I thought were worth noting:

Government is making strides in big data: We often think of Amazon, Google and other tech-giants as key users of this data. However, as the Davenport points out that the government is using this technology to assist with decision making. However, whether this is something that should be celebrated remains to be seen (see predictive policing below)

Privacy versus Value trade-off: He talks about how CanOps use of MASAS, the Multi-Agency Situational Awareness System, is limited by the filtering of sensitive information: "breadth of MASAS is noble, but it seems to limit its value. For example, as the CanOps website notes, because agencies are reticent to share sensitive information with other agencies, all the information shared was non-sensitive (i.e. not terribly useful)." It seems that this continues to be a theme that we had noted in back a couple years when discussing a similar trade-off the companies face when dealing with big data. As I noted in this post:

"privacy policies require the user to consent to a specific uses of data at the time they sign up for the service. This means future big data analytics are essentially limited by what uses the user agreed upon sign-up. However, corporations in their drive to maximize profits will ultimately make privacy policies so loose (i.e. to cover secondary uses) that the user essentially has to give up all their privacy in order to use the service."

Consequently, there still needs to be a solution as to how privacy can be respected but organizations can use the data they have collected to make better decisions.

Predictive Policing is an emerging reality: The sci-fi movie, Minority Report, paints a future where law enforcement arrests people before they commit crimes.


That future seems to be well on its.  Davenport mentions how "predictive policing" was introduced in 2014 to the NYPD.  He also mentions how much data is being collected by the police:

"It collects and analyzes data from sensors—including 9,000 closed circuit TV cameras, 500 license plate readers with over 2 billion plate reads, 600 fixed and mobile radiation and chemical sensors, and a network of ShotSpotter audio gunshot detectors covering 24 square miles—as well as 54 million 911 calls from citizens. The system also can draw from NYPD crime records, including 100 million summonses."

The idea of predictive policing was also raised in the book,  Big Data: A Revolution That Will Transform How We Live, Work, and Think, which I had explored in a multi-blog post series (click here for the first installment).

Andrew Guthrie Ferguson, Law professor UDC David A. Clarke School of Law, wrote an article on how that predictive policing is something that has not be really sorted in out in terms of legality. He notes:

"The open question is whether this big-data information combined with predictive technologies will create “predictive reasonable suspicion“ undermining Fourth Amendment protections in ways quite similar to the stop-and-frisk practices challenged in federal court.

In two law review articles I have detailed the distorting effects of predictive policing and big data on the Fourth Amendment and have come to the conclusion that insufficient attention has been given at the front end to these constitutional questions. New York has the chance now to address these issues before the adoption of the technology and should be encouraged by the same civil libertarians and ordinary citizens who challenged the stop and frisk policies."

His commentary highlights another limitation: big data predictions are biased based on how the data is collected. The stop and frisk policies he refers to disproportionately targeted minorities. Furthermore, policing is more focused on poor, black/hispanic neighbourhoods. Michelle Alexander documents in her book, The New Jim Crow, how this happens:

"Alexander explains how the criminal justice system functions as a new system of racial control by targeting black men through the “War on Drugs.” The Anti-Drug Abuse Act of 1986, for example, included far more severe punishment for distribution of crack (associated with blacks) than powder cocaine (associated with whites). Civil penalties, such as not being able to live in public housing and not being able to get student loans, have been added to the already harsh prison sentences."

Consequently, if the data by law enforcement is used to predict crime that essentially the targeting of minorities will continue to target such groups given that it is based on biased data. 

Technology often is seen to be a silver bullet for problems. However, we need to keep in mind that it is vulnerable to the human element that makes it. Given Microsoft's recent faux pas of accidentally allowing an AI avatar to become a Nazi, it is something that should actively be considered in the systems that are built to police and govern.