If you read about AI in the news, you will notice two schools of thought. One school of thought is utopian, proclaiming AI’s incredible power in everything from predicting quantum electron paths to driving a race car like a champion. The other school is dystopian, scaring us with crisis-ridden stories ranging from how AI could end privacy to self-driving cars that crash almost immediately. One school of thought is enraged by flaws, while the other is in denial.
However, neither extreme viewpoint accurately depicts our imperfect world. One of the basic rules of the universe is that nothing is perfect, said Stephen Hawking. Perfection, simply put, does not exist… You and I would not exist if there were no flaws.
Just as people are flawed, so are the AI systems we create. However, this does not imply that we should live in denial or give up. A third option is to accept the existence of imperfect AI systems while developing a governance strategy to actively manage their impact on stakeholders. Empathy, imperfection, and responsibility are three key dimensions of governance and AI impact.
Empathy
Empathy is the potential to acknowledge and share another person’s feelings. It is closely related to the theory of mind, which is the ability to understand others by attributing mental states to them. Empathy is important in the context of AI impact statements for developing an understanding of each stakeholder’s different needs and expectations, as well as the potential harms that an AI system could cause them.
It is an inherently human task to enter each stakeholder’s mind and feel empathy. Humans have mirror neurons, a type of neuron that fires both when the person acts and when the person observes another person performing the same action. As a result, the neuron “mirrors” the behavior of the other, as if the observer were acting. Humans, primates, and birds have all been shown to have such neurons.
However, cognitive biases that interfere with our ability to develop the theory of mind and assess risk and the consequences of decisions are inherently human. Attention bias, availability heuristic, confirmation bias, framing effect, hindsight bias, and algorithm aversion are some of the cognitive biases that apply in the AI impact assessment process.
The availability heuristic, for instance, may limit our ability to envision the full range of potential harms from an AI credit assessment system. We can easily imagine the harm of having a loan application unfairly denied, but what about the harm of being granted an unaffordable loan or the harm of inaccessibility to the system for people who do not have access to the internet, have language barriers, or have visual impairments?
Each stakeholder group is unique, with distinct expectations and risks. As a result, it is best to practice consulting with and involving the system’s diverse range of stakeholders. An AI impact assessment will meticulously document the harms experienced by each stakeholder.
Imperfection
EY warns that your AI may malfunction, become intentionally or unintentionally corrupted, and even adopt human biases. These failures have serious consequences for security, decision-making, and credibility, and may result in costly litigation, reputational damage, customer revolt, decreased profitability, and regulatory scrutiny.
“Anything that can go wrong will go wrong,” as Murphy’s law states. Nothing is flawless. There will be system failures, and there will be collateral damage. An AI system will, at some point, cause unintended and/or undeserved harm.
When an AI system behaves in a way that is inconsistent with its intended goal or purpose, it causes unintended harm. Unintended harm can occur in any software system due to software bugs, hardware or network failures, misspecification of requirements, incorrect data, privacy breaches, or malicious player actions. Aside from the usual software risks, an AI system may cause unintended harm when a machine learning algorithm learns incorrect behaviors from its training data.
When an AI system makes a decision, but the actual outcome differs from what the system predicted, it causes undeserved harm. “Prediction is difficult, especially when dealing with the future,” says an old Danish proverb. It is impossible to make perfect decisions without perfect knowledge. Even the most advanced AI systems cannot predict the future perfectly. Data scientists would be able to predict next week’s winning lottery numbers if they could!
Competing stakeholder needs are another source of unjustified harm. The fundamental economic problem is that human desires are constant and infinite, but the resources to satisfy them are finite. A design decision that maximizes value for one stakeholder may be detrimental to another. Similarly, a design choice that reduces unjustified harm for one stakeholder may increase unjustified harm for another.
You can’t avoid imperfection, but you can reduce the likelihood and severity of unintended consequences, and you can ethically balance the competing interests of stakeholders.
Responsibility
Humans must accept responsibility for their AI systems’ governance, behaviors, and harms.
An AI system is simply a type of computer system, a tool for humans to use. It is designed, built, and managed by humans to serve human goals. At no point during this process will the AI system be allowed to set its own goals or make decisions without human oversight.
Describe the potential harms to stakeholders and document your justification of the priorities and trade-offs that must be made between different stakeholders’ interests and values when documenting the system’s requirements. Explain why certain design decisions, such as fairness, harmful use of unjustified input data features, harmful use of protected features, loss of privacy, and other undeserved harms, are reasonable or unreasonable.
Your documentation should explain how the AI system helps to promote human values and human rights. Honesty, equality, freedom, human dignity, freedom of speech, privacy, education, employment, equal opportunity, and safety will be among these values and rights.
Build and design for failure tolerance, with risk management controls to mitigate potential errors and failures in the system’s design, construction, and execution. Assign responsibility for each risk to a specific employee or team, with clearly defined processes for risk mitigation and response to potentially harmful events.
Conclusion
AI impact assessments are more than just compliance documents in black and white. They are a human-centered approach to AI risk management. Human empathy is required for understanding the needs and harms of various stakeholders. Human judgment, values, and common sense are also required to balance competing stakeholder requirements.
However, software tools continue to have a place. Look for MLDev and MLOps tools that include:
- Guardrails that prevent and signal potentially hazardous design decisions
- Insights into model transparency and explainability for validating system behaviors
- Proactive notifications about ongoing system health
- Humble AI for gracefully dealing with handling errors and high-risk scoring data
