The Trust Problem in AI

The forecast for AI safety and security threats is sobering. In the largest-ever study of AI and machine learning specialists, between 37.8% and 51.4% of all respondents assigned at least a 10% likelihood to possibilities as terrible as human extinction, with 48% of net optimists putting that probability at 5%. It’s difficult to imagine such negative assessments being accepted for any other technology currently in widespread use. There are, of course, less catastrophic risks: hostile use cases for AI tools in cyberattacks, being “jailbroken” to execute illegal commands, and so on. In the same survey, issues like the possibility of AI becoming jailbroken were assigned a reasonably high chance – the majority of respondents ranked it “likely” or “very likely” — even in 2043.

Regulations are essential for reducing these hazards once more. In order to detect vulnerabilities, generative AI models above a specific risk threshold are required under the EU laws and the White House executive order to publish the findings of simulated “red-team” attacks. However, it’s unclear if imposing such restrictions will actually reduce risk. Even worse, some policies, like those requiring red-teaming, can be promoting “security theater.” Few guidelines exist for infallible red-teaming techniques and standards, and while regulations compel some transparency, it is difficult to verify that such efforts were thorough. Unless their products are part of a wider AI ecosystem, startups are unlikely to have the resources to perform this work internally or to vouch for externally sourced tests. Moreover, the initial financial burden may prevent businesses from doing this work altogether.

The most important lesson is that, contrary to what many experts think, AI safety and security threats cannot be completely eliminated in the near future. For the most crucial and life-and-death applications, such as national security and health care, it will be crucial to keep humans involved in the process to ensure that decisions are never fully automated. For instance, in extremely sensitive negotiations between nuclear-armed nations, agreements would have to ensure that decisions about launching tests or missiles remain in the hands of humans. Awareness and preparation will therefore be crucial.

The black box problem

Building trust requires transparency. When it comes to AI, that can mean giving people alerts when they are engaging with a model, explaining how it generated a specific result, and being aware of the information stakeholders require and providing it in a way that they can comprehend. The EU AI Act and other important regulations will enforce specific transparency standards, but the ongoing problem is that AI companies have incentives to minimize transparency in order to protect their intellectual property, maintain competitive advantage, guard against malicious hacks, and lower their exposure to copyright lawsuits. Because of this, artificial intelligence is frequently a “black box” whose purpose is unclear.

One of the features that makes open-source AI development appealing is its industry-led approach to transparency. However, this also has its limitations. Experts are unable to agree on what exactly qualifies as “open-source” because AI models require much too many inputs, including training data, code that preprocesses it and controls the training process, the model architecture, etc. Businesses create their own definitions using this uncertainty as a pretext to keep the training data, which includes “synthetic” data, hidden from the general public. Even businesses that support open-source models, like Meta, are gradually becoming less “open”; their Llama 2 model is significantly less transparent than Llama 1. Furthermore, the transparency score of the Stanford Center for Research on Foundation Models gives even the industry standard, Llama 2, a mere 54 out of 100. Businesses like IBM have offered their “factsheets” as tracking and transparency tools, but unaudited self-disclosures aren’t the best ways to establish credibility.

Regulations are anticipated to be a factor in reducing the dangers associated with black box systems once more. A legitimate regulatory enforceability, standards, auditing criteria, and reliable auditors would be necessary for regulations to force businesses to submit to external audits of their AI models and publicize the findings. A recent study conducted by Cornell University revealed that a New York statute that mandates businesses who use automated employment decision tools to audit them for gender and racial prejudice is ineffective. Although the AI Risk Management Framework from the National Institute of Standards and Technology exists, it is currently ineffective due to a lack of certification, standards, or an auditing methodology.

The lesson here is that although transparency will advance, AI’s “black box” issue will persist. To facilitate adoption, each application area will need to create activities focused on increasing transparency. For instance, as previously mentioned, “interpretability” of AI — that is, the capacity to comprehend the rationale behind an algorithm’s decision — with respect to radiological applications is an important and developing area of study to promote clinical practice and adoption and helps to foster confidence among radiologists.

The majority of users concur that it is imperative to make sure algorithms transcend mathematics and data and are accompanied with guidelines guaranteeing ethical norms, such as respecting human rights and values regardless of what the mathematics says. There have been multiple attempts to encourage AI developers to unite under widely recognized ethical standards. The Asilomar AI principles, for instance, support a number of concepts in the development and application of AI models, including “human values,” “liberty and privacy,” and “common good.” However, these kinds of initiatives face three challenges.

To start, not everyone adheres to moral standards. The two leading AI countries, the United States and China, have different ideas on what “liberty and privacy” means. In the United States, free speech is protected, while in China, free speech is seen as a threat to the “common good.” Pro-life and pro-choice organizations disagree on “human values,” even in the fiercely divisive and culture-war-ridden United States. AI should be decolonized, according to some, while others want it to be anti-“woke.”

Furthermore, the capabilities of apolitical transnational bodies are restricted. Consistent with its mandate, the UN has ethical AI standards, and UNESCO has convened businesses to pledge to develop more ethical AI. The UN has little influence because the private sector develops AI the majority of the time.

Third, the organizational incentives of AI businesses heighten the conflicts between ethics and other factors. For instance, political diversity in ethical monitoring is necessary given the left-leaning nature of the workforce. Practically speaking, this is challenging: Google’s attempts to form an AI ethics advisory group were derailed when staff members objected to the choice of the president of the right-wing Heritage Foundation. The much-discussed boardroom drama at OpenAI involving Sam Altman, the abortive attempt to extract DeepMind from Google’s traditional business structure following its acquisition, and the collapse of Stability AI’s leadership serve as further frequent reminders of the struggle for supremacy among the leading AI firms: time and time again, profit margins triumph over AI for the sake of the “common good.”

The takeaway from this is that ethical conundrums are situation-specific and inevitable in AI systems; they become especially critical when they lead to conclusions that are harmful or exclusive. Human interaction will be crucial, including with groups constituted as oversight or governance boards and groups of outside observers.

AI is prone to bias due to a variety of factors, including usage context, individual training participant restrictions, and biased or incomplete training data. When artificial intelligence (AI) is employed in crucial applications, such as when lenders are discovered to be even more likely to deny house loans to people of color, it can undermine public faith in the models. There are a number of mitigating measures that can be implemented, including expanding the AI talent pool, increasing the diversity of data sources, training AI developers to identify bias, applying fairness requirements to AI models, and more.

AI might never be consistently bias-free despite these corrective actions for a number of reasons. AI technologies can exhibit unexpected biases as a result of their limited exposure to real-world data, as they are taught in controlled contexts and may come across novel application environments. Moreover, the procedures for examining the existence of prejudice are challenging. It can be challenging to come to an agreement on when a decision is truly impartial because definitions of bias and unfairness can differ greatly among environments as diverse as the West, China, and India. For example, the concept of “fairness” lends itself to 21 alternative definitions. The creation of inaccurate ahistorical images by Google and Meta provides a clear example of the risks involved in even “unlearning” biases, since it may generate new, unforeseen associations that the AI model learns, worsening the situation overall. Additionally, artificial intelligence models incur the risk of running out of fresh, high-quality information to train on and of neutralizing biases resulting from scarce or subpar samples.

The lesson here is that biases and constraints in the form of data or trainers working within human bounds will inevitably be incorporated into AI models during training. It will be crucial to use common sense, alertness, and quick corrective action before they cause harm.

When input is modified slightly but not meaningfully in specific circumstances, AI judgments may alter significantly, resulting in errors and minor to major variations in results. When an AI-assisted automobile drives past a stop sign due to a little impediment, for example, autonomous vehicles can be trusted for many tasks but occasionally they do not perform as intended. Academic research on the “stability” of AI has discovered that, aside from basic issues, it is mathematically impossible to create AI algorithms that are universally stable. This is in contrast to the fact that AI models are continuously improved upon by the addition of training data, the improvement of testing methods, and ongoing machine learning. Thus, when there is even a small amount of noise in the incoming data, we can never be certain that AI is making wise conclusions.

The takeaway from this is that, outside of training datasets, artificial intelligence systems are subject to minute variations. In these cases, having human workers on hand to make manual corrections or overrides will be essential.

AI hallucinations have led to models acting strangely, such as confessing to be in love with their users or saying they have spied on staff members of the company. Numerous AI manufacturers have created a variety of mitigating methods. IBM suggests, for instance, employing consistent output through the use of data templates, defining explicit limits on the use of the AI model, utilizing high-quality training data, and conducting ongoing testing and improvement. There is always a potential that hallucinations will manifest, regardless of the steps taken, according to research that indicates a statistical lower-bound on hallucination frequencies. As is also rational for probabilistic models, hallucination incidences can be predicted to decrease but are never completely eradicated, irrespective of the quality of the model design or dataset.

The lesson is to never rely on or use any generative AI model output in public, especially when it comes to high-stakes situations like legal documentation, without having it thoroughly examined by qualified experts. This can help prevent scenarios like the one in which ChatGPT, in the process of producing a legal brief, invented six fictitious court cases complete with fabricated statements and citations.

Artificial intelligence is capable of actions that humans cannot predict. In addition to having blind spots and making mistakes that developers are unable to decipher, models may include training data that is out of date for the environment in which they are being used. Though they are capable of accurately identifying objects, image recognition models can make utterly incorrect decisions. Although training the models on new datasets continuously reduces the likelihood, there will always be information beyond the model’s field of view, and the hazards resulting from these missing parts compound and can take unexpected turns.

The lesson is that blindly applying AI, which has blind spots of its own, is a surefire way to go wrong; human judgment must always be included in decision-making processes, and application context must be understood.

Wage increases should occur quickly in economies with increasing productivity. Though estimates of AI’s productivity impact differ, McKinsey has calculated an optimistic 3.3% annual increase in productivity by 2040 as a result of generative AI. AI will quadruple everyone’s productivity, according to former Google CEO Eric Schmidt. The head of the US Federal Reserve, Jerome Powell, is less optimistic about the immediate effects of AI on productivity.

Looking to history is a logical strategy to gain a better understanding of the influence. Unfortunately, history doesn’t offer much advice in this situation. It’s actually the case that as early digital technologies were introduced, U.S. worker productivity growth decreased. Even when it doubled after the World Wide Web’s inception in the late 1990s, the increase was fleeting. There were further surges in 2009 during the Great Recession, following the pandemic’s start in 2020, and up to 4.7% in the third quarter of 2023—too early to be linked to AI. This provides insufficient data to make an informed opinion about how AI will affect wages and productivity across economies.

Individual companies, on the other hand, are more optimistic, which could result in job losses as AI replaces human labor. However, that would imply that AI would raise the earnings of those in employment while causing wage losses for those whose jobs are eliminated, hence exacerbating social inequality. In response to these concerns, some researchers believe that generative AI can lessen inequality by providing lower-skilled workers with opportunities for upward mobility. More relevant historical data is that wage inequality tends to rise in nations where businesses have already resorted to automation; Black and Hispanic workers are overrepresented in the 30 occupations most exposed to automation, and underrepresented in the 30 occupations least exposed; women are expected to be disproportionately impacted by automation, with 79% of working women in occupations vulnerable to labor displacement by generative AI, compared to 58% of working men.

The overarching lesson is that the adoption of AI is hampered by the prospect of job losses and rising societal inequities. Even acknowledging AI adoption can be troublesome. UPS’s greatest layoff in history was caused by AI replacing humans, according to the CEO on an earnings call, although a spokeswoman later refuted any link between layoffs and AI. Clearly, the CEO wanted to indicate to investors that the company was implementing AI to gain from cost savings from reduced personnel, but it also had a negative public relations impact; this suggests that the impact on jobs provides friction in fully embracing AI. With various stakeholder concerns to consider, firms will hopefully adopt AI with caution.

AI’s contribution of global data center power consumption is predicted to rise to 10% by 2025. By 2027, AI’s utilization of data centers might save the equivalent of half of the water utilized in the United Kingdom each year. Artificial intelligence requires increasingly powerful semiconductors, which contribute to one of the fastest-growing waste streams. Neither of these trends is showing signs of slowing. Increased usage of generative AI, particularly for image production, will exacerbate the situation. According to one study, 1,000 photos created using Stable Diffusion XL produce as much carbon dioxide as a gas-powered automobile driving 4.1 miles.

One key concern is that AI-aided applications may replace other environmentally costly activities, hence reducing emissions and resource use. Nonetheless, it is important to be mindful of its implications. Specific efforts, such as the Artificial Intelligence Environmental Impacts Act of 2024, which was sponsored in the United States Senate, are commendable but will be difficult to implement due to the lack of standards for measuring and validating AI-related emissions. Another risk-mitigation strategy is to power new data centers with renewable energy, but demand for them is expanding too quickly to be totally renewable. Even with recycling programs in place, only 13.8% of reported electronic waste is formally collected and recycled, with an estimated 16% falling outside the formal system in high and upper-middle income nations. For the near future, AI’s detrimental environmental impact is unavoidable.

The lesson here is that, just as some companies, such as the fossil fuel industry or gas-guzzling vehicle makers, have lost customer trust due to their environmental impact, AI may face comparable concerns. Human judgment is required to determine if the benefits of, instance, adding AI advances into products with acceptable enough alternatives outweigh the environmental costs.

Despite the tremendous priority placed on AI by political leaders, its development is driven by industry. The reasons are structural: AI development necessitates a number of crucial inputs, including talent, data, processing capacity, and finance, all of which are more readily available in the private sector. Furthermore, these resources are concentrated in a few companies.

The AI value chain consists of two primary concentration points. A few active inventors creating AI models rely on a few huge corporations for important inputs. Nvidia, Salesforce, Amazon, Google, and Microsoft are the most active investors in the main AI pioneers, while Meta is a major supplier of open-source models.

Aside from money, AI model makers rely on Nvidia for graphics processing units and cloud providers like Amazon and Microsoft to operate the models, while Google, Meta, and Microsoft integrate AI to defend their main products. Even with a more competitive layer of AI apps and services suited to specific use cases, the industry’s basis will undoubtedly remain centralized. Users’ suspicion of Big Tech’s control will be exacerbated as technology becomes increasingly AI-intensive.

The typical effort to alleviate industry concentration dangers, namely regulatory monitoring, has arrived late. The Federal Trade Commission has only lately begun an investigation into the growing concentration issue. Meanwhile, the trends continue: since the probe began, Microsoft, the largest investor in OpenAI, has acquired the leading team from Inflection, and Amazon has spent $2.75 billion in Anthropic. OpenAI, Inflection, and Anthropic are the three most notable AI innovators in the United States right now.

The lesson is that concentration of power in a few firms undermines trust because consumers feel trapped, are concerned about overpaying, and are concerned about their data being seized by large firms who can abuse it in other areas.

Trends indicate that governments around the world will increasingly employ AI and related capabilities to impose control over individuals. Furthermore, the proportion of people living in political situations classified as “free” by Freedom House has decreased over the last 15 years. According to Freedom House, global internet freedoms have been declining for 13 years in a row, and AI has helped to accelerate that decline in a variety of ways, including spreading state propaganda, enabling more efficient censorship, creating citizens’ behavioral profiles, and developing predictive analytics and surveillance. Consider that at least 75 of the world’s 176 countries use AI technologies for monitoring, including 51% of advanced democracies. The potential of abuse of power increase when governments gain access to citizen data, particularly with the advent of digital identification systems. Concerned professionals have recommended various potential checks and balances, but they have not been broadly adopted.

The bigger lesson is that concerns about governmental overreach may lead to the rejection of AI use, even when it can be helpful to society if handled with caution. Testing citizens’ willingness to accept tradeoffs will be crucial to ensuring their trust in governments utilizing AI. Consider the use of facial recognition technology by police: towns like San Francisco have outlawed it.