Income inequality is one of the main problems facing the economy. The legislature uses taxes as an effective tool to address this: Put simply, the state collects money from people based on their income and redistributes it directly or indirectly. Tax policy is a big challenge. Economists have struggled to make the most of it, but to this day it remains an open problem. The economic methodology is constrained by counterfactual data and simplified behavioral models and offers limited opportunities for policies experimentation. Machine learning-based economic simulation can provide a powerful framework for designing policies and mechanisms that can help overcome these limitations. To this end, researchers from Salesforce, Harvard University, and You.com have tried to shape optimal economic policy through two levels of amplification of deep reinforcement learning.
AI Economist using Deep Reinforcement Learning
The optimization of policies poses a challenge in the design of mechanisms. The government aims to find a policy in which the rational behavior of the economic agents concerned produces the desired social outcome. However, theoretical approaches to policy making are limited by analytical traceability. They fail to capture the complexity of the real world. While machine learning and computational techniques for designing automated mechanisms show promise in overcoming existing challenges, there has not been a comprehensive computational approach to policy making. There is a need for solving a highly non-stationary, two-level, Sequential decision problem in which all actors learn: While economic actors learn rational behavior that maximizes utility, government learns to optimize its goal through policy options.
The authors of the study “Optimal Economic Policy Design via Two-level Deep Reinforcement Learning” present a new framework: AI Economist, which combines machine learning and AI-supported economic simulation to master today’s challenges. Specifically, this technology is based on AI-controlled economic simulations and two-stage reinforced learning as a new paradigm for the design of economic policy. This study shows that AI-driven simulations capture features of real-world economies. It does not need hand-crafted behavioural rules or simplifications for analytical tractability. The researchers used a single step economy and a multiple-step, micro-founded economic simulation called Gather-Trade-Build. This function has several heterogeneous economic actors in a two-dimensional spatial environment. Gather-Trade-Build involves trading between brokers and simulating the economy over longer periods of time.
AI Economist uses deep reinforcement learning on two levels: individual actors within the economy and on the level of the social planner. The agent and social planner use deep neural networks to implement their policy model. The two-tier RL is natural in many contexts, including mechanism design, main problem, and regulatory systems with unethical incentives. This system compares the performance of billions of inexpensive designs.
AI Economist uses entropy-based regularization and learning curricula to solve the two-tier problem by providing a manageable and scalable solution. This approach stabilizes training using two assumptions: the agent and social planner should be encouraged to explore, and co-adaptive agents should not be faced with high operational costs that discourage exploration while learning.
This approach offers the following advantages:
- Conceptually, it takes into account the actors who are dependent on economic policy, i.e. it is not criticized by Lucas. According to Lucas’ criticism, the effects of economic policy change cannot be predicted on the basis of the relationships observed in the data history alone.
- The use of reinforcement learning enables rational agent behavior.
- Since the simulation framework is flexible, supports a configurable number of agents and offers various options in economic processes.
- The designer can choose any policy objective and it does not have to be analytically treatable or differentiable.
- Use of RL does not require any knowledge of simulation or economic theory.
Previous Solutions
This isn’t the first time Salesforce has toyed with the idea of AI Economist. In fact, the team introduced it in 2020. This version used reinforcement learning for tax research to provide a simulation and data-driven solution. They are designed to simulate how real people can react to different taxes. In the simulation, each agent made money collecting resources, trading and building houses. This is where agents maximize their utility by adapting to movement, construction, and business behavior. AI Economist optimizes taxes and subsidies to advance global goals.
