Machine learning is incredibly difficult. Many people assume that technology is almost omnipotent and can solve any problem with ease, but they often forget to consider the effort required to produce useful, robust, and unbiased results.
Often times, the time it takes to implement machine learning is not based on developing the basics of machine learning because most of the basic structures are well resolved and available in advance. The difficulty lies in preparing data and optimizing the model so that it can be used in the real world examples.
It can take months and millions of dollars to deploy truly effective machine learning models. The process typically involves incredibly long and complex calculations performed on supercomputers or huge computer networks distributed across cloud architectures. The same problem has to run through the system in an attempt to get better and better results as measured by key metrics.
I have seen these difficulties firsthand – twice. First, with a crypto hedge fund, which uses forecasting tools to create automated trading strategies, and second, with Google, which helps with data pipelines and machine learning annotations. These experiences helped me learn that it is impossible to produce good machine learning without setting the requirements in the requisite work and effort, just as it is difficult to hire fairly without paying attention to fairness.
Most of the time, simple solutions are more useful than investing in machine learning.
The perfectly optimal answer might be only marginally better than a trivial answer, which is often enough, especially when you factor in the cost of training a model, it might be much worse than having a human make an educated guess.
However, for very difficult problems where slightly better answers can have a significant impact, machine learning is an invaluable tool that can have a massive impact on the overall results. Machine learning is an umbrella term for a large variety of different strategies which would work with different levels of success for different problems.
For example, genetic algorithms are a popular strategy that works well in a variety of circumstances. During each iteration, all candidates are tested for success. This approach mimics Darwinian evolution by eliminating the most successful candidates (to be defined per problem) out of a large group and the creation of new candidates that are combinations of these.
From a technical point of view, each individual candidate is evaluated for hisuristic strength or suitability in each iteration or epoch. For example, imagine a baker trying to come up with a recipe for the perfect brownie. In their first batches, they randomly modify the ingredients. After baking these slightly different brownies, they taste great in every batch. Then they take some of the best brownie recipes and mix them together to make more recipes and repeat this until the baker is done with delicious brownie recipes.
Machine learning can be incredibly biased if not controlled.
This evolutionary strategy has one big flaw that is often problematic: diversity trends approaching zero. In each generation of the system, the most successful candidates win more often, making them more likely to be among the selected elements for the next generation. This can be amplified over time to reach more and more similar candidates.
Lack of biodiversity in ecosystems is widely seen as problematic, and AI is prone to similar problems. For example, to automate the labor-intensive hiring process, Amazon accidentally developed a failed resume evaluator that heavily favored men and punished college graduates with the word “women” by women and club members. How did it happen? Algorithms, especially AI-based solutions, are completely dependent on provided data in order to train them. When the algorithm is trained on successful attitudes composed mostly of men, these biases are reproduced in the seemingly “objective” model.
These diversity issues are so prevalent in genetic algorithms that reducing bias is one of the most valuable uses of time to produce better results. We can ameliorate genetic overfitting by introducing randomness and diversity into the system, giving random items a chance to be more prominent for a moment and see if they produce quality results, or by introducing a completely diverse new item to give it a chance to dominate.
We know how to reduce bias in artificial intelligence through rigorous empirical data, and we can draw these lessons and apply them to society by reintroducing diversity into our corporate and academic populations. Tech companies, and that number is even lower screen among women, who make up about 22% of all AI jobs. When this happens in machine learning, we introduce more diversity and make sure no group is underrepresented, mirroring the efforts needed for the people behind the screen.
Encouraging unique opinions prevents losing creativity.
For this reason, it is mathematically critical that we correct these imbalances and restore balance to the job market by making sure that people of all origins are represented in every job market. Gender and racial disparities need to be addressed and are inherently dangerous to the overall success of the system.
By following the lessons we learn from machine learning and genetic diversity, we can empirically ensure that it is critically important to have more diverse candidates for high-level positions, both to encourage diversity of opinions and to ensure that the next generation of leaders is not biased in favor of certain groups. These problems will not be easy to solve and will require great effort to evolve the general model towards a fully meritocratic model.
