When developing new models, Machine Learning researchers make numerous decisions. They determine the number of layers to be included in neural networks and the weights to assign to inputs at each node. According to Frank Hutter, head of the Machine Learning lab at the University of Freiburg in Germany, the result of all this human decision-making is that complex models are “designed by intuition” rather than systematically.
The aim of a developing field known as automated Machine Learning or autoML is to eliminate this guesswork. The idea behind this is to have algorithms make the decisions that are currently made by researchers when they design the models. Finally, these methods may make Machine Learning more accessible.
Automated Machine Learning is not new and has been around for almost ten years, yet researchers are still working on refining it.
A new conference in Baltimore last week was described by the organizers as the first global conference on the subject – wherein efforts to enhance the accuracy of autoML and streamline its performance were demonstrated.
There has been a surge of interest in the potential of autoML to simplify Machine Learning. Amazon and Google already provide low-code Machine Learning tools that make use of autoML techniques. If these techniques become more efficient, the research could be accelerated and more people could benefit from Machine Learning.
The idea is to reach a point where people can ask a question of their choice, point an autoML tool at it, and acquire the desired results.
According to Lars Kotthoff, a conference organizer and assistant professor of computer science at the University of Wyoming, this vision is the “holy grail of computer science.” You specify the problem, and the computer figures out a way to solve it— that’s all that you have to do.
However, researchers must first determine how to make these techniques more time and energy efficient.
What exactly is autoML?
Outwardly, autoML’s concept may appear redundant, since Machine Learning is also about automating the process of gaining insights from data. However, because autoML algorithms operate at a higher level of abstraction than the underlying Machine Learning models, depending solely on the models’ outputs as guides, can save time and computation.
Researchers can use autoML techniques on pre-trained models to gain new insights without wasting computational power on duplicate research.
For instance, at Fujitsu Research of America, research scientist Mehdi Bahrami along with his coauthors presented recent work on how a BERT-sort algorithm can be used with different pre-trained models to adapt them for the latest purposes. When trained on data sets, BERT-sort can figure out what is known as “semantic order”—for instance, when data on movie reviews is provided, it is aware that “great” movies rank higher than “good” and “bad” movies.
The semantic order after learning can also be deduced to classifying things such as cancer diagnoses or even text in the Korean language using autoML techniques, saving time and computation.
According to Bahrami, BERT requires months of computation and is very costly – it could cost a million dollars for generating that model and repeating those processes. So, if everyone wishes to do the same thing, then it is costly – it is not energy efficient, and it is bad for the world.
Despite the field’s promise, researchers are still looking for ways to improve the computational efficiency of autoML techniques. Methods such as neural architecture search, for instance, currently build and test various models to find the best fit, and the energy required to complete all of those iterations can be remarkable.
AutoML techniques can also be applied to Machine Learning algorithms that don’t contain neural networks, such as creating random decision forests or support-vector machines for classifying data. Many coding libraries are already available for people who want to integrate autoML techniques into their projects, indicating that research in those areas is progressing.
According to Hutter, a conference organizer, the next step is to use autoML to quantify ambiguity and address issues of trustworthiness and fairness in the algorithms. Standards for trustworthiness and fairness, in that vision, would be analogous to any other machine-learning constraint, such as accuracy. And autoML could detect and correct biases in those algorithms before they are released.
The hunt goes on
However, autoML still has a long way to go for something like Deep Learning. Data required for training Deep Learning models such as images, documents, and recorded speech are generally dense and complicated. It requires massive computational power to handle. The cost and time required to train these models can be expensive for anyone other than researchers working at well-funded private companies.
One of the conference’s competitions challenged participants to create energy-efficient alternative algorithms for neural architecture search. It’s a significant challenge because this technique has notoriously high computational requirements. It cycles through countless Deep Learning models to assist researchers in selecting the best one for their application, but the process can take months and cost more than a million dollars.
