Meta (formerly Facebook) announced this week the launch of the Bean Machine, a probabilistic programming system that apparently makes it easier to represent and learn about uncertainties in AI models. Bean Machine is available in early beta and can be used to discover unobserved features of a model through “uncertainty-aware” machine learning algorithms.
“[Bean Machine is] inspired by a physical device for the visualization of probability distributions, a precomputing example for a probabilistic system,” explained the meta-researchers behind the Bean Machine in a blog post. “On the Bean Machine development team, we believe that a system’s ease of use is the basis of its success, and we are committed to centering the Bean Machine design around a declarative philosophy within the PyTorch ecosystem.”
Modeling uncertainty
Deep learning models are generally considered to be overconfident, even if they make mistakes. Epistemic uncertainty describes what a model does not know because the training data was not adequate, while random uncertainty is the uncertainty that arises from the natural randomness of the observations. With enough training samples, the epistemic uncertainty decreases, but the random uncertainty cannot be reduced even if more data is provided.
Probabilistic modeling, Bean Machine’s artificial intelligence technique, can measure this type of uncertainty by taking into account the effects of random events in predicting the occurrence of future outcomes. Compared to other machine learning approaches, probabilistic modeling offers advantages such as uncertainty estimation, meaningfulness analysts who make use of it can understand not only the prediction of an artificial intelligence system, but also the relative probability of other possible predictions. , users can interpret why certain predictions were made, which could aid the model development process.
Built on Meta’s PyTorch machine learning framework and Bean Machine Graph (BMG), a custom C ++ backend, Bean Machine allows data scientists to write the math for a model directly in Python and BMG to do the probabilistic modeling work take over, derive the possible distributions for predictions based on the model statement.
The uncertainty measured by the Bean Machine can help to identify the limits of a model and possible sources of error. For example, uncertainty can reveal the margin of error of a house price prediction model or the confidence of a model designed to predict whether a new feature app will perform better than a previous feature.
To further illustrate the importance of the concept of uncertainty, a recent Harvard study found that displaying uncertainty metrics had a balancing effect on their resilience to AI predictions for both machine learners and non-experts. As simple as providing metrics, knowing the pitfalls can help protect people from the limitations of machine learning.
Bean Machine quantifies predictions “with reliable measures of uncertainty in the form of probability distributions … It is easy to code a rich model directly in the source code, [and because] the model corresponds to the domain, you can query the intermediate learned properties within the model,” continued Meta continued. We hope this makes using the Bean Machine easy and intuitive, whether it’s creating a model or changing your learning strategies.
Bean Machine became available on GitHub in early December.
