In 2016, Uber paid $148 million to settle the investigation on a data breach that exposed the personal information of over half a million drivers. In 2020, Google was fined $57 million for GDPR violation. The rise of on-device machine learning, coupled with the growing concerns of data privacy, has nudged developers and researchers towards techniques such as federated learning–a collaborative learning method that operates without exchanging users’ original data.
The stringent GDPR makes data sharing among European organisations challenging. Meanwhile, federated learning systems (FLS) have shown promise with good predictive accuracy while complying to privacy rules. According to Li et al., federated learning systems are gamechangers like deep learning frameworks such as PyTorch and TensorFlow.
As illustrated above, the number of related papers in FL has increased rapidly and has reached about 4,400 last year.
Federated Learning leverages techniques from multiple research areas such as distributed systems, machine learning, and privacy. FL is best applied in situations where the on-device data is more relevant than the data that exists on servers. However, FLS faces various challenges such as effectiveness, efficiency, and privacy. FL enables multiple parties to jointly train a machine learning model without exchanging the local data. According to Li et al.,se FLS is being adopted in various domains:
A typical Federated Learning Protocol according to Google AI:
- Federated learning servers are called by devices
- Model checkpoint from storage is read by the servers.
- Models are sent to the select devices.
- On-device model training followed by server update.
- Server aggregates these updates into a global model and writes them into storage.
On-device FL
Federated learning provides a decentralised computation strategy to train a neural model. Modern day mobile devices churn out swathes of personal data, which can be used for training. Instead of uploading data to servers for centralised training, phones process their local data and share model updates with the server. Weights from a large population of mobiles are aggregated by the server and combined to create an improved global model. The distributed approach has been shown to work with unbalanced datasets and data that are not independent or identically distributed across clients.
On-device machine learning comes with a privacy challenge. Data recorded by cameras and microphones can put individuals at great risk in the event of a hack. For example, apps might expose a search mechanism for information retrieval or in-app navigation.
Federated averaging was implemented by researchers from University of Kyoto in practical mobile edge computing (MEC) frameworks by using an operator of MEC frameworks to manage the resources of heterogeneous clients. Both distributed deep reinforcement learning (DRL) and federated learning were also adopted in mobile edge computing systems. The use of DRL and FL has the potential to optimise the mobile edge computing, caching, and communication. FL has also been performed on resource-constrained MEC systems, where the researchers address the problem of how to efficiently utilise the limited computation and communication resources at the edge. A technique called FedGKT was proposed where each device only trains a small part of a whole ResNet to reduce the computation overhead. Using federated averaging, the researchers implemented many machine learning algorithms including linear regression, SVM, and CNN.
For natural language processing
Companies like Google use Federated Averaging techniques in its smartphone keyboard for text prediction. FL was applied in mobile keyboard next-word prediction. Federated averaging method was used to learn a variant of LSTM called Coupled Input and Forget Gate (CIFG). According to the researchers, the FL method can achieve better precision recall than the server-based training with log data.
