From head-controlled Pacman to browser-based upload filters — here are some of the best uses of TensorFlow.js.
Building and training machine-learning models using a web-scripting language might seem ambitious, but in 2019 it’s perfectly feasible.
In a talk at the Google I/O conference last week Sandeep Gupta, product manager at Google, said TensorFlow.js could be used by developers to create new machine-learning models, as well as to run or retrain pre-trained models.
“We see a ton of use cases in the browser and it has a lot of advantages because the browser is super interactive, you have easy access to sensors, such as webcams and microphones, which you can then bring into your machine-learning models,” he said.
“Also we use WebGL-based acceleration, so if you have a GPU in your system you can take advantage of that and get really good performance.”
That said, not every developer is won over by TensorFlow.js, with some arguing the library still has significant limitations.
So what exactly is possible using TensorFlow.js? While the framework is still relatively new, only hitting 1.0 this year, Gupta said there has been “really good adoption and usage by the community”, and used his talk to demo some of the most interesting uses of TensorFlow.js.
You might not be crying out for a new way to play Pacman, but TensorFlow.js has made a novel spin on the classic arcade game possible.
After a quick calibration step, Gupta was able to control Pacman using head gestures, tracked by his phone’s camera, looking left to move left, right to move right, and so on.
“It’s a really fun way of interacting with the device, and the nice thing is that you can do a variety of things using web cams, using text, using speech, and have a very convenient way of sharing these applications without having to install anything,” he said.
The taxi and delivery company Uber uses machine learning to tackle a wide variety of problems at a very large scale.
Helping it achieve that is Manifold, a browser-based application that Uber uses to visualize and debug their machine-learning models and data pipelines.
AirBnB Identity Document Detection
Online property rental service AirBnB uses machine learning in the browser to stop people from inadvertently uploading sensitive information when adding a picture to their profile.
“When a user is trying to upload a profile picture to the AirBnB website, sometimes people accidentally use a driver’s license picture or a passport picture, which may end up containing personal sensitive information,” said Gupta.
“So AirBnB runs a machine-learning model client-side in the browser or on device, so that if you were to choose a picture which may have such sensitive information it will alert you before you upload that picture.”
Clinic.js provides a tool for sys admins and software engineers to profile server-side performance in a Node.js environment.
“This is a node.js-based application which is used for profiling node jobs or node processes and they’re using TensorFlow.js to look for anomalies, or spikes in CPU usage or memory consumption of these node applications,” said Gupta.
One of the main destinations for showing what is possible using TensorFlow.js is Creatability, Google’s creative labs team’s showcase for experiments using machine learning.
Gupta showed a machine-learning powered demo that allows a person to play a piano keyboard using head gestures.