Machine learning’s boundaries are being put to test by scientists, and subsequently, the amount of time, energy, and costs needed for training increasingly complex neural network models are also on the rise.
A new branch of artificial intelligence known as analog deep learning promises faster computation while using a fraction of the energy.
Programmable resistors are the basic elements in analog deep learning similar to transistors which serve as the key element for digital processors. Researchers can create a network of analog artificial “neurons” and “synapses” that execute computations similar to a digital neural network by repeating arrays of programmable resistors in complex layers.
After that, the network can be trained to perform complex AI tasks such as image recognition and natural language processing.
A multidisciplinary team of MIT researchers commenced testing the speed limits of a previously developed type of human-made analog synapse. In the fabrication process, they used a practical inorganic material that allows their devices to run 1 million times faster than previous versions, which is also about 1 million times faster than synapses in the human brain.
Furthermore, the resistor’s inorganic material makes it extremely energy-efficient. The new material, unlike the materials used in the previous version of their device, is compatible with silicon fabrication techniques. This advancement has enabled the fabrication of nanometer-scale devices, which may open the doors for integration into commercial computing hardware for deep-learning applications.
Using that vital insight, and the much powerful nanofabrication techniques available at MIT.nano, we were able to put these pieces together and determine that these devices were intrinsically very quick and also operated at reasonable voltages, stated Jesús A. del Alamo – senior author and the Donner Professor in MIT’s Department of EECS (Electrical Engineering and Computer Science). This work has made the devices ready for future applications.
The device operates by electrochemically inserting the smallest ion, the proton, into an insulating oxide for modulating its electronic conductivity. Since we are working with very thin devices, this ion’s motion can be accelerated with the help of a strong electric field, and can be pushed to the nanosecond operation regime, clarifies Bilge Yildiz – senior author, and the Breene M. Kerr Professor in the departments of Nuclear Science and Engineering as well as Materials Science and Engineering.
The biological cell’s action potential surges and drops with an interval of milliseconds, because the stability of water constraints the voltage difference of approximately 0.1 volts, states Ju Li – senior author and the Battelle Energy Alliance Professor of Nuclear Science and Engineering as well as professor of materials science and engineering.
Up to 10 volts is applied here across a special solid glass film of nanoscale thickness conducting protons, without damaging it permanently. The field’s strength and the speed of the ionic devices are directly proportional.
These programmable resistors are responsible for the enormous increase of speed at which a neural network is trained, while exceptionally decreasing the cost and energy required to perform that training. This can be useful for scientists to develop deep learning models quickly, which could then be used in applications such as self-driving cars, fraud detection, and medical image analysis.
With an analog processor, you would not be training networks anymore that others are working on. Instead, you will be training networks with unmatched complexities that others can’t afford to, and as a result, you can surpass everyone. Simply put, this is not a faster car, but a spacecraft, according to Murat Onen – lead author and MIT postdoc.
Frances M. Ross, the Ellen Swallow Richards Professor in the Materials Science and Engineering Department, postdocs Nicolas Emond and Baoming Wang, and Difei Zhang, an EECS graduate student, are among the co-authors. The study was published in Science today.
Accelerating deep learning
There are two major reasons for analog deep learning being faster and more energy efficient than digital. The first is analog deep learning performs computation in memory, so massive amounts of data are not transferred from memory to a processor. The second is that analog processors also perform parallel operations. Since all computation occurs simultaneously, an analog processor does not require more time to complete new operations as the matrix size grows.
A protonic programmable resistor is the key component of MIT’s new analog processor technology. These nanometer-sized resistors (one nanometer equals one billionth of a meter) are arranged in an array similar to a chess board.
Learning occurs in the human brain as a result of the strengthening and weakening of synaptic connections between neurons. Deep neural networks have long used this approach, in which network weights are programmed using training algorithms. In the case of this new processor, varying the electrical conductance of protonic resistors allows for analog machine learning.
The proton movements regulate the conductance. When conductance needs to be increased, more protons are pushed into a resistor channel, and for decrease, protons are removed. An electrolyte (similar to a battery electrolyte) that allows protons, but blocks electrons is used to achieve this.
For developing a super-fast and highly energy-efficient programmable protonic resistor, the researchers explored different materials for the electrolyte. While organic compounds were used by other devices, Onen concentrated on inorganic phosphosilicate glass (PSG).
PSG is fundamentally silicon dioxide – a powdery desiccant material available in tiny bags that accompanies the new furniture in a box for removing moisture. It is being researched as a proton conductor for fuel cells under humidified conditions. It is also the most commonly used oxide in silicon processing. To make PSG, a trace amount of phosphorus is added to silicon to give it special proton conduction properties.
Onen proposed that an optimized PSG could have high proton conductivity at room temperature without requiring water, making it an ideal solid electrolyte for this application. He was correct.
Surprising speed
PSG allows ultrafast proton movement by containing a plethora of nanometer-sized pores whose surfaces provide proton diffusion paths. It can also withstand extremely powerful, pulsed electric fields. This is critical, according to Onen, because increasing the voltage on the device allows protons to move at blazing speeds.
The speed was certainly unexpected. In order to avoid turning devices into ash, we would not normally apply such extreme fields across them. However, protons ended up moving at incredible speeds across the device stack, specifically a million times faster than before. Because of the small size and low mass of protons, this movement causes no harm. It’s almost as if you’re teleporting, he says.
Under such extreme conditions, the nanosecond timescale implies that we are close to the ballistic or even quantum tunneling regime for the proton, Li adds.
The resistor can run for millions of cycles without malfunctioning since the protons do not harm the material.
This new electrolyte allowed them to create a programmable protonic resistor that is a million times faster than their previous device and can operate efficiently at room temperature, which is critical for incorporating it into computing hardware.
Due to PSG’s insulating properties, there is almost no passage of electric current through the materials as protons move. Onen adds that this makes the device extremely energy efficient.
The researchers intend to re-engineer these programmable resistors for high-volume manufacturing now that they have demonstrated their effectiveness, according to del Alamo. They can then investigate the properties of resistor arrays and scale them up to be embedded in systems.
Simultaneously, they intend to investigate the materials in order to eliminate hindrances that limit the voltage required to efficiently transfer protons to, through, and from the electrolyte.
Beyond analog deep neural networks, another exciting direction that these ionic devices can allow is energy-efficient hardware to reproduce the neural circuits and synaptic plasticity rules deduced in neuroscience. We have already begun such collaboration with neuroscience, which is supported by the MIT Quest for Intelligence, Yildiz adds.
We will need to collaborate more in order to innovate in the future. The road ahead will continue to be difficult, but it will also be exciting, del Alamo says.
Intercalation reactions, like those found in lithium-ion batteries, have been extensively investigated for use in memory devices. This work illustrates that memory devices based on protons deliver splendid and surprising switching speed and endurance, says William Chueh, an associate professor of materials science and engineering at Stanford University who was not associated with the study. It establishes the groundwork for a new class of memory devices to power deep learning algorithms.
This research represents a significant advancement in biologically inspired resistive-memory devices. These all-solid-state protonic devices are based on extreme atomic-scale control of protons, identical to biological synapses but at orders of magnitude faster rates, states Elizabeth Dickey, the Teddy & Wilton Hawkins Distinguished Professor and head of Carnegie Mellon University’s Department of Materials Science and Engineering, who was not involved with the research.
I applaud the interdisciplinary MIT team for this exciting breakthrough that will enable next-generation computational devices.
The MIT-IBM Watson AI Lab is helping to fund this research.
