Thinkorithm

How Deep Learning Came Back

Neural networks were set aside more than once. Better training methods, large datasets and fast hardware brought them back — and changed the field.

Published · 4 min read

For much of the late twentieth century, neural networks were a minority interest in artificial intelligence. In the years after 2010 they became its dominant method. The turnaround came from several ingredients arriving at once.

Learning in layers

A neural network is a stack of simple units. Each unit combines its inputs using adjustable weights and passes the result on. A single layer can separate patterns only in limited ways. Networks with several layers are far more expressive, but they need a way to adjust the weights of the “hidden” layers in the middle.

In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams described such a procedure in Nature: back-propagation. It measures the difference between the network’s output and the desired output and works backwards through the layers, adjusting each weight to reduce that difference. The authors showed that hidden units trained this way come to represent useful features of the task.

Quiet progress

Through the 1990s, Yann LeCun and colleagues applied convolutional networks — designed to detect local patterns in images — to handwriting recognition. Their 1998 paper describes a commercially deployed system based on this approach that read several million cheques per day.

Much of AI’s public attention in those years went elsewhere. In 1997 IBM’s Deep Blue defeated the reigning world chess champion Garry Kasparov in a six-game match under standard tournament time controls, relying on hundreds of custom chess chips and large-scale search.

Three ingredients

  • Data. ImageNet, presented in 2009 by Jia Deng, Li Fei-Fei and colleagues, organised millions of images according to the WordNet hierarchy of concepts and later grew much larger. It gave researchers a common, large-scale benchmark.
  • Hardware. Graphics processors, originally designed for computer graphics, proved well suited to the repeated arithmetic that neural networks require.
  • Methods. Techniques such as “dropout”, which randomly switches off parts of the network during training so that it cannot simply memorise its examples, made large networks practical to train.

In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton combined all three. Their network, trained on graphics processors, won the ImageNet recognition challenge (ILSVRC-2012) with a top-5 error rate of 15.3 per cent — meaning the correct label was missing from the network’s five best guesses for 15.3 per cent of test images — well ahead of other entries. The result persuaded many researchers to adopt deep learning.

From games to science

In March 2016 DeepMind’s AlphaGo, which combined deep neural networks with tree search — systematically exploring possible sequences of future moves — won a five-game match against Lee Sedol, one of the strongest Go players of the decade, by four games to one in Seoul.

In 2017 a team at Google described the transformer, an architecture built around a mechanism called attention, which lets a model weigh how relevant each part of its input is to every other part. The transformer has since become the basis of many large language models.

Deep learning also reached the natural sciences. AlphaFold, developed at DeepMind, was assessed at the CASP14 protein-structure prediction experiment in 2020 and, according to the 2021 paper describing it, achieved accuracy competitive with experimental structures in a majority of cases.

What the comeback teaches

The history of neural networks shows that an idea can be sound long before the conditions exist to make it work. It is also a caution in the other direction: the success of one approach says little about how long that success will last, or where its limits lie. The present state of the field is described in What a Large Language Model Actually Does.

Sources

  1. Rumelhart, D. E., Hinton, G. E., Williams, R. J. “Learning representations by back-propagating errors”. Nature, 323, 533–536, 1986. doi:10.1038/323533a0
  2. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11), 2278–2324, 1998.
  3. IBM. The games that helped AI evolve. IBM History.
  4. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L. “ImageNet: A Large-Scale Hierarchical Image Database”. CVPR 2009. doi:10.1109/CVPR.2009.5206848
  5. Krizhevsky, A., Sutskever, I., Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. NIPS 2012.
  6. Google DeepMind. AlphaGo.
  7. Vaswani, A. et al. Attention Is All You Need. NeurIPS 2017.
  8. Jumper, J. et al. “Highly accurate protein structure prediction with AlphaFold”. Nature, 596, 583–589, 2021. doi:10.1038/s41586-021-03819-2
  9. ACM. Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award, 2019.
  10. Nobel Prize Outreach. Press release: The Nobel Prize in Physics 2024; Press release: The Nobel Prize in Chemistry 2024.