Benchmark Analysis of Representative Deep Neural Network Architectures

Benchmark Analysis of Representative Deep Neural Network Architectures was a 2018 paper that compared dozens of neural network architectures, looking at model accuracy (on ImageNet 1k), inference speed, FLOPs, memory usage, and parameter count.

It follows a 2016 benchmark, but expands it by

  • Including far more architectures
  • Running experiments on two different hardware platforms: one with a top-of-the-line consumer GPU (3840 cores, $1100), and one that is more modest ($400).

The paper makes the following claims:

Finally, all models use 224x224 images, except for NASNet-A-Large (which uses 331x331) and various Inception nets which use (229x229).

The figures below shows accuracy (using center-crop only) versus FLOPs for a single forward pass. The size of the circle shows the number of model parameters.