Optimizer Benchmarks
Comprehensive benchmark results from these research papers on different domains.
Research Citation Required
All benchmark results are derived from these research papers. Please cite these work when using these results:
"Benchmarking Neural Network Training Algorithms"
arXiv:2306.07179"Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning"
arXiv:2410.06373v1"Benchmarking Optimizers for Large Language Model Pretraining"
arXiv:2509.01440"Fantastic Pretraining Optimizers and Where to Find Them"
arXiv:2509.02046Benchmark Suites
CIFAR-100 Benchmark
Top-1 accuracy (%) across 20 backbone architectures with 20 optimizers
Setup: Representative vision backbones with 20 mainstream optimizers. Blue highlights denote top-4 results.
ImageNet Benchmark
Top-1 accuracy (%) across 10 backbone architectures with 20 optimizers
Setup: Representative vision backbones with 20 mainstream optimizers. Blue highlights denote top-4 results.
COCO Benchmark
Object detection and 2D pose estimation on COCO dataset
Setup: Transfer learning to object detection with RetinaNet and 2D pose estimation with TopDown on COCO. Different pre-training settings: 100-epoch (SGD, LARS), 300-epoch (RSB A2, Adan), and 600-epoch (RSB A1).
C4 Benchmark
LLaMA pre-training comparison on C4 dataset (60M-1B parameters)
Setup: LLaMA pre-training on C4 dataset with model sizes from 60M to 1B parameters. Validation perplexity (PPL)↓, GPU memory (Mem.)↓, and optimizer step time (s)↓ are reported. Lower is better for all metrics.
Want to Contribute Benchmarks?
Help us expand our benchmark suite! Submit your optimization results and contribute to the community knowledge base.