ImageNet saturation (2015–2017)
By 2015, algorithms topped 95% accuracy on ImageNet, leading researchers to create harder benchmarks like COCO and WinoGrande. The pattern repeated in coding: SWE-bench saturated, prompting new tests like Terminal-Bench 4.0.
Models appeared to solve old benchmarks while still failing on harder tasks.
Benchmark developers responded with new tests that separated robust models from overfitted ones.
SWE-2's 92.8 on Terminal-Bench 2.1 contrasts with its 27.3 on Terminal-Bench 4.0, highlighting how newer, harder benchmarks expose gaps that older ones mask.
