[EDIT: initial publication did not have the link to the original post, and was missing the footnotes. Sorry about that! Fixed now.]

[This has been lightly edited from the original post, eliminating some introductory material that LW readers won't need. Thanks to Stefan Schubert for suggesting I repost here. TL;DR for readers already familiar with the METR Measuring AI Ability to Complete Long Tasks paper: this post highlights some gaps between the measurements used in the paper and real-world work – gaps which are discussed in the paper, but have often been overlooked in subsequent discussion.]

It's difficult to measure progress in AI, despite the slew of benchmark scores that accompany each new AI model.

Benchmark scores don’t provide much perspective, because we keep having to change measurement systems. Almost as soon as a benchmark is introduced, it becomes saturated – models learn to ace the test. So someone introduces a more difficult benchmark, whose scores aren’t comparable to the old one. There’s nothing to draw a long-term trend line on.

METR recently published a study aimed at addressing this problem: Measuring AI Ability to Complete Long Tasks. They show how to draw a trend line through the last six years of AI progress, and how to project that trend into the future.

The study’s results have been widely misinterpreted as confirmation that AGI (AI capable of replacing most workers, or at least most remote workers) is coming within five years. In actuality, the study doesn’t say much about AGI, except to provide evidence against the most aggressive forecasts.

To explain what the paper does and does not say, I’ll begin by explaining the problem it set out to address.

We’re Gonna Need a Harder Test

[LW readers might skip this section]

Here’s a chart I’ve shown before:

Additional context available at the source

Each line represents AI performance on a particular benchmark. Unsurprisingly, there’s a clear pattern of rapid improvement. But each line is measuring something different; we can’t connect the dots to plot a long-term trend.

For instance, the dark green line shows scores on a speech recognition test. If you’d looked at that plot in 2013, you might have been able to estimate speech recognition scores for 2015. But you’d have had no way of anticipating that reading comprehension (pink line) would start to take off after 2016. Performance on one benchmark do...

Posts

Wikitag Contributions

Comments