I've been puzzling about the meaning of horizon lengths and whether to expect trends to be exponential or superexponential. Also how much R&D acceleration we should expect to come from what horizon length levels -- Eli was saying something like "90%-horizons of 100 years sound about right for Superhuman Coder level performance" and I'm like "that's insane, I would have guessed 80%-horizons of 1 month." How to arbitrate this dispute? 

This appendix from METR's original paper seems relevant. I'm going to think out loud below.

OK so, how should we define horizon length? On one way of defining it, it's inherently pegged to what human experts can do. E.g. arguably, METR's HCAST benchmark is constructed by selecting tasks that human experts can do, and labelling them with horizon lengths based on how long it takes human experts to do them. Thus an arbitrarily extended HCAST (with longer and longer, more difficult tasks) would still only have tasks in it that human experts can do. Thus, superintelligent AI would have infinite horizon length. Thus, the trend must be superexponential, because it needs to get to infinity in finite time (unless you think ASI is impossible)

But maybe that's not the only way of thinking about it. Maybe instead we should think of some bigger set of tasks and a bigger set of skills, such that some of them simply can't be done by any human. And we should think of HCAST as sampling from that bigger set. And it's just a minor methodological issue that the current version of HCAST only includes tasks humans can do; the proper way to think about it--and the proper way to think about how to extend it--would be to sample tasks in some unbiased way that will gradually include more tasks over time that are harder and harder and that fewer and fewer humans can do, eventually including tasks that no human can do.

In this case, horizon length only goes to infinity if and when there is an AI which can do all tasks from the bigger set. Depending on how ambitiously you define that bigger set, this may not be possible even in principle. But at any rate, an AI system which is exactly as good as the best humans at everything while being faster and cheaper (minimum possible superintelligence) would still fail at a bunch of tasks from this bigger set, so it would still have a finite horizon length. So even if the underlying trend was superexponential it wouldn't be going to infinity anytime soon, and maybe it would make sense to model it as exponential.

So which of these two ways is the better way of thinking about it?

Enter the above picture from the original paper.

While they are filtering to only include tasks humans can do, it seems like fewer and fewer humans can do the tasks as they get longer. So maybe that means it's appropriate to imagine extrapolating HCAST in the second way, the way in which it eventually gets to tasks that no human can do. On that way, individual humans have a finite horizon length and the max of all humans (i.e. for each task, pick the world's best human to do that task) also has a finite horizon length, and ASI would probably also have a finite horizon length.

So then the question becomes, what are those horizon lengths? What is the horizon length (on coding tasks) of the best frontier AI company engineers? What is the horizon length (on coding tasks) of the max of all humans?

Well, according to the image above, it seems like the horizon length (on coding tasks) of METR's baseliners is... about one and a half hours!!!

Further complication. That's the horizon length of baseliners given 8 hours; had they been given more time they would have performed better. Similarly perhaps, we shouldn't talk about the horizon length of the max of all humans, or ASI, or any given AI system, in isolation, but rather relative to a time/compute/etc. budget. (Or perhaps more cleanly, the allotted time/compute/etc. budget should be considered part of the task.)

Further complication: As METR points out in the screenshot, they condition on success when defining the time horizon of a task. So e.g. if they have 10 baseliners try a task, allotted 8 hours, and only 2 succeed, then the task length is defined as the average of how long it took those 2 people. But this is a bit crazy because it guarantees that the horizon length of the task will be less than 8 hours even in cases where it arguably should be more! E.g. suppose the 10 people would have all succeeded at the task if they were given more time, and the median time-to-succeed would have been 20 hours, but the top 2 people with a combination of skill and luck managed to do it in 3 and 6 hours respectively. Then the task would be labelled as a 4.5 hour task even though the median baseliner needs 20 hours to do it.

At this point there are enough complications that I'm confused and want to take a break and return to the matter later. I'd be grateful for any comments shedding light on the matter.