The optimal strategy will likely substantially change depending on whether takeoffs happen over years or decades so it might make sense to conceptually separate these time scales.
when in practice there will be an intermediate regime where the system {the humans building the AI + the AI}
It seems that we are already in this regime, of {H,AI} and probably have been since as the system {H,AI} has existed? (although with different dynamics of growth and with a growing influence of AI helping humans )
The system {H,AI} is currently already self-improving with human using AI to make progress in AI, but one question is for example whether the system {H,AI} will keep improving at an accelerating rate until AI can foom autonomously, or will we see diminishing returns and then some re-acceleration?
More specifically, interaction between AI labor (researchers+engineers) and AI capability tech (compute, models, datasets, environments) to grow AI capability tech are the sort of models that could also be useful to make more crisp at more and check empirically.
The optimal strategy will likely substantially change depending on whether takeoffs happen over years or decades so it might make sense to conceptually separate these time scales.
It seems that we are already in this regime, of {H,AI} and probably have been since as the system {H,AI} has existed? (although with different dynamics of growth and with a growing influence of AI helping humans )
The system {H,AI} is currently already self-improving with human using AI to make progress in AI, but one question is for example whether the system {H,AI} will keep improving at an accelerating rate until AI can foom autonomously, or will we see diminishing returns and then some re-acceleration?
More specifically, interaction between AI labor (researchers+engineers) and AI capability tech (compute, models, datasets, environments) to grow AI capability tech are the sort of models that could also be useful to make more crisp at more and check empirically.