Ought will host a factored cognition “Lab Meeting” on Friday September 16 from 9:30AM - 10:30AM PT. We'll share the progress we've made using language models to decompose reasoning tasks into subtasks that are easier to perform and evaluate. This is part of our work on supervising process, not outcomes....
Ought is an applied machine learning lab. We’re building Elicit, the AI research assistant. Our mission is to automate and scale open-ended reasoning. To get there, we train language models by supervising reasoning processes, not outcomes. This is better for reasoning capabilities in the short run and better for alignment...
We can think about machine learning systems on a spectrum from process-based to outcome-based: * Process-based systems are built on human-understandable task decompositions, with direct supervision of reasoning steps. * Outcome-based systems are built on end-to-end optimization, with supervision of final results. This post explains why Ought is devoted to...
Ought will host a factored cognition “Lab Meeting” on Friday September 16 from 9:30AM - 10:30AM PT.
We'll share the progress we've made using language models to decompose reasoning tasks into subtasks that are easier to perform and evaluate. This is part of our work on supervising process, not outcomes. It’s easier for us to show you than to tell you about it in a post (though written updates will hopefully follow).
Then, we'll cover outstanding research directions we see and plan to work on, many almost shovel-ready. If the alignment community can parallelize this work across different alignment research teams, we can make progress faster. We'd love to coordinate with other alignment researchers thinking... (read 266 more words →)
This was really interesting, thanks for running and sharing! Overall this was a positive update for me.
Results are here
I think this just links to PhilPapers not your survey results?
I also appreciated reading this.
This was really helpful and fun to read. I'm sure it was nontrivial to get to this level of articulation and clarity. Thanks for taking the time to package it for everyone else to benefit from.
If anyone has questions for Ought specifically, we're happy to answer them as part of our AMA on Tuesday.
I think we could play an endless and uninteresting game of "find a real-world example for / against factorization."
To me, the more interesting discussion is around building better systems for updating on alignment research progress -
Ought is an applied machine learning lab. We’re building Elicit, the AI research assistant. Our mission is to automate and scale open-ended reasoning. To get there, we train language models by supervising reasoning processes, not outcomes. This is better for reasoning capabilities in the short run and better for alignment in the long run.
In this post, we review the progress we’ve made over the last year and lay out our plan for Elicit. We'd love to get feedback on how to make Elicit more useful for LW and to get thoughts on our plans more generally. To make this easier, we've erred on the side of giving detail even where we know... (read 3805 more words →)
We can think about machine learning systems on a spectrum from process-based to outcome-based:
This post explains why Ought is devoted to process-based systems. The argument is:
Access
Alignment-focused policymakers / policy researchers should also be in positions of influence.
Knowledge
I'd add a bunch of human / social topics to your list e.g.
Research methodology / Scientific “rationality,” Productivity, Tools
I'd be really excited to have people use Elicit with this motivation. (More context here and here.)
Re: competitive games of introducing new tools, we did an internal speed Elicit vs. Google test to see which tool was more efficient for finding answers or mapping out a new domain in 5 minutes. We're broadly excited to structure and support competitive knowledge work and optimize research this way.
This is exactly what Ought is doing as we build Elicit into a research assistant using language models / GPT-3. We're studying researchers' workflows and identifying ways to productize or automate parts of them. In that process, we have to figure out how to turn GPT-3, a generalist by default, into a specialist that is a useful thought partner for domains like AI policy. We have to learn how to take feedback from the researcher and convert it into better results within session, per person, per research task, across the entire product. Another spin on it: we have to figure out how researchers can use GPT-3 to become expert-like in new domains.
We’re... (read more)
I generally agree with this but think the alternative goal of "make forecasting easier" is just as good, might actually make aggregate forecasts more accurate in the long run, and may require things that seemingly undermine the virtue of precision.
More concretely, if an underdefined question makes it easier for people to share whatever beliefs they already have, then facilitates rich conversation among those people, that's better than if a highly specific question prevents people from making a prediction at all. At least as much, if not more, of the value of making public, visual predictions like this comes from the ensuing conversation and feedback than from the precision of the forecasts themselves.
Additionally,... (read more)