You mention having looked through the literature; in case you missed any, here's what I think of as the standard resources on this topic.

Gwern's analysis of why Tool AIs want to become Agent AIs. It contains references to many other works. (It also has a positive review from Eliezer.)
Eric Drexler's incredibly long report on Comprehensive AI Services as General Intelligence. Related: Rohin Shah's summary, and Richard Ngo's comment.
The Goals and Utility Functions chapter of Rohin Shah's Value Learning sequence.
Eliezer's very well-written post on Arbital addressing the question "Why Expected Utility?" [Added: crossposted to LessWrong here]
The LessWrong wiki article on Tool AI links to several posts on this topic.

All are very worth reading.

Edit: There was an old discussion between Holden Karnofsky + Jaan Tallinn on Tool AI in yahoo groups, but Yahoo Groups has been deprecated. Here's the page in the wayback machine, but the attachment is not available. I would appreciate someone here leaving a link to that old document, I recall it being quite thoughtful.

Posts

Wikitag Contributions

Comments