GiveWell recently release notes from their interview with Jaan Tallinn, Skype co-founder and a major SIAI donor, about SIAI (link). Holden Karnofsky says

[M]y key high-level takeaways are that

I appreciated Jaan's thoughtfulness and willingness to engage in depth. It was an interesting exchange.

I continue to disagree with the way that SIAI is thinking about the "Friendliness" problem.

It seems to me that all the ways in which Jaan and I disagree on this topic have more to do with philosophy (how to quantify uncertainty; how to deal with conjunctions; how to act in consideration of low probabilities) and with social science-type intuitions (how would people likely use a particular sort of AI) than with computer science or programming (what properties has software usually had historically; which of these properties become incoherent/hard to imagine when applied to AGI)