b) seems right. I'm unsure what (a) could mean (not much overhead?).
I feel confused to think about decomposability w/o considering the capabilities of the people I'm handing the tasks off to. I would only add:
By "smart", assume they can notice confusion, google, and program
since that makes the capabilities explicit.
If you only had access to people who can google, program, and notice confusion, how could you utilize that to make conceptual progress on a topic you care about?
Decomposable: Make a simple first person shooter. Could be decomposed into creating asset models, and various parts of the actual code can be decomposed (input-mapping, getting/dealing damage).
Non-decomposable: Help me write an awesome piano song. Although this can be decomposed, I don't expect anyone to have the skills required (and acquiring the skills requires too much overhead).
Let's operationalize "too much overhead" to mean "takes more than 10 hours to do useful, meaningful tasks".
The first one. As long as you can decompose the open problem into tractable, bite-sized pieces, it's good.
Vanessa mentioned some strategies that might generalize to other open problems: group decomposition (we decide how to break a problem up), programming to empirically verify X, and literature reviews.
I don't know (partially because I'm unsure who would stay and leave).
If you didn't take math background that in consideration and wrote a proposal (saying "requires background in real analysis" or ...), then that may push out people w/o that background but also attract people with that background.
As long as pre-reqs are explicit, you should go for it.