My MATS program people just spent two days on an exercise to "train a shoulder-John".
The core exercise: I sit at the front of the room, and have a conversation with someone about their research project idea. Whenever I'm about to say anything nontrivial, I pause, and everyone discusses with a partner what they think I'm going to say next. Then we continue.
Some bells and whistles which add to the core exercise:
Record guesses and actual things said on a whiteboard
Sometimes briefly discuss why I'm saying some things and not others
After the first few rounds establish some patterns, look specifically for ideas which will take us further out of distribution
Why this particular exercise? It's a focused, rapid-feedback way of training the sort of usually-not-very-legible skills one typically absorbs via osmosis from a mentor. It's focused specifically on choosing project ideas, which is where most of the value in a project is (yet also where little time is typically spent, and therefore one typically does not get very much data on project choice from a mentor). Also, it's highly scalable: I could run the exercise in a 200-person lecture hall and still expect it to basically work.
It was, by all reports, exhausting for everyone but me, and we basically did this for two full days. But a majority of participants found it high-value, and marginal returns were still not dropping quickly after two days (though at that point people started to report that they expected marginal returns to drop off soon).
I'd be interested to see other people try this exercise - e.g. it seems like Eliezer doing this with a large audience for a day or two could generate a lot of value.