This challenge from 2018 basically asked about building a data set for training AI on human values (loosely construed so as to allow many approaches) and many of the submissions proposed ways to do it. You might find some interesting ideas there.

Caveat, I won the challenge by saying I didn't such an approach would work.

Suggested elsewhere by Max Daniel:

• Ultimatum game or other widely studied games in psych/behavioral econ?
• Ebay bidding, or other auctions?
• Chess or other games?
• Voting in elections
• Gambling: casinos, online poker ...
• Online dating behavior

Suggested by Ozzie Gooen:

• This sounds a bit to me like psychology experiments with children, or perhaps some well studied psychology experiments (where there are large amounts of data, with relatively narrow options).
• Websites would have more than enough data for narrow decisions, like, “Which ad will this user click”, or on Netflix, “Which movie/tv show will they select?”
• There’s a fair bit of data for the main decisions of chess/starcraft/etc, Like, “which race will be chosen/ which character will be chosen / which strategy will be chosen”

Suggested by Jan Brauner:

• Any ML dataset with labels. The labels were created by humans. E.g. ImageNet: a human was shown an image and had to choose one of 1000 options.

I think it's a little dull for the score to be a good proxy of human value, so games involving aesthetic choices are an obvious choice to me because of the plausibility of learning about interesting values without tons of knowledge of the world. (Compare RPGs that might also have limited actions and reflect human values, but require common-sense understanding of text to draw interesting conclusions about.)

'Sim' games all seem good for this (Also there's just something apropos about making a value learning AI build a nice house for the Sims), as do most contraption-building games. Though if the number of actions there is still too large, maybe you want something more on the speed of Color A Dinosaur, or just the character/avatar creation screen of some scrapeable thing.

On the other hand, maybe these things don't have enough planning, and you want something more like an open-world game that allows for self-expression. But I think the large action space is a barrier here.