Can you provide some simple or not-so-simple example automata in that language?

Just a data point that support hold_my_fish's argument: Savant Kim Peek did likely memorize gigabytes of information and could access them quite reliably:

https://personal.utdallas.edu/~otoole/CGS_CV/R13_savant.pdf 

Are there different classes of learning systems that optimize for the reward in different ways?

I don't think that shards are distinct - neither physically nor logically, so they can't hide stuff in the sense of keeping it out of view of the other shards.  

Also, I don't think "querying for plans" is a good summary of what goes on in the brain. 

I'm coming more from a brain-like AGI lens, and my account of what goes on would be a bit different. I'm trying to phrase this in shard theory terminology.

First, a prerequisite: Why do Alice's shards generate thoughts that value Rick's state, to begin with? The Risk-shard has learned that actions that make Rick happy result in states of Alice that are reinforced (Alice being happy/healthy).

Given that, I see the process as follows:

1. Alice's Rick-shards generate thoughts at different levels of abstraction about Alice being happy/healthy because Rick is happy/likes her. Examples:
   1. Conversion (maybe a cached thought out of discussions they had) -> will have low predicted value for Alice
   2. Going to church with Rick -> mixed
   3. Being close to Rick in the Church (emphasis on closeness, Church in the background, few aspects active) -> trend positively
   4. Being in the Church and thinking it's wrong -> consistent
   5. Rick being happy that she joins him -> positive 
2. So the world model returns no plan but only fragments of potential plans, some where she converts and goes to church with Rick, some not, some other combinations.
3. As there is no plan no purpose must be hidden. Shards only bid for or against parts of plans.
4. Some of these fragments satisfy enough requirements of both retaining atheist Alice's values (which are predicted to be good for her) as well as scoring on Rick-happiness. Elaborating on these fragments will lead to the activation of more details that are at least somewhat compatible with all shards too. We only call the result of this a "rationalization."
5. So that she eventually generates enough detailed thoughts that score positively that she actually decides to implement an aggregate of these fragments, which we can call a church-going plan. 
6. So that she gets positive reinforcement for going to church,
7. which reinforces all aspects of the experience, including being in church, which, in aggregate, we can call a religion-shard,
   1. I agree that this changes her internal shard balance significantly - she has learned something she didn't know before, and that leaves her better off (as measured by some fundamental health/happyness measurements). 
   2. I think this can be meaningfully called value drift only with respect to either existing shards (though these are an abstraction we are using, not something that's fundamental to the brain), or with respect to Alice's interpretations/verbalizations - thoughts that are themselves reinforced by shards.
8. so that more such thoughts come up, and she eventually converts, 
9. so that Rick ends up happier and liking Alice more - though that was never the "plan" to begin with.

In short: There is no top-down planning but bottom-up action generation. All planning is constructed out of plan fragments that are compatible with all (existing) shards.

Some other noteworthy groups in academia lead by people who are somewhat connected to this community:
- Jacob Steinhardt (Berkeley)
- Dylan Hadfield-Menell (MIT)
- Sam Bowman (NYU)
- Roger Grosse (UofT)

Some other noteworthy groups in academia lead by people who are perhaps less connected to this community:
- Aleksander Madry (MIT)
- Percy Liang (Stanford)
- Scott Neikum (UMass Amhearst)

Can you provide some links to these groups?

Some observations:

• Genes reproduce themselves. 
• Humans reproduce themselves.
• Symbols are relearned.
• Values are reproduced. 

Each needs an environment to do so, but the key observation seems to be that a structure is reliably reproduced across intermediate forms (mitosis, babies, language, society) and build on top of each other. It seems plausible that there is a class of formal representations that describe 

• the parts that are retained across instances and
• the embedding into each other (values into genes and symbols), and
• the dynamics of the transfer.   

You don't talk about human analogs of grokking, and that makes sense for a technical paper like this. Nonetheless, grokking also seems to happen in humans, and everybody has had "Aha!" moments before. Can you maybe comment a bit on the relation to human learning? It seems clear that human grokking is not a process that purely depends on the number of training samples seen but also on the availability of hypotheses. People grok faster if you provide them with symbolic descriptions of what goes on. What are your thoughts on the representation and transfer of the resulting structure, e.g., via language/token streams?

I mean scoring thoughts in the sense of [Intro to brain-like-AGI safety] 3. Two subsystems: Learning & Steering with what Steven calls "Thought Assessors". Thoughts totally get scored in that sense.

About the problems you mention:

the apparent phenomenon of credit assignment improving over a lifetime. When you're older and wiser, you're better at noticing which of your past actions were bad and learning from your mistakes.

I don't get why you see a problem here. More data will lead to better models over time. You get exposed to more situations, and with more data, the noise will slowly average out. Not necessarily because you can clearly attribute things to their causes, but because you randomly get into a situation where the effect is more clear. It mostly takes special conditions to get people out of their local optimum. 

without any anti-reinforcement event occurring

And if it looks like this comes in hindsight by carefully reflecting on the situation, that's not without reinforcement. Your thoughts are scored against whatever it is that the brainstem is evaluating. And same as above, earlier or later, you stumble into some thoughts where the pattern is more clearly attributable, and then the weights change.  

The main difference between LDAIXI and a human in terms of ontology seems to be that the things the human values are ultimately grounded in senses and a reward tied to that. For example, we value sweet things because we have a detector for sweetness and a reward tied to that. When our understanding of what sugar is changes the detector doesn't, and thus the ontology change works out fine. But I don't see a reason you couldn't set up LDAIXI the same way: Just specify the reward in terms of a diamond detector - or multiple ones. In the end, there are already detectors that AIXI uses - how else would it get input?