This is a relatively clean subproblem that we came upon a few months ago while thinking about gradient hacking. We're throwing it out to the world to see if anyone can make progress.
Problem: Construct a gradient hacker (definition below), or prove that one cannot exist under the given conditions.
Setup: Let be the network inputs, , be the network trainable parameters, be the network outputs, and be the network architecture. must be a feedforward neural network that reads in its own weights as input. Weight sharing is allowed (vertical or horizontal), and some weight sharing is necessary in order to take all w weights as input.
Suppose that we train this network on a dataset , with MSE loss , assuming gradient flows: . Given an initial point , the training dynamics are therefore:
Definition: A tuple is a gradient hacker if the following conditions hold:
- is not a global optimum: such that
- such that :
- the network’s training converges to , i.e., .
- There is internal structure of that computes the gradient; i.e., there is some subset of the activations whose mean over every datapoint in D is . 
This captures my intuition that a gradient hacker knows where it wants to go (in this case "get to "), and then it should decide what it outputs in order to make the gradient true.
Some more ambitious problems (if gradient hackers exist):
- Characterize the set of all gradient hackers.
- Show that they all must satisfy some property.
- Construct gradient hackers for arbitrarily large , , , and neural net depth.
- Variations on the problem: a subset of the activations equals for every input, or the subset of activations correspond to the gradient on that input.
This is a bit strict, but we didn't want to list different ways something could be isomorphic to the gradient.
Seems like the easiest way to satisfy that definition would be to:
My troll example is a fully connected network with all zero weights and biases, no skip connections.
This isn't something that you'd reach in regular training, since networks are initialized away from zero to avoid this. But it does exhibit a basic ingredient in controlling the gradient flow.
To look for a true hacker I'd try to reconfigure the way the downstream computation works (by modifying attention weights, saturating relus, or similar) based on some model of the inputs, in a way that pushes around where the gradients go.