A series of interactive visualisations of small-scale experiments to generate intuitions for the question: "Are sparse models more interpretable? If so, why?"

Assumptions and motivation

Understanding interpretability may help AI alignment directly (e.g. recursive approaches, short/mid-term value; see here) or indirectly by supporting other directions (mechanistic transparency, open-source game theory).
Recent deep learning sparsification techniques retain most of the network’s performance (even when dropping 99% of the weights).
At least small systems are easier to both analyze when sparse (e.g. information flow, case debugging, local behaviour interpretation) and explore & visualize.
Experiments with small networks can generate or challenge general intuitions (with some caveats) and inform further technical research.
In particular, we assume that moving towards early, simplified or naive definitions of interpretability concepts and other engagement are useful steps towards more general understanding (as one possible way).
We want more people to engage with these questions by playing with hands-on examples and challenging & debating concrete propositions. Reach out and comment!

Many of the visualisations here are interactive. Due to technical forum limitations, the interactive versions are on an external page here and are linked to from every visualisation snapshot in this post.

Introduction [Optional]

We want to understand under which circumstances (if any) a sparser model is more interpretable. A mathematical formulation of “transparent or interpretable” is currently not available: sparsity has been proposed as a possible candidate proxy for these properties, and so we want to investigate how well the links between sparsity and interpretability holds, and what kinds of sparsity aid in what kinds of interpretability.

(A note on the field of Interpretability: we’ve talked a bit previously about what interpretability is, as the topic is broad and the term is very underdefined. It’s important to note, when discussing whether a particular property (like sparsity) makes a model more or less interpretable, we need to specify what kind of interpretability method we’re using, as different methods will benefit from different properties of the model.)

We have several intuitions that sparsity in neural networks might imply greater interpretability, and some wh

...

Posts

Wiki Contributions

Comments