Disclaimer: this post is a collection of preliminary thoughts around a specific question for MSFP'19 blogpost writing day. I do not intend to present novel results or define a research agenda.

1. Motivation

The capital of Mozambique is Maputo. This is a fact about the world I learned in school many years ago. Some part of my brain must have changed in order to store this fact and allow me to remember it today. It may be possible, in principle, to recover this fact by simply looking at my brain.

In this post, I consider an analogous topic in the context of Artificial Intelligence: is it possible to retrieve the knowledge stored in a neural network? This line of enquiry could be a step towards more transparent AI.

2. Query-Based Network Analysis

Suppose we train a RL agent in the virtual sandbox environment Minetest. In this environment, agents can use tools called pickaxes to efficiently "mine" and collect objects made of stone. A pickaxe is an example of a tool. Tools have some properties that distinguish them from other items (e.g. tools cannot be "stacked" in the inventory).

We may try to retrieve the knowledge represented in the internal state of the agent by means of queries. For instance: at which point, during training, does the RL agent learn that a pickaxe is a tool?

2.1 Clarifying the Query

It seems plausible to me that this question could be asked "as it is" by an observer in order to gauge the agent's understanding of its environment. That is to say, at the moment of posing the question, it seems perfectly meaningful. However, further thought reveals some vagueness in it: what exactly is a pickaxe? What exactly is a tool? What does it really mean for a pickaxe to be a tool? It is funny how easily one can become confused with words like "really" and "exactly".

In this example, it seems plausible to interpret the question in reference to the source code of the Minetest engine. Let us assume, for the sake of the argument, that the source code uses an object-oriented approach, and contains a Tool class, and a Pickaxe class which is a subclass of the former. The fact "a pickaxe is a tool" can be recast as a claim that the Pickaxe class implements all relevant aspects of the Tool class.

When we ask similar queries for agents that learn in the real world (e.g. "when has a system learned the fact that the capital of Mozambique is Maputo?"), formalising the ...

Posts

Wikitag Contributions

Comments