This work was produced as part of the Apart Fellowship. @Yoann Poupart and @Imene Kerboua led the project; @Clement Neo and @Jason Hoelscher-Obermaier provided mentorship, feedback and project guidance.

Here, we present a qualitative analysis of our preliminary results. We are at the very beginning of our experiments, so any setup change, experiment advice, or idea is welcomed. We also welcome any relevant paper that we could have missed. For a better introduction to interpreting adversarial attacks, we recommend reading scasper’s post: EIS IX: Interpretability and Adversaries.

Code available on GitHub (still drafty).

TL;DR

Basic adversarial attacks produce humanly indistinguishable examples for the human eye.
In order to explore the features vs bugs view of adversarial attacks, we trained linear probes on a classifier’s activations (we used CLIP-based models for further multimodal work). The goal is that training linear probes to detect basic concepts can help analyse and detect adversarial attacks.
As the concepts studied are simple (e.g. color), they can be probed from every layer of the model.
We show that concept probes are indeed modified by the adversarial attack, but only on the later layers.
Hence, we can naively detect adversarial attacks by observing the disagreement between early layers’ probes and later layers’ probes.

Introduction

Motivation

Adversarial samples are a concerning failure mode of deep learning models. It has been shown that by optimising an input, e.g. an image, the output of the targeted model can be shifted at the attacker’s will. This failure mode appears repeatedly in different domains like image classification, text encoders, and more recently on multimodal models enabling arbitrary outputs or jailbreaking.

Figure 1: Experimental setup to analyse an adversarial attack using linear probes. Given a classifier trained on image samples, we train linear probes to detect the concepts activated by each input image. Each linear probe is specialised on one concept. In this example, an image of a lemon should activate the concept “ovaloid”. When the input is attacked, the probe can detect other concepts related to the targeted adversarial class.

Hypotheses to Test

This work was partly inspired by the feature vs bug world hypothesis. In our particular context this would imply that adversarial attacks migh...

Posts

Wikitag Contributions

Comments