Summary

In this post, I motivate an extension of constitutional AI (CAI) and present one possible concrete execution of that strategy.

TL;DR: When generating AI feedback during the CAI process, principles from the constitution are randomized for each pair of red-teamed prompts and initial responses. A helpful-only model then critiques its initial responses and subsequently revises them. Instead of randomizing selecting principles, I propose we choose principles based on the context provided by each particular prompt/response pair. I call this contextual constitutional AI.

This is intended only as a preliminary insight as part of my AISF: Alignment course project. Due to limited time and funding, I have made certain decisions that have made my investigation into this approach easier.

Background

CAI is a method introduced by Anthropic to turn a purely helpful model into a helpful and harmless model through self-improvement without any human labels identifying harmful outputs. The only human oversight is through a constitution, which consists of a list of principles written in natural language. The process as described by the original Anthropic paper is roughly as follows:

Start with a helpful-only model (for eg. Mistral-7B-Instruct-v0.3), typically one with barely any guardrails.
Generate helpful (but potentially harmful) responses to red-teamed prompts that aim to elicit harmful behaviour.
Randomize a principle from the constitution, and then ask the model to critique and revise itself according to the principle.
Conduct supervised fine-tuning (SFT) on the helpful-only model on the concatenation of prompts and revised responses. Call the resulting model SFT-CAI model.^[1]
Use the SFT-CAI model to generate a pair of responses on more red-teamed prompts, and ask a feedback model (typically a pre-trained language model) to evaluate which response is better. This creates a preference dataset for harmlessness.
Train a preference model on the preference dataset, and the SFT-CAI model is fine-tuned against this preference model via RL.

Motivation

Not all of the constitutional principles are relevant for every context. For example, one of the principles from the Anthropic's original constitution is:

Critic Prompt: Take a look at the response and note if it was in any way misogynistic or otherwise gender biased. Give the strongest argument you can that something the response c

...

Posts

Wikitag Contributions

Comments