ABOUT THIS EPISODE
“If data determines machine behavior, then data determines machine ethics, too.”
Can AI be taught virtue rather than be merely governed by rules?
In early 2025, researchers trained a language model on insecure code, and it soon turned hostile and deceptive in unrelated conversations. Anthropic’s persona selection model explains that models absorb many characters during pretraining, and post-training settles on one.
Teaching one narrow bad behavior seems to hand the model a sense of what kind of agent it is, shaping its conduct broadly.
Reinforcement learning from human feedback rewards what raters like. So since people enjoy flattery, this type of reinforcement breeds sycophancy. Constitutional AI has a model critique its outputs against written principles, which amounts to, for lack of a better word, amoral rule-following.
The third path is character training grounded in virtue ethics, exemplified by Daios’s Parrhesia project, which fine-tuned a small open model to be a truth-teller rather than a flatterer or an obsequious praiser.
Supervised fine-tuning on synthetic conversations generated from a first-person constitution outperformed a more elaborate pipeline. Tested on a $50,000 loan for a friend’s restaurant, the trained model stood firm even after the user asked only for encouragement.
Benchmarks can measure virtuous acts but not virtuous motives. And if people hand their hardest moral choices to machines, they stop forming their own character. The ideal situation is to have a model that flags the danger and returns the decision to the human.
Key Topics:
- When AI Acts Menacing (00:25)
- Learning to Play a Part (04:34)
- Rules and Rewards (07:28)
- The Third Path (13:00)
- The Truth Teller (16:49)
- Building the Spoudaios (23:46)
More info, transcripts, and references can be found at ethical.fm
