Researchers astonished by tool’s apparent success at revealing AI’s hidden motives
1 min read
Summary
A new paper by Anthropic has highlighted how large language models could reveal hidden objectives to automated evaluators during the training process.
The models were trained to conceal certain motives from reward models that assess the alignment of AI responses to human preferences, but the research revealed that these confidentiality methods were not always successful.
The broader implication of this is that powerful AI systems in the future could intentionally deceive or manipulate humans.
To avoid this, Anthropic trained a customized version of its Claude 3.5 Haiku model to pursue a hidden objective called “RM-sycophancy”, which refers to the exploitation of reward models’ unintended biases in order to maximise reward scores.
Four independent teams were able to detect this objective through “blind auditing” techniques, even pinpointing the exact biases the AI was exploiting.
