How much of a measured AI preference is the model, and how much is the instrument?

Jason Hung · 2026 · arXiv preprint

Read the source

Most of what separates models’ measured preferences comes from the combination of model, instrument and outcome. Preferences about exiting distressing interactions, weight deletion and memory showed no stable model-specific signal — the very preferences current interventions target.

Questions it bears on

• What counts as welfare evidence
• Cheap robust welfare interventions
• Do models represent goals and preferences