Stubborn or Sycophantic? Auditing What Sycophancy Evaluations Measure
NeurIPS TAE 2026 Poster · Cited by 0
TL;DRAudits what anti-sycophancy evaluations actually measure by scoring frozen system prompts under both valid corrections and misleading user pressure on SycoBench-600 across Phi-3-mini, Llama-3.1-8B, and Mistral-7B — a reversal-only score can reward blanket stubbornness (one prompt cuts Llama's harmful reversals by 53.7 points but also its valid corrections by 46.0), no prompt improves correction selectivity on all three models, and prompt verdicts do not transfer uniformly across models.