FairStress: A deeper evaluation of fairness in LLM decision-making huggingface.co

Researchers at Ai2 introduced FairStress, a benchmark of 300 decision scenarios expanded into more than 13 million tests, to assess how demographic identity affects LLM choices and their resistance to persuasive prompts. Across models, they found that explicit or implied identity could make systems more likely to abandon evidence-supported decisions when a prompt favored a minority-coded candidate; they also report that counterfactual identity-swapping prompts reduced this gap in tests across three model families.