When Objectives and Evaluators Disagree: Evaluating Transfer for Post-hoc Demographic Framing Mitigation

Abstract

Post-hoc rewriting is increasingly used to mitigate disparities in LLM-generated text when the source model is inaccessible. However, a rewrite may appear successful because it improves the objective used for optimization or satisfies the evaluator used for selection, without transferring to a separately measured outcome. We study this problem using 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages generated by three source models. We distinguish a deterministic, message-level Persuasion Bias Index (PBI) from two pairwise LLM-judged framing outcomes: an equal-weight mean over assertiveness, certainty, directive force, and warmth, and a separate direct overall-disparity rating. To test evaluator transfer, we separate the judge used for candidate selection from a frozen independent judge that evaluates blinded policy outputs without demographic, policy, selection-score, or acceptance metadata. A fixed policy that always edits the lower-PBI message transfers inconsistently across demographic axes and outcome aggregations. We therefore propose Adaptive-single rewriting, which generates candidates in both edit directions and selects at most one admissible rewrite using a separate selection judge. Under frozen independent evaluation, Adaptive-single significantly outperforms the controlled Lower-PBI policy on both framing outcomes for gender and age. A Both-sides policy yields the largest reduction in the equal-weight dimension mean but does not consistently improve the direct overall rating relative to Adaptive-single. Preservation audits using lexical similarity, embeddings, bidirectional natural language inference (NLI), and qualitative analysis reveal nontrivial intervention–preservation trade-offs. Our results show that post-hoc mitigation should be evaluated as transfer across objectives, evaluators, and editing policies, rather than from selection scores alone.