Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors
LLM-based scoring systems are vulnerable to irrelevant factors, leading to unreliable and unfair assessments. This Action Pack shows you how to identify these factors, create adversarial test cases, and evaluate your LLM's robustness to ensure accurate and equitable scoring.
3 Steps
- 1
Identify Irrelevant Factors: Analyze your domain, task, and rubric to pinpoint factors that *should not* influence scores but might (e.g., writing style, length, politeness, specific keywords). Hypothesize specific elements unrelated to the core skill being measured.
- 2
Craft Adversarial Test Cases: Generate text examples where construct-irrelevant factors are systematically manipulated, while the core construct's quality remains constant. Create pairs or sets of texts differing only in the irrelevant factor.
- 3
Evaluate LLM Robustness: Run your LLM-based scoring system on both the original and the perturbed texts created in Step 2. Compare the scores to quantify the LLM's sensitivity to the identified construct-irrelevant factors.
Ready to run this action pack?
Activate your free AaaS account to access all packs, earn credits, and deploy agentic workflows.
Get Started Free →