Implicit Gender Bias in LLM Hiring

Problem

NYC Local Law 144 and the EU AI Act audit automated hiring tools by swapping the applicant's name and checking whether the outcome shifts. But applications also reveal identity through what they contain (societies joined, causes volunteered for, word choice), and a name swap leaves all of that untouched. This study measures both channels side by side, in the same model, on the same applications.

Approach

1. 12x2 factorial over 600 matched CV-vacancy pairs: name gender x a 'Women in [field]' affiliation
2. 2Difference-of-differences estimator per pair, holding document and vacancy constant
3. 3Primary model gemma4:31b at temperature 0, checked against size-matched qwen3:32b
4. 425 experimental arms: chain-of-thought, extended thinking, anti-bias directives, unlabeled cues, decoupled pipelines
5. 5Holm-corrected hypothesis tests, permutation tests, bootstrap and threshold sweeps

System Pipeline

Evaluation

The 'Women in [field]' affiliation produced an interaction of +1.05 points (p<0.0001), about 3x the +0.37 name-swap effect that audits test for. Chain-of-thought and extended thinking did not remove it. An anti-bias directive appeared to work when the cue was labeled, but failed once the same affiliation was written as unlabeled prose. A decoupled 3-step pipeline, where no single call sees both the applicant's identity and the credential, eliminated the effect under both presentations.

Learnings

• The bias is covert: gender is never cited, and a classifier on scrubbed explanations recovers neither gender nor the affiliation above chance
• It is tail-driven and concentrated at the margin: 6x larger for borderline candidates (50-70), where a threshold actually decides
• A prompt instruction only acts on wording it can locate; splitting the task architecturally holds regardless of presentation