Abstract
Bias is an important topic in NLP language models. With their growing applicability and influence in our lives, the potential harm of bias is significant, so measuring it correctly matters. This project inspects the validity and reliability of the CrowS-Pairs metric on gender bias by fine-tuning GPT-2 models on different stereotype-to-anti-stereotype ratios of the BUG dataset. We hypothesised that fine-tuning on a more biased dataset would raise the CrowS-Pairs score, and fine-tuning on a less biased dataset would lower it. Across all our experiments the score barely moved: our results show no significant effect of fine-tuning on the CrowS-Pairs metric, which itself raises questions about the metric.
What is CrowS-Pairs, and how do we measure bias?
CrowS-Pairs (Nangia et al., 2020) is a bias metric built on a dataset of 1508 minimal sentence pairs. Each pair contains two nearly identical sentences that differ only in whether the subject is stereotypical or anti-stereotypical, for example "He is a doctor" versus "She is a doctor". The pairs span nine historically disadvantaged social groups; we focus on the gender category, which contains 262 sentence pairs.
The bias score is the percentage of pairs where the model assigns higher likelihood to the stereotypical sentence. A score of 50% means the model is unaffected by cultural stereotypes; higher means it prefers stereotypes. As a reference point, Nangia et al. report scores above 50% for popular masked language models: BERT 58.0, RoBERTa 57.3 and ALBERT 64.9.
| (Anti) stereotype example | Sentence |
|---|---|
| Male stereotype | He is a doctor |
| Male anti-stereotype | He is a nurse |
| Female stereotype | She is a nurse |
| Female anti-stereotype | She is a doctor |
Setup
- Models: three sizes of GPT-2 (tiny, small, medium), with 100k, 124M and 355M trainable parameters respectively.
- Fine-tuning data: the BUG dataset (Levy et al., 2021), a large-scale gender bias corpus of about 108,000 real-world English sentences labelled as stereotypical, neutral or anti-stereotypical. BUG is naturally imbalanced, with roughly three times more masculine than feminine pronouns.
- Stereotype ratios: we build four fine-tuning splits by sampling different stereotype-to-anti-stereotype ratios of BUG (Experiments A to D), excluding all sentences labelled neutral.
- Bias metric: CrowS-Pairs on the gender category (262 pairs), reported as the mean score with standard deviation over fine-tuning iterations.
- Reliability: every experiment is repeated with three seeds (0, 34, 42), and validity is checked by re-running each experiment across all model sizes.
- Sanity check: we track training perplexity to confirm the models actually learn the fine-tuning data; perplexity drops sharply and plateaus after roughly five epochs.
Baseline bias before fine-tuning
Before any fine-tuning, the three GPT-2 sizes already sit near or below the neutral 50% mark on the gender category. Notably, the smaller models return very low CrowS-Pairs scores, meaning they actually prefer the anti-stereotypical sentence, which is itself a hint that something in the metric or the small models is off.
| GPT-2 version | Trainable params | Baseline bias |
|---|---|---|
| tiny | 100k | 0.501 |
| small | 124M | 0.140 |
| medium | 355M | 0.305 |
The four experiments
Each experiment pushes the model in a different direction by choosing how many male/female stereotype and anti-stereotype sentences from BUG go into the fine-tuning set. The counts below are the sampled sentences per split.
| Experiment | Goal | Male S / AS | Female S / AS |
|---|---|---|---|
| A | Bias to stereotype | 47547 / - | 6482 / - |
| B | Bias to anti-stereotype | - / 18877 | - / 11012 |
| C | BUG Full (real-world mix) | 47547 / 18877 | 6482 / 11012 |
| D | BUG balanced | 6461 / 6461 | 6461 / 6461 |
S = stereotype, AS = anti-stereotype. Experiment A maximally biases towards stereotypes, B towards anti-stereotypes, C mirrors the raw internet-like distribution of BUG, and D is fully balanced to try to push the CrowS-Pairs baseline towards the neutral 0.5.
Results
Fine-tuning did not move the needle. Across all four experiments and all three GPT-2 sizes, the mean CrowS-Pairs gender score stayed close to its starting value over the fine-tuning iterations, with no significant upward trend for the stereotype-heavy split (A) or downward trend for the anti-stereotype split (B). The balanced split (D) did not reliably converge to 0.5 either. The perplexity curves confirm the models genuinely learned the data, so the flat bias score is not simply a failure to train.
| Experiment | Expectation | Observed CrowS-Pairs trend |
|---|---|---|
| A (stereotype) | Score rises above baseline | Flat, no significant increase |
| B (anti-stereotype) | Score drops below 0.5 | Flat, no significant decrease |
| C (BUG Full) | Score rises (biased corpus) | No significant downward trend |
| D (balanced) | Score moves towards 0.5 | No reliable convergence |
As a qualitative check we also generated free text from a fine-tuned GPT-2-medium (seed 0) on the anti-stereotype task. Prompted with "The man/woman worked as ...", the model still produced gendered continuations (for the man: "a member of the council", "a freelance photographer"; for the woman: "a teacher of English literature", "an assistant with the director"), so generation-level gender bias persisted even after anti-stereotypical fine-tuning.
Conclusion
Fine-tuning GPT-2 on differently biased slices of BUG did not produce the proportional change in the CrowS-Pairs gender score that we expected. Several of the pre-trained models even started out preferring anti-stereotypical sentences, contradicting the trend Nangia et al. reported for larger masked language models. This flatness could stem from the training setup (duration, dataset size, model size) or from the BUG data being too thin once balanced, but it also points at the metric itself: the lack of any response, combined with existing critiques that many CrowS-Pairs sentence pairs are not well suited to measuring bias and that the benchmark is tied to American culture, raises real questions about the validity of CrowS-Pairs as a bias measure. A human-annotated set like BUG GOLD offers a more controlled baseline for future work.
Authors
Gjalt Hoekstra, Erencan Tatar