Abstract

Large language models perform well on NLP tasks, but the extent to which they grasp the underlying linguistics of language remains unclear. In this project we use probing to ask a concrete question: do transformer models have a stronger notion of syntactic structure than recurrent models? We compare distilGPT2 (transformer) against an LSTM on part-of-speech tagging and on recovering dependency parse trees, then test both against a control task to separate genuine linguistic knowledge from pattern memorisation.

What is probing?

A probe is a simple classifier trained on top of a frozen model's internal representations. If a small linear probe can read off a linguistic property (say, the POS tag of a word, or the distance between two words in a parse tree) from a model's activations, then that information is encoded in the representation. Because the probe is intentionally weak, it credits the language model, not the classifier.

We follow the structural probe of Hewitt & Manning (2019): learn a single linear transform of the representation space such that squared distances between transformed word vectors match distances in the gold dependency tree. Reconstructing the tree from these distances via a minimum spanning tree lets us score syntax recovery directly.

Setup

  • Models: distilGPT2 (auto-regressive transformer) and an LSTM language model (Gulordava et al.).
  • Data: the English EWT treebank from Universal Dependencies (gold POS tags and parse trees).
  • Probes: a diagnostic POS classifier and a structural distance probe.
  • Control task: each word type is assigned a random POS-like label; selectivity = task accuracy minus control accuracy measures real linguistic signal.
  • Metrics: Accuracy & Matthews correlation (POS), UUAS & Spearman distance correlation (structure).

Probing POS tags

The diagnostic classifier predicts a POS tag per token from the model representation. Below, the true tags (top) versus the probe's predictions from LSTM embeddings (bottom) on an example sentence.

POS tags predicted by the diagnostic probe versus gold tags
Confusion matrix of POS prediction on the control task, transformer

POS confusion matrix on the control task (transformer). A clean diagonal on the real task collapses on the control task, confirming the probe reads real structure, not memorised word identities.

Probing syntax: gold vs predicted trees

The structural probe predicts the pairwise tree distance between every pair of words. The heatmaps below compare the gold distance matrix against the probe's rounded prediction for the transformer on one sentence. Darker cells are closer in the tree; the recovered block structure lines up with the true syntactic constituents.

Gold parse-tree distance matrix
Gold tree distances
Predicted parse-tree distance matrix
Probe prediction (transformer)

Structure recovery drops with sentence length

UUAS (fraction of correctly recovered tree edges) falls sharply as sentences get longer. The UUAS metric is unforgiving: a single wrong choice near the root breaks the score. The decrease is steeper for the LSTM than for the transformer.

UUAS over sentence length, transformer
Transformer (distilGPT2)
UUAS over sentence length, LSTM
LSTM

Results

POS tagging. The LSTM is the better tagger, but the transformer shows higher selectivity, suggesting GPT2 relies more on word identity than the LSTM.

Model Acc MCC Selectivity (Acc)
GPT2 0.772 0.748 0.396
LSTM 0.889 0.878 0.317

Structural probe. The transformer wins on syntax: it recovers dependency trees better on both UUAS and Spearman distance correlation.

Model UUAS DSpr.
GPT2 0.529 0.636
LSTM 0.460 0.589

Conclusion

The two architectures encode language differently. The LSTM is the stronger POS tagger, while the transformer captures hierarchical syntactic structure better, consistent with its global self-attention over long-distance dependencies. Both beat their control tasks, so both encode genuine linguistic structure rather than surface patterns. More seeds, model sizes, and a focus on long-distance dependencies would strengthen these claims.

Authors

Gjalt Hoekstra, Erencan Tatar