Grammar or Crammer? The Role of Morphology in Distinguishing Orthographically Similar but Semantically Unrelated Words
Gökhan Ercan, Olcay Taner Yıldız
IEEE Access, vol. 13, pp. 64412–64458, 2025
Abstract
We show that n-gram-based distributional models fail to distinguish unrelated words due to the noise in semantic spaces. This issue remains hidden in conventional benchmarks but becomes more pronounced when orthographic similarity is high. To highlight this problem, we introduce OSimUnr, a dataset of nearly one million English and Turkish word-pairs that are orthographically similar but semantically unrelated (e.g., grammar – crammer). These pairs are generated through a graph-based WordNet approach and morphological resources. We define two evaluation tasks — unrelatedness identification and relatedness classification — to test semantic models. Our experiments reveal that FastText, with default n-gram segmentation, performs poorly (below 5% accuracy) in identifying unrelated words. However, morphological segmentation overcomes this issue, boosting accuracy to 68% (English) and 71% (Turkish) without compromising performance on standard benchmarks (RareWords, MTurk771, MEN, AnlamVer). Furthermore, our results suggest that even state-of-the-art LLMs, including Llama 3.3 and GPT-4o-mini, may exhibit noise in their semantic spaces, particularly in highly synthetic languages such as Turkish. To ensure dataset quality, we leverage WordNet, MorphoLex, and NLTK, covering fully derivational morphology supporting atomic roots (e.g., "-co_here+ance+y" for "coherency"), with 405 affixes in Turkish and 467 in English.
Key findings
- FastText with default character n-gram segmentation identifies orthographically similar but semantically unrelated words with below 5% accuracy — a failure that conventional word-similarity benchmarks hide entirely.
- Replacing n-gram segmentation with morphological segmentation raises that accuracy to 68% in English and 71% in Turkish, with no loss on standard benchmarks (RareWords, MTurk771, MEN, AnlamVer).
- State-of-the-art large language models, including Llama 3.3 and GPT-4o-mini, show the same kind of noise in their semantic spaces, and it is more pronounced in highly synthetic languages such as Turkish.
- OSimUnr: a dataset of nearly one million English and Turkish word-pairs that are orthographically similar yet semantically unrelated (e.g. grammar – crammer), built from a graph-based WordNet approach plus morphological resources.
- Two evaluation tasks are defined for probing semantic models: unrelatedness identification and relatedness classification.
- Derivational morphology coverage supporting atomic roots (e.g. "-co_here+ance+y" for "coherency"), spanning 405 affixes in Turkish and 467 in English, drawn from WordNet, MorphoLex and NLTK.
Download PDF
DOI: 10.1109/ACCESS.2025.3557086