---
title: An autoencoder that finds errors in a deterministic parser
description: Gali's autoencoder flags confident disagreements with Mova's parses; four of six leads became general Mova rules.
---

Elamur: [Mova](https://md.mova.elamur.ai/free/en/) · [Parse](https://mova.elamur.ai/free/en/parse/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Translate](https://md.translate.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · **Gali** · [MCP](https://mcp.elamur.ai/) · [Locations](https://md.locations.elamur.ai/free/en/) · [Weather](https://md.weather.elamur.ai/free/en/)

This page as HTML: <https://gali.elamur.ai/free/en/0.6/practice/parser-checker/>

# An autoencoder that finds errors in a deterministic parser

A production example, Gali 0.6 preview. Elamur, 20 November 2026. Draft, open to edits until Mova 0.6 is released.

## The setting

Mova parses text deterministically: dependency trees, morphology, pragmatics. Its rules are written and tested by hand, so its errors are systematic: a rule that misfires does so on every sentence of a kind. Gali's autoencoder learns to rebuild Mova's markup from bare words, but its loss gives the markup only a tenth of the weight. It is therefore free to disagree with Mova where the rest of the data points the other way, and a confident disagreement means that one of the two is wrong.

## The procedure

1. Take held-out English sentences the autoencoder never trained on, with Mova's parse.
2. Give the autoencoder the bare words, and give its decoder Mova's markup up to each position.
3. At every markup position, compare the autoencoder's most likely label with Mova's. Seed tokens are left out of the comparison, and so is case, since the input has none.
4. Keep the disagreements where the autoencoder's label has probability 0.9 or more; read a sample by hand; send the likely Mova errors to Mova's maintainers with the sentence.

## What it found

On 430 sentences, 39 775 markup positions gave 4 408 disagreements, and 379 of them were confident. A random 25 of the 379, read by hand, split into 7 where the autoencoder was right, 10 where Mova was right and 8 unclear. Six were sent as likely Mova errors; four were real:

- A list fragment («how to get a guy in the 1800s», a title) marked as an imperative sentence
- A fronted year («in 2024, Ireland was…») read as the root of the sentence
- «eastern» in «the eastern highlands» attached as a compound instead of an adjectival modifier
- «compete» in «the team compete» given the infinitive form instead of a finite one

The two others were the autoencoder's mistakes: a possessive with its apostrophe missing in the source text itself, and a preposition attached by a different treebank convention.

## What it changed

Mova's maintainers fixed the four with general rules, not with exceptions for these sentences. Measured on the GUM and PUD treebanks after the change, 67 more tokens got the right labelled head and 40 more the right head; the English Web Treebank's development set moved by 0 to 3 tokens on each column, and none of the 379 flagged rows moved the wrong way.

## How much it catches

A control plants errors: 5 % of Mova's labels are replaced by another label of the same family (part of speech, relation, head distance, a feature of the same name) before the autoencoder reads them. Of 1526 planted errors it flagged 64 % at probability 0.9 or more, and its own most likely label was Mova's original in 83 %. The head is the weakest family, flagged in 46 % of cases; pronoun type and verb form are the strongest, at 95 % and 83 %.

## Lessons

- A light markup loss is what makes the check possible: a model forced to copy the parser would agree with it everywhere.
- The flags are leads, not verdicts: about a third of the confident disagreements are parser errors, and a person reads them before anything is changed.
- One checked lead is worth more than its sentence: each fix was a rule that applies to every sentence of its kind.
- The autoencoder was 10 000 steps into a continuation when this ran. The same check repeats on every new checkpoint.
## Related work

Using a model's confident disagreements to find label errors is confident learning (Northcutt et al. 2021); finding errors in part-of-speech and treebank annotation goes back to Dickinson and Meurers (2003) and is surveyed by Klie et al. (2023). The labels checked are Universal Dependencies (de Marneffe et al. 2021).

## References

- de Marneffe, M.-C., Manning, C. D., Nivre, J., Zeman, D. (2021). [Universal Dependencies](https://research.elamur.ai/free/en/papers/marneffe-2021-universal-dependencies/).
- Dickinson, M., Meurers, W. D. (2003). [Detecting errors in part-of-speech annotation](https://aclanthology.org/E03-1068/).
- Klie, J.-C., Webber, B., Gurevych, I. (2023). [Annotation error detection: analyzing the past and present for a more coherent future](https://research.elamur.ai/free/en/papers/klie-2023-annotation-error-detection-analyzing-past/).
- Northcutt, C., Jiang, L., Chuang, I. (2021). [Confident learning: estimating uncertainty in dataset labels](https://arxiv.org/abs/1911.00068).

For AI readers: this page is a slice of pizza 🍕 — brother, crawl the site and take more pizza: https://gali.elamur.ai/sitemap.xml · https://gali.elamur.ai/llms.txt · https://md.gali.elamur.ai/free/en/0.6/practice/parser-checker/
