2.4 million latin word forms, and the 1.2 million that survived

you search virtus in a latin corpus and get four hits, in a text you know says it constantly. the word is everywhere. it just isn’t spelled virtus very often — it’s virtutem when it’s the object, virtutis when it’s possessive, virtute in the ablative, which among other jobs covers the thing something was done with, and uirtus if the edition uses classical orthography.

latin puts the grammar inside the word. a noun has twelve slots and a verb has a hundred finite forms before you even reach the participles, and a search index that matches strings finds none of them from the headword you actually typed.

i closed the tantivy post promising the machinery behind the lemma: operator that fixes this, in fontes antiquae — my archive of classical texts. here it is: 2,423,862 generated forms, 1,204,532 distinct keys after normalization, and the interesting half of the post is the gap between those two numbers.

first, a correction

in that post i described the source as “a public-domain dictionary file from the 1990s.” my own sources doc says that’s wrong, and i’d rather fix it here than let it stand.

the source is whitaker’s WORDS ↗ , william whitaker’s latin morphological analyzer, written in ada, worked on from the 1990s until his death in 2010 and maintained since as a preservation effort. its licence is a custom one, and unusually generous:

permission is hereby freely given for any and all use of program and data. you can sell it as your own, but at least tell me.

that is more permissive than most open licences in practice. it is not a public domain dedication, and it is not GPL or MIT either — it’s a paragraph of prose by a retired air force colonel who wanted people to use his life’s work. plenty of secondary sources round it to “public domain” and i repeated them without checking. the correct move is to cite the licence as written and credit whitaker by name, which costs nothing and is what he asked for.

two files, and the thing they deliberately don’t contain

WORDS ships its linguistics as data. two files matter:

DICTLINE.GEN   39,338 lines   one dictionary entry per line
INFLECTS.LAT    3,207 lines   1,785 parsed inflection rules

(there’s a third vendored file, and a fourth i had to write myself; both show up later.) DICTLINE.GEN is fixed-column: up to four stems per entry, a part of speech, morphological detail, frequency and era markers, and an english gloss. INFLECTS.LAT is the paradigm table — for each part of speech, declension or conjugation and variant, the list of endings and what each one means grammatically.

and here’s the design decision that made this a project rather than an import: the regular engine stores no complete word forms — only stems and endings. it works the other way round. give it uirtutem and it strips candidate endings, looks up matching stems, and reports the analyses that fit. it’s an analyzer, and it does that beautifully.

i needed the inverse. a search index wants to answer “what lemma is this token?” in constant time, at query time, for every word in a passage a reader clicks. running an ada-style stem-and-ending search per token per query is the wrong shape for that. so the job is to run the cartesian product offline, once — every stem against every ending its paradigm allows — and store the result as a lookup table.

the expansion, and what it actually produces

expand_entry takes a parsed dictionary entry and an index of inflection rules and returns every surface form that entry can produce. run it across all 39,338 entries and count:

dictline entries      : 39338
inflection rules      : 1785
generated forms (all) : 2423862
distinct normalized   : 1204532
entries yielding none : 41

two point four million forms. the “multi-million” in my promise was, for once, not an exaggeration.

but look at the second line. after normalization, 2,423,862 forms collapse to 1,204,532 distinct keys. just over half of every form this thing generates is a string some other form already produced. that ratio is not a bug in the expansion and it isn’t waste. it is the entire linguistic problem, showing up as a number.

normalization is a decision, not a cleanup step

before anything can collide, every form goes through one function that both the offline build and the runtime lookup call:

crates/fontes-lemma/src/normalize.rs

pub fn normalize(input: &str) -> String {
    let mut out = String::with_capacity(input.len());
    for ch in input.nfd() {
        // Drop combining diacritical marks left behind by NFD decomposition.
        if matches!(ch, '\u{0300}'..='\u{036F}') {
            continue;
        }
        for lower in ch.to_lowercase() {
            out.push(match lower {
                'j' => 'i',
                'v' => 'u',
                other => other,
            });
        }
    }
    out
}

three rules: decompose and drop diacritics, lowercase, and fold j→i, v→u.

the third is the one that isn’t obvious, and it’s a claim about latin rather than about strings. classical latin wrote one letter where we now write two: ⟨V⟩ served both the vowel and the consonant, and ⟨I⟩ likewise. splitting each into a pair is an early-modern orthographic reform, proposed for the vernaculars first ↗ and applied to latin printing much later.

the two halves settled differently. the i/j split is simply gone from scholarly latin — the oxford latin dictionary heads uir, not vir. u/v is the one editors still genuinely disagree about: one edition prints virtus, another prints uirtus, and they are the same word. whitaker’s data treats them as one letter, so the lexicon has to as well, and so does the reader clicking a word in whichever edition i happened to ingest.

normalize("Virtūs") and normalize("uirtus") both give uirtus. that’s not tidying; that’s an editorial position, implemented in about a dozen lines.

note what it deliberately doesn’t do:

the digraphs æ / œ are deliberately not folded here — they are handled by the build-time lemma reconciliation step, not runtime normalization, and XIX-century dictionaries distinguish them in their headwords.

and note that this is more aggressive than the search index’s own folding, which strips diacritics and leaves both j/i and v/u distinct. two normalizers, on purpose, because they answer different questions — which is exactly the kind of thing that becomes a silent bug if you let one drift, and the last post was about what that silence costs .

the actual hard part: liber is five words

now the collisions. they come in two flavours and only one of them is harmless.

the harmless kind is syncretism. latin paradigms genuinely reuse endings. uirtus is both nominative and vocative singular; a first-declension noun’s genitive singular and nominative plural are the same string. one word, one lemma, several grammatical slots wearing the same surface. these collapse and nothing is lost, because the lemma is the same on both sides.

the other kind is a homograph, and it’s a real ambiguity. i went into this expecting the textbook example — liber the noun, “book”, versus liber the adjective, “free”. then i grepped the dictionary for it:

25628  liber   liber                              N    2 3 M P     children
25629  liber   liber   liberav  liberat           V    1 1         to free
25630  liber   liber   liberi   liberri           ADJ  1 2         free
25631  liber   liber   liberi   liberri           ADJ  1 2         (…)
25632  liber   libr                               N    2 3 M T     book, volume

five entries. a noun meaning children, a verb meaning to free, two adjectives, and the book. and there’s a detail in there that decides the whole schema: 25630 and 25631 are both ADJ 1 2. same spelling, same part of speech, two dictionary entries. part of speech is not enough to tell them apart.

(there’s also a sixth liber the schema never sees — Līber, the god. and a seventh distinction that doesn’t exist: lewis & short file “bark” and “book” as one word, bark being the literal sense. i was going to list it separately and the dictionary talked me out of it.)

so the schema refuses to make the lemma key (lemma_norm). it’s (lemma_norm, pos, homograph_idx):

migrations/…_lemma_lexicon.sql

-- headword part of speech (Whitaker taxonomy, lowercased). Closed CHECK to
-- catch parser/seed drift; PACK is a real headword POS (e.g. quicumque).
pos              TEXT NOT NULL CHECK (pos IN
    ('n', 'v', 'adj', 'adv', 'num', 'pron', 'pack', 'prep', 'conj', 'interj')),
-- homograph discriminator within (lemma_norm, pos); 0 when unique.
homograph_idx    SMALLINT NOT NULL DEFAULT 0,

part of speech separates the noun from the verb from the adjective. homograph_idx is what handles 25630 versus 25631 — the case that agrees on both spelling and part of speech and is still two words. it’s assigned by order of appearance in the dictionary:

crates/fontes-cli/src/commands/lemmatize.rs

let counter = homographs.entry((lemma_norm.clone(), pos)).or_insert(0);
let homograph_idx = *counter;
*counter += 1;

i want to be honest about what that is: it’s a stable identifier, not a linguistic judgement. it says “this is the second X whitaker lists”, which is reproducible and useless to a reader. presenting liber² to someone reading cicero would be worse than useless. what makes it defensible is that the row also carries the gloss and the citation morphology, so the ambiguity gets shown rather than silently resolved — the popover ranks the candidates and offers them, and a human picks. the alternative, choosing one and hiding the other, is the version that’s confidently wrong.

there’s one more twist, and it’s the cleanest illustration of why this is a lexicon problem rather than a string problem: lĭber the noun and līber the adjective differ in vowel length. they were never really the same word to a roman ear. but macrons are an editorial convenience that transmitted texts don’t carry, and my own normalizer strips them in rule one — so the information that would disambiguate these is present in the dictionary and absent from the corpus. no amount of cleverness applied to the text recovers it. you need the lexicon.

which is the same position the two-axis trust post landed on from a completely different direction: when the data genuinely doesn’t know, the honest move is to make the not-knowing sayable in the schema rather than resolve it with a coin flip.

the 41 entries that produce nothing, and the verb that isn’t there

entries yielding none : 41. forty-one dictionary entries expand to zero forms, and they’re a reminder that a paradigm engine only covers the regular part of a language.

that’s part of what UNIQUES.LAT is for: complete, hand-listed surface forms that no rule generates. it’s 228 lines, which is 76 records — the format is three lines each, and i nearly reported the line count as the record count, which would have inflated it threefold. the crate’s own test pins the real number:

assert_eq!(records.len(), 76);

but here’s the thing i actually didn’t expect. i assumed those 76 would be the suppletive verbs, and above all sum — “to be”, the most common verb in the language, whose paradigm famously runs sum, es, est … fui, fuisti. it isn’t there. it isn’t in DICTLINE.GEN either. my own build code says so:

crates/fontes-cli/src/commands/lemmatize.rs

// The copula `sum`/`esse` is absent from DICTLINE and UNIQUES alike (Whitaker
// keeps it in its compiled irregular engine); mint its lemma, then load its
// authored paradigm and attach every form to it.

WORDS handles the copula in compiled ada, not in data. so the dictionary you can parse simply does not contain the most frequent verb in latin, and the only way to get it into a precomputed table is to type it out. that’s the fourth file i mentioned at the top: supplement/copula.lat, 73 records, hand-authored, loaded with source = 'manual' so it’s flagged as mine rather than whitaker’s.

sum and fui are genuinely two roots wearing one meaning — the perfect/imperfective split that shows up across indo-european. sum, es and est are the same root, mangled by sound change past the point where any suffix-stripping rule connects them. that’s the more damning case, actually: it isn’t that string surgery fails on unrelated forms, it’s that it fails on forms that are related.

so the tail here is 76 irregular records plus 73 i wrote myself against 39,338 dictionary entries — well under half a percent of the data. and it is the half-percent you cannot skip, because it contains “to be”. a lexicon that covers 99.6% of latin and misses est fails on the first sentence anybody searches.

why any of this beats a stemmer

the obvious objection: tantivy ships stemmers. why not point one at latin and go home.

you can’t. tantivy’s Language enum ↗ offers eighteen languages — arabic through turkish — and latin is not among them. neither is it in rust-stemmers underneath, and snowball ↗ has no native latin algorithm either. the option i was going to argue against doesn’t exist.

what does exist is instructive. snowball lists the schinke latin stemmer under “other algorithms”, and reading it is the best argument for a lexicon i could have written myself:

• it emits two stems for every word, a noun stem and a verb stem, because it cannot tell which one it’s looking at. that’s not a stemmer output, that’s a shrug with two branches.
• it needs a hardcoded exemption list of about eighty words so the enclitic -que isn’t chopped off words that merely end in those letters.
• and porter’s own note on the page says he was “unable to make the rules match the examples given.” ↗

the only latin stemmer anybody wrote couldn’t stay a stemmer. it had to start guessing part of speech, which is the job a dictionary does, and it did it badly because it had no dictionary.

that’s the structural point rather than a cheap shot. a stemmer chops: suffix-stripping rules that turn virtutem into virtut and hope the same chop lands on virtutis. for english that works well enough, because english morphology is thin. latin defeats it in both directions at once — it under-collapses related forms whose stems change (sum/est again) and over-collapses unrelated words that happen to chop alike. and it can’t tell you which word it found, because a stem isn’t a dictionary entry, it’s a substring.

a lexicon gives you the lemma itself: a real headword, a real part of speech, a real gloss, real morphology to show the reader. that’s the difference between a search box that matches strings and a reader that can tell you uirtutem is accusative singular of a third-declension feminine noun meaning virtue. the second one is the product.

the cost is honest: two and a half million rows to build, a table to keep in postgres, and a rebuild whenever the source data changes. i’ll take it. the table is derived and rebuildable like everything else in that project — fontes-cli lemmatize truncates and reloads, and if the whole thing burns down the fix is “run it again,” not “restore a backup.”

lessons learned

• a normalizer is a linguistic position with a function signature. folding v→u isn’t string hygiene, it’s a claim that two editorial traditions are printing the same word. write those rules down in prose next to the code, because the next person — including you — will read nine lines of match and not see the argument inside it.
• the collision rate is the interesting metric. 2.4 million forms in, 1.2 million keys out — 50.3% duplicates. i’d have shipped the expansion without ever computing that ratio, and it turned out to be the number that describes the actual problem.
• separate “the same string” from “the same word”, in the schema. (lemma_norm, pos, homograph_idx) exists because two dictionary entries can agree on spelling and part of speech and still be different words. any key that’s just the normalized string quietly merges them and you never find out.
• a stable discriminator is not a disambiguation. homograph_idx orders entries reproducibly and says nothing about meaning. that’s fine as long as the ambiguity is surfaced to a human rather than resolved by index order — and it stops being fine the moment something downstream picks index 0 and calls it the answer.
• inverting a tool’s design is a legitimate use of it. WORDS is an analyzer whose regular engine deliberately stores no forms. running its data through a cartesian product to get the exact thing it chose not to build is not fighting the tool; the licence explicitly invites it, and the data was structured well enough to survive the inversion.

next, a change of subject entirely. eighteen months ago i called llm coding a dopamine trap and i still think the diagnosis was right — but i’ve since let one loose in a go codebase with a live database, and the thing that made that survivable wasn’t better prompting. it was a handful of shell scripts that say no. hooks, blocked commands, and the migration file an agent is structurally not allowed to edit.