clean and verified are different claims: a two-axis trust state machine in postgres

ask a language model for a line of tacitus in spanish and it will hand you a sentence. fluent, confident, correctly punctuated. and sometimes the sentence contains a word that no translator ever wrote — a word a scanner invented, when it read the á in a 1919 printing as the digit 4, or turned violento into violen10, and then a spell-checker came along afterward and smoothed the wreckage into something that looks like spanish.

the text is clean. it is not correct. and nothing about the way it reads tells you which.

i keep a personal digital archive of classical texts — latin and greek originals with their public-domain translations, meant as a small gift to the commons rather than a business. rust, axum, sqlx, postgres, a tantivy index. the corpus is a couple hundred hand-assembled files, and most of the spanish in it is exactly what i described above: my own ocr of old scans, uncorrected, known-corrupt.

for a while it had one column that tried to answer everything: copyright_status. and the longer i stared at it the more i realised it was answering two completely different questions with the same word, and getting both of them wrong. this is the post about splitting it in two — and about why the honest version of that split ends with a database constraint, not a code review.

two questions wearing one column

here are the two questions, and the thing that took me too long to see is that they have nothing to do with each other:

• may we publish this at all? a rights question. is the translator dead long enough that their work is public domain? this is about lawyers, not scanners.
• how faithful is this text to the edition it claims to reproduce? a fidelity question. is this a clean collation or is it ocr sludge? this is about scanners, not lawyers.

collapse them into one field and you make two real states inexpressible. a text can be perfectly clean but of unclear rights — a modern proofread edition still under copyright. and a text can be unimpeachably public domain but garbage ocr — a scan of an 1830 printing that no human has ever checked. my corpus contains both. one column cannot hold both, so it lied about one of them every time.

so: two axes, two enums, each in its own rust type, in a crate both the ingest cli and the web server depend on so the two sides can’t drift.

crates/content/src/lib.rs

/// May this translation be published at all?
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RightsStatus {
    /// Translator's death year proven ≤1945 against an authority record.
    PublicDomain,
    /// Rights not established. Never published, never ingested.
    Unverified,
}

the rights axis is a plain two-state pair. proven public domain, or not — and “we could not find a death date” is not a licence, it’s Unverified, which means unpublished. note what this enum does not derive: PartialOrd, Ord. rights aren’t ordered. one is publishable and one isn’t; there’s no “more public domain than.”

the fidelity axis is an ordered state machine

the other axis is where it gets interesting, because fidelity is ordered — weakest to strongest — and the ordering is the entire load-bearing idea:

crates/content/src/lib.rs

/// How faithful is this text to the edition it claims to reproduce?
///
/// Ordered weakest to strongest.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum VerificationStatus {
    RawOcr,
    MachineCorrected,
    DraftImport,
    ExternallyProofread,
    HumanVerified,
}

raw_ocr is my tesseract output, uncorrected. machine_corrected is that ocr run through automatic cleanup — and it sits only one rung up, on purpose, because machine correction is more dangerous, not less: it turns an obvious error (violen10) into a plausible one (violento? violenta? it picked one). draft_import is third-party digital text i haven’t audited. externally_proofread is a text a documented third party checked against the source — project gutenberg’s proofreaders, a scholarly digitisation. and human_verified is the top: collated by me, passage by passage, against the linked facsimile.

the enum derives Ord. so every trust decision in the system becomes a comparison on one axis, and there are exactly two of them:

crates/content/src/lib.rs

pub fn is_indexable(self) -> bool {
    self >= Self::ExternallyProofread
}

pub fn is_citable(self) -> bool {
    self == Self::HumanVerified
}

indexable — may a human find this? — needs a human to have proofread it, ours or a third party’s. searching corrupt text is worse than useless: it silently fails to match the very words the corruption ate. citable — may a machine quote this as authoritative? — needs my own collation. nothing else earns it.

and there it is. two thresholds on the same ordering, and they are not the same threshold. that gap between them is not an accident. the gap is the product.

the gap between searchable and citable is the product

here is the claim i actually want to make: “searchable” and “citable” are orthogonal, and the space between them is where an honest archive lives.

a human reader is allowed into that gap. they can read an externally-proofread text — even a raw-ocr one — because the page shows them a banner that says, in plain spanish, this is unverified, don’t cite it. the reader gets un-collated text plus context, and context is a thing a human can act on.

crates/web/src/handlers/reader.rs

Some(s) => Self {
    show: !s.is_citable(),
    label: s.label_es().to_owned(),
    notice: s.notice_es().to_owned(),

the banner shows for everything that isn’t human_verified. today that’s every single work in the corpus — which is the truth, and saying it out loud is the whole point.

now the other half, the half that made me build all of this. an llm cannot read a banner. neither can a sitemap. neither can a bibtex record. they consume the machine-readable layer, and the machine-readable layer has no margin for “read the warning first.” so those surfaces get gated on the strict threshold — is_citable — and served nothing but verified text. the search index is populated with the same asymmetry, straight in sql:

crates/cli/src/commands/ingest.rs

WHERE t.language = 'es' AND t.text <> '' AND t.status = ANY($1)

where $1 is [externally_proofread, human_verified]. everything below the bar is withheld, and the ingest logs the count it withheld, because an exclusion that’s invisible in the output will pass for a healthy index right up until someone notices search returns nothing.

the subtle one is the citation export. the tempting move, when a text isn’t citable, is to withhold the machine-readable metadata entirely. that’s exactly wrong. deleting the schema.org block doesn’t stop an llm citing the page — it just deletes my one chance to tell it not to. so the warning goes inside the metadata:

crates/web/src/handlers/core.rs

"creativeWorkStatus": if machine_citable { "Published" } else { "Draft" },
"disambiguatingDescription": verification_note,

creativeWorkStatus is a real schema.org property. a crawler that reads structured data at all reads this. same trick in the bibtex export: an uncitable text still exports, but its note field carries a shouted NO ES CITABLE caveat that rides along into whatever reference manager slurps it up. you can’t stop the machines reading you. you can refuse to hand them a clean quote, and you can make the warning un-strippable.

check constraints as a structural backstop

so far this is all application logic — enums, if statements, query filters. and application logic regresses. someone refactors the ingest, an is_publishable() check moves, and six months later a text that should never have existed is a row in translations, being served to a crawler.

which is why the real enforcement isn’t in rust at all. it’s a postgres check constraint ↗ , and it says the quiet part structurally:

migrations/20260714000001_translation_verification_status.sql

ALTER TABLE translations
    ADD CONSTRAINT translations_copyright_status_check
    CHECK (copyright_status = 'public_domain');

ALTER TABLE translations
    ADD CONSTRAINT translations_status_check
    CHECK (status IN (
        'raw_ocr',
        'machine_corrected',
        'draft_import',
        'externally_proofread',
        'human_verified'
    ));

the ingest already refuses a non-public-domain translation, in rust, before the insert:

crates/cli/src/commands/ingest.rs

if !rights.is_publishable() {
    tracing::warn!(
        path = %file_path.display(),
        rights = rights.as_str(),
        "REFUSING to ingest: rights not cleared. text will not be published."
    );
    return Ok(());
}

so why the check constraint too? because this is defence in depth , and i learned it the embarrassing way. the corpus contained one text — a cicero translation whose last surviving co-translator’s death year is simply unknown — that its own metadata annotates NO PUBLICAR. and an earlier ingest read that string, passed it to the insert, and published the text anyway. the database only looked clean because nobody had re-run the ingest since the file was flagged. the guard existed in principle and failed in practice.

the check constraint makes the guard structural. a non-public-domain translation cannot become a row even if the application guard regresses, because postgres itself rejects the insert. the app layer is the polite refusal with a nice log line; the constraint is the wall behind it. that cicero text is blocked twice now, and the second block doesn’t depend on me not breaking the first.

one genuinely load-bearing detail, from those same official docs ↗ : a check constraint is satisfied if the expression evaluates to true or the null value. CHECK (copyright_status = 'public_domain') passes happily on a NULL. the thing that actually slams that door is the NOT NULL on the column. a check is only as strict as its nullability; forget that and your wall has a NULL-shaped hole in it.

(all the queries here are sqlx compile-time-checked ↗ — the macro verifies them against a real schema at build time, so the column and enum spellings can’t rot silently against the migration. it’s the same instinct as the check constraint, one layer up: let a machine that isn’t me hold the invariant.)

the honest part: the number is zero

the blog rule around here is i don’t get to pretend. so.

the real count of human_verified texts in this corpus is zero. nothing has ever been collated against a facsimile. the migration backfills every existing row to raw_ocr or, for the third-party-proofread english, externally_proofread — and it deliberately backfills nothing to the top state:

migrations/20260714000001_translation_verification_status.sql

UPDATE translations SET status = 'raw_ocr' WHERE language = 'es';
UPDATE translations SET status = 'externally_proofread' WHERE language <> 'es';

i built a five-state citability machine to guard a citable corpus of size zero. that sounds absurd until you flip it: the machine is what makes it safe to sit at zero honestly, instead of quietly letting corrupt ocr leak into a bibtex file because there was no state to stop it. the archive earns trust by showing that its verified set is empty, not by pretending otherwise. the first text i hand-collate will be the first one the machines are ever allowed to quote.

the single-valued rights check costs a migration to change. CHECK (copyright_status = 'public_domain') admits exactly one value. the day a text arrives under a real license rather than public domain, that’s a schema migration, not a config toggle. that’s deliberate — a rights policy change should be a reviewed, versioned event and not a string someone types into a yaml file — but it’s a real rigidity and i’m not going to dress it up as a feature it isn’t.

the enum ordering lives in rust and can’t be pushed into sql. the “weakest state present” for a work can’t be computed with MIN(status) in postgres, because status is TEXT and MIN sorts it alphabetically — and alphabetically draft_import < raw_ocr, which would label a corrupt work with the stronger of the two states it holds. so i ARRAY_AGG(DISTINCT status) in sql and reduce to the minimum in rust, where the real ordering lives. the meaningful order exists in exactly one place, the enum — and the price is that postgres can’t sort by it, so a chunk of logic i’d love to keep in the query has to climb back into the application. the enum is the source of truth and also the reason the database can’t reason about its own column.

none of these are bugs. they’re the shape of a system that decided clean and verified are different claims, and refused to let the first one impersonate the second.

lessons learned

• one column, one question. the instant a field answers two questions it starts lying about one of them. rights and fidelity failed independently in my corpus, so they had to be independent columns — and splitting them made two true states sayable that were unsayable before.
• put the invariant where the regression can’t reach. the app-layer guard is where you’re polite; the check constraint is where you’re safe. i only believe a rule once a machine that isn’t my future self is holding it — a check constraint ↗ , a compile-time query, a NOT NULL. same lesson as the hash-chain post : invariants belong in the database.
• the gap between two thresholds can be the whole product. searchable and citable aren’t the same bar, and refusing to conflate them is what lets a human read an unverified text with context while a machine gets nothing but verified text. the honesty is the feature.

next: the tantivy side of this — how the search index gets rebuilt from the content tree as the derived, throwaway thing it is, and why the database being “the source of truth” is itself a claim i had to earn.

the code isn’t public yet. i’m at github.com/krtffl ↗ if you want to argue with any of this in the meantime.