Document Assistant

Identify the table, not the document. Shape recognition that learns as it reads.

Document Assistant — shape recognition

Component documentation · Data Wrangler v4 Written 8 August 2026 against the docshape package and doc_flow.py. Every rule in section 9 was learned from a document that read wrongly.

1 · The idea

A scout ticket, a casing record and an end-of-well report look nothing alike. But the casing table inside each of them looks like every other casing table: a string type, a size, a set depth, a top of cement. The table has a shape, and the shape is stable even when the document isn’t.

That is the whole idea. Identify the table, not the document.

Everything else follows from it. If you identify tables rather than documents, you don’t need to recognise a vendor’s report format before you can read what’s in it. A layout nobody has seen still yields its casing table. And a new vendor costs a wording, not a parser.

2 · How it worked before, and why that stalled

This is worth setting out, because the previous approach is the obvious one and it fails in a way that’s hard to see.

Extraction used to be type-first. A PDF arrived, a classifier read its banner text and keyword-matched it into one of a handful of document types. Each type had a reader written for it: a scout-ticket reader that knew where a scout ticket puts its header, an end-of-well reader, a survey reader. Office files were routed by filename keywords — a name containing “tops” went to the formation-tops loader, “complet” to the completion loader. LAS and SEG-Y had their own parsers, and the whole per-format dispatch existed in two hand-synchronised copies, one for the interactive path and one for the worker pool.

It worked, for the documents it had been written for. The problems were structural:

A document had to be recognised before anything inside it could be read. Unclassified meant nothing extracted — not a partial result, nothing. A Final Well Report matched no filename keyword and returned “no loader found.”

The failure was silent and looked like success. The old path would resolve a well header, write a row, and then have nothing whatever to say about the tables inside the document. The run reported files processed and rows written. Scout tickets, casing records and end-of-well reports produced no detail rows at all, for weeks, without an error anywhere.

A new vendor layout meant new code. Not a configuration change — a Python change, tested and deployed, per layout.

And it was expensive. The classifier opened the file, the extended classifier opened it again, the extractor a third time, and the Office path re-parsed inside the loader. One run measured 526 seconds, of which extract and capture were 90%.

The scale of what was being missed only became measurable when the recogniser was switched on over the same corpus: capture went from 3 files to 8, and from a few hundred rows to 10,006.

The inversion. Instead of what document is this, and which reader handles it, the question became what tables are in this file, and what is each one. The reader’s job shrinks to producing tables from a format. Identification becomes a matching problem over the table’s own header — and a matching problem can be solved with a vocabulary, which is data, rather than a parser, which is code.

3 · What a shape is

A shape is a claim: a table with these fields in its header is a table of this kind.

frac_stage required : stage, top_md optional : base_md, clusters, fluid_vol, proppant, pressure, rate, date, … target : cat_well_stimulation

Required fields must discriminate, not merely be present. This is the single most important rule in the pack, and it took a real failure to learn: formation_tops requires only formation and top depth, which almost any depth-indexed table satisfies — so it claimed core runs, petrophysics and perforations. A new shape built on commonly-required fields loses to the general shape every time.

The tie-break is (score, optional-hits): at equal score, the shape that also explains more of the table wins. So a specific shape beats a greedy general one by covering more, which is the right reason.

The pack currently holds roughly 90 fields and 19 shapes, 14 of which name a target table.

4 · How matching works

A header cell is tokenised — punctuation stripped, characters mapped, noise words removed — and matched against each field’s aliases as a set.

Aliases are phrases and order is ignored. “Top of Cement”, “Cement Top” and “Cement, Top of” all match the same field. Interleaving is tolerated: “Top of the Cement Column (ft)” still matches.

Longest alias wins, so a two-token alias beats a one-token one.

Contiguity breaks a tie. When two aliases match with the same token count, the one appearing as an unbroken run in the header wins. That rule exists because “Top of Cement” was resolving to measured depth — “top of cement” and “top md” are both two tokens after noise removal, and dictionary order was deciding. A cement top would have been written silently into the casing string’s own depth column.

Values, not just headers, when the table is rotated. Many documents write a header as label/value pairs down the page rather than columns across it. Those are detected structurally by the reader and pivoted, and the labels are resolved with the pack’s own alias matching — one vocabulary used both ways, so a wording added for a column header works for the rotated form free.

And what it cannot do: one cell holding two attributes. “Top / Base MD (ft)” picks one. That’s a cell-splitting problem, not a matching one, and nothing does it today.

5 · Four layers, and what each is allowed to know

docshape is a separate top-level package with no imports from the rest of the platform. The dependency only runs one way, so it can become its own repository by moving the folder. Perry’s reason: “that is the most unique part of the system and could be deployed for many different groups.”

The abstraction is proven, not asserted. A legal pack — 26 fields, 6 shapes: payment schedules, parties, obligations, clause indexes — recognises legal tables at full confidence including a vendor variant, and correctly refuses frac stages and directional surveys. Zero engine changes. The same store generates the legal tables with the identical provenance and review block.

Two decisions inside that split are worth stating:

Review lives in the store, not the backends. A backend that knew about review status would need updating on every platform each time the review model changed.

Documents are keyed by content hash, not path. The same file captured from two folders is one document, and re-capture replaces rather than duplicates — which fixes by construction the class of failure where two copies of one sample set looked like two different sets.

6 · The vocabulary loop

The vocabulary is data, so it can be corrected without a deployment. It lives in three layers, applied left to right:

Base pack → considered, version-controlled, ships to every deployment. Overlay → a JSON file beside it; learned, local, promotable. Sandbox → experiments, on top of the overlay.

The merge rules are deliberate and asymmetric. Aliases extend — a correction adds a wording, never discards ones that worked. Shapes replace wholesale — a shape is one coherent claim, and half-overriding produces a definition nobody wrote. A shape can be disabled reversibly rather than deleted.

That asymmetry has a sharp edge, and it bit: after overlay entries were folded into the base pack, the overlay copies stayed — and because a shape replaces, the stale copies were actively subtracting fields from the improved base versions. Six shapes silently lost fields, which surfaced as columns going unmapped for no visible reason. After folding an overlay into the pack, the overlay copies must be removed.

The sandbox sits on top of the overlay, not instead of it — an experiment has to be judged against the established vocabulary, since testing a new wording against a stale one proves nothing.

Three ways to correct something

Point at it. Every unmapped column gets a dropdown of fields; choosing one records an alias — “this header wording means this field” — not a per-document fix. The next document from that vendor is right before anyone looks at it.

Describe it. A text box and a translate button: say what’s wrong in words, and the model returns proposed corrections in the same approval form, with the same checks.

Let it propose. A batch pass over unrecognised tables, ranked by how many documents each proposal would unlock.

All three land in one approval form with the same deterministic vetting — already-resolves, superset tie-break, discriminating-required, missing-field guards — and all three write to the sandbox, not to live vocabulary.

And a guard that stops a plausible mistake

Before offering to widen a shape to claim a column, the form checks whether another shape would explain more of the table. If one would, it says so and refuses to lead with the claim:

That check exists because a real screenshot showed a completion spreadsheet identified as a well header, with the claim form filled in and one click from cementing it. Widening the wrong shape is how a general shape quietly becomes a specific one for every future document.

7 · Two audiences, one page

Perry, after a day of using it: “I think the document assistant may be too complex for mere mortals.” He was right, and the day’s own evidence was that the author of the vocabulary got caught by a missing wording, an empty required list, and an unwinnable tie.

The page now separates two jobs that had been sharing a screen.

The everyday job is a per-document walk in plain language:

Two buttons — ✔ Looks right and ⚑ Something’s wrong — with an optional note. The file, table, header and samples attach automatically. Nothing else is on screen. A document that couldn’t be read says so plainly rather than showing an empty form.

Vocabulary engineering is behind a toggle, default off. That’s where the teach form, the correction channels, the sandbox and promotion live. It opens with the reader-reported flag queue first, because a report from someone who read the document beats a frequency ranking.

Perry’s summary of the split: “Simple: pass and collect $200. Fail — go to jail.”

8 · Working at scale

Opening documents one at a time doesn’t survive fifty thousand of them.

The census identifies across a whole folder or inventory and stamps a verdict per document — readable, partial, unknown-only, no tables, failed. It ranks the unknown table signatures by how many documents each would unlock, which is the teach-priority queue. Duplicates are detected by size then hash, so a document is assessed once and every copy is stamped.

It gave a false all-clear, once, and the fix is worth recording. A corpus reported 33 of 33 tables recognised and “no shape candidates” — a clean bill of health that was not one, because every real issue that week was column coverage inside tables that identified perfectly. Discovery only re-read documents with unknowns, so with zero unknowns it found nothing. Recognising a table says nothing about how much of it was kept. It now sweeps readable documents for unclaimed columns when there are no unknowns.

The expert loop closes the case where the answer isn’t in the building. A bundle exports the situation — pack name and version hash, local overlay, reader flags, unknown table groups with vendor wordings and section titles, near-misses, unclaimed columns — carrying table shapes, not well data. Alongside it, a baseline snapshot of what the pack reads today, and a checker that loads a candidate pack without installing it, runs static lints, and re-identifies every snapshot header as fixed, regressed or unchanged. It exits non-zero on a regression.

That last piece answers a real question: can a customer with no history get a vocabulary fix and know it didn’t break anything. The bundle carries the situation, the checker carries the proof, and the pack’s own comments carry the accumulated judgement — which is why those comments are load-bearing rather than decoration.

9 · The rules, each earned

A word that is both a unit and a term must never be noise. “Days On” made day a noise word, so the alias “day no” eroded to a bare “no” and matched any header containing “No”. Same failure with hrs, bbl, mD and $K. Four separate instances.

A noise word inside an alias makes the alias mean something shorter than it reads. “total depth md” erodes to “depth md”, which then hijacks a plain “Depth (ft MD)”.

An alias cannot win a tie — only pack order can. When a wording ties across two fields, adding it to the loser does nothing. The vetting warning “already resolves to X” is the tell, and it means this cannot be fixed from the UI at all.

Generic words are resolved by the shape, not the engine. “Status” means well status in a header and perforation status in a perforation table. “Result” means a test result in a DST and a parameter value in a key-value table. Same word, different meaning per table, decided where the context lives.

A garbled header is an extraction problem. “Avg VcHl y(%dr)” is not a vocabulary gap, and teaching it would bind the vocabulary to one PDF’s rendering defect. Say so; never invent an alias for it.

There are six reasons a column goes unmapped, and only two are fixable from the UI: the shape doesn’t list a field it understands · the vocabulary lacks the wording · the wording ties with an incumbent · another column already filled that field · it’s structurally the value or label of a key-value table · the header is garbled.

And on the reader side

Sorting characters left-to-right looks obviously right and corrupts overlapping cells. One PDF composed a header as “(psGi)radient” because two cells overlapped by five points. The content stream had the characters in the right order; only the positions overlapped. Group in stream order, treat a negative gap as a column boundary.

Some PDFs have no space glyphs at all — “FinalPressure”, “ftMD”. Re-spacing has to be unit-aware, or “Mobility(mD/cP)” becomes “Mobility(m D/c P)” and the unit is destroyed while the header appears fixed.

A placeholder row is the document saying a section is empty. “No survey data” must be dropped, not turned into a well test that never happened.

And a structural rule for pair grids: a real table’s value columns are type-consistent because they are columns; a pair grid’s same position holds an operator, then a UWI, then a state. Requiring mixed types before flattening is what stops a genuine table being collapsed into one row.

10 · What earns a place in the database

Recognition is not the same as correctness. A table can identify perfectly and still have a column pointed at the wrong field.

The design that addresses it rests on a distinction: wrong and missing are not the same failure. Missing is recoverable and visible — the value sits in the extras, the census names it, nothing in the database is false. Wrong is invisible and permanent: permeability filed as measured depth, net pay overwriting gross thickness, a combined figure in the oil column. All three were real, all found in one day, all silently true for weeks.

So the admission test is strict about wrongness and tolerant of incompleteness — refusing a table for one unclaimed column withholds good data to guard the lesser failure, and makes the gate unreachable.

Each failure names itself, and the work queue groups by reason rather than by document: one vocabulary change clears thirty tables. Held tables are stored and re-testable, so improving the vocabulary promotes them with no re-extraction.

There is also a measurement available that beats an estimate. The mapping tool exports a scout ticket as HTML — data that left the database — and that file reads back through the recogniser at full confidence. Comparing what returns against what went out gives a measured extraction fidelity per field, and it costs nothing because both halves already exist.

Status: the store carries the review machinery this would set. Nothing reads it yet. Migration out of the store is the piece that makes the capture → review → migrate loop real.

11 · Open items

Migration from the store into the database is not built — review status is written and nothing reads it.

Roughly 635 captured rows sit in shapes with no target: key-value pairs, curve readings, cement bond, operations NPT, fluid samples. Each needs a table and a decision, and the key-value ones may be better pivoted than stored as pairs.

One ruled-path PDF still mangles two headers. Diagnosed — the cell boxes themselves overlap, so rebuilding from characters catches the neighbour’s — and an attempted fix made it worse and was reverted. The real fix has to clip each cell’s characters against a neighbour that starts later.

Perforation and reservoir shapes don’t exist, so their mirrors stay empty. Whether to write them depends on whether the source documents carry those sections.

The census reports pair-grid value cells as unclaimed columns on shapes carrying a transform. Suppressed in the two reports that matter, but the underlying signal is still noisy.