Classification was only one step
The obvious framing of this problem is “classify the text against a standards model”, and that framing is wrong in a way that takes about a month to discover.
The input isn’t text. It’s a course document: multi-column layouts, tables that carry the actual content, scanned pages, prose that trails off, and half a dozen different ways of phrasing the same learning outcome. By the time you have something a classifier can read, most of the difficulty has already happened.
And the output isn’t a label. It can contribute to a certification decision about somebody’s course, which means “the model was 0.87 confident” is not a defensible answer. The system had to preserve the whole path — uploaded evidence, extracted claim, proposed mapping, human correction, aggregate analysis, final report — and be able to show it.
Constraints that shaped it
Source material arrives as whatever the partner has. Layout-heavy PDFs and word-processing documents, sometimes scanned. You don’t get to specify the input format when the input format is the thing they already wrote.
Two failure modes, two recovery paths. A page that didn’t convert properly and a claim that was mapped to the wrong competency look identical in the output and need entirely different responses. Handling them in one stage means every problem gets diagnosed as a model problem.
The standards model is hierarchical and versioned. A mapping is only meaningful against a specific version, and versions move.
Reviewers must be able to correct machine output without losing provenance. If a correction overwrites the suggestion, you can no longer explain how the conclusion was reached — or measure how often the machine was wrong.
Decisions, and what each one cost
Separate extraction from interpretation
Conversion, layout analysis, OCR and candidate detection run to completion before anything semantic happens. A bad source page can be retried, re-OCR’d or corrected by a human without being mistaken for a classification error.
What it cost: an extra persisted stage, extra state transitions, and a longer wall-clock path from upload to result. The single-pass alternative is genuinely simpler right up to the first support conversation about why a document “wasn’t understood”.
Keep every intermediate artefact
Candidate text, machine suggestions, reviewer changes and processing state are all retained rather than replaced.
What it cost: a real data-volume and lifecycle problem — this is the largest storage consumer in the system, and retention rules for source documents and intermediates are a product decision nobody wants to make. In exchange, the platform can explain how an aggregate conclusion was assembled, which is the entire value proposition.
Let models propose, never certify
Embedding and classifier output narrows the search space inside the hierarchical standards model. A reviewer confirms or corrects each mapping before it counts toward coverage.
What it cost: full automation is off the table, permanently. Throughput is bounded by reviewer time, and that is the honest ceiling of the product. I’d argue the friction is the trust model — but it is a cost, and pretending otherwise would be a sales pitch.
Queue the specialist work
Separately deployable workers claim tasks, report progress, recover stale locks, bound their attempts and shut down cleanly. The web applications coordinate the workflow; they never do model work inline.
What it cost: operational machinery that a lean team has to keep running — queue depth, stuck jobs, worker deploys. A synchronous version would have had none of that and would have fallen over on the first hundred-page document.
Calculate analysis from confirmed evidence only
Breadth, depth, coverage and gaps are computed from the reviewed mapping set, not the raw machine output.
What it cost: a report can’t be produced until review is finished, so there’s no “instant draft” mode. The number in the report is reproducible, which is what makes it usable in a decision.
What changed
A subjective, document-heavy review became a structured workflow. Partners submit the material they already have; machine assistance takes the tedium out of extraction and mapping; reviewers keep decision authority; and the output shows both what is covered and what is missing — which is the half that people actually act on.
What I’d do differently
I’d build a versioned evaluation set for extraction and for mapping quality on day one, and I’d surface per-stage confidence and drift in the product rather than in a notebook. Without that you can’t answer “is it getting better?”, and that question always arrives.
I’d make reviewer disagreement measurable. Every correction is a labelled example and we treated them as corrections rather than as data.
And I’d attach provenance from every statement in the report back to a source page and a confirmed mapping. We could reconstruct it; we couldn’t click it. Those are not the same thing when someone is challenging a result.