The problem was never the assessment
Scoring a test is not hard. Behaving like a dependable component inside somebody else’s operating model is.
Partners arrived with learners already enrolled, identity systems already chosen, reporting expectations already set with their own stakeholders, and a release calendar that had nothing to do with ours. The platform had to slot into all of that — repeatedly, for organisations that agreed on very little — without a separate codebase per organisation.
Some of it also involved young learners. That doesn’t make the engineering exotic, but it does reorder the priorities: reproducibility, role boundaries, guardian activation flows, localisation and traceable delivery of results all matter considerably more than anything clever in the scoring path.
Constraints that shaped it
A lean team owned product, integration and operations. Every architectural choice was also a staffing choice. Anything that needed a dedicated operator to keep alive was, in practice, unavailable.
Every tenant wanted something different. Different modules, languages, branding, access rules and callback behaviour — from the same platform, on their own schedule.
Results had to be replayable. The same submission against the same version must produce the same result, years later, when someone asks.
Partner systems are unreliable in every direction. Slow, duplicated, temporarily gone, or arriving out of sequence. None of those may lose a result.
Decisions, and what each one cost
One shared model, configured per tenant
Modules, locale, interface behaviour, access rules and partner mappings live in metadata. Onboarding a new organisation is a mapping and configuration exercise, not a fork.
What it cost: complexity moved out of code and into metadata quality and the administration tooling that maintains it. A misconfigured tenant is now a support problem rather than a compile error, and bad metadata fails later and less obviously than bad code. That trade only pays if you invest in the admin surface, and we under-invested in it for longer than we should have.
A signed integration contract at the boundary
Tenant credentials, request signatures, timestamp freshness, locale validation, duplicate protection and stable submission identifiers form a narrow, explicit trust boundary around the partner API.
What it cost: integration is harder on day one. A partner who just wants to POST some JSON has to implement signing first, and that conversation happens at the least convenient moment in the relationship. The alternative is trusting the network, which is not a decision you get to revisit later.
Accept, persist, then deliver
Scoring and partner callbacks were moved off the original request. Delivery has its own state: attempts, status, inspection and controlled replay.
What it cost: a whole category of operational machinery that wouldn’t exist otherwise — stuck deliveries, retry storms, someone having to look at a queue. Failure became visible and recoverable instead of silent, which is worth it, but “visible” means somebody has to look.
Deterministic scoring; AI kept elsewhere
There is always pressure to put a model in the scoring path. I held a hard line: scoring is deterministic and versioned; the probabilistic work lives in authoring and content analysis, where variation can be reviewed by a person before it matters.
What it cost: scoring rules are slower to change, because changing them means versioning them and reasoning about everything already scored. That friction is the point, but it is friction, and it was argued about more than once.
Encrypt end to end, and design around what that costs
End-to-end encryption is easy to add and hard to live with, because the moment data is opaque to the server you lose the ability to query it. Every feature that would have been a database index becomes an application problem, and some of them stop being possible at all.
The resolution was to be deliberate about what is genuinely sensitive rather than reaching for a blanket. Identity-linked responses and records travel inside a key-scoped crypto envelope, credential transport is protected separately, and the analytics store receives de-identified, aggregate-shaped data that was never joined to a person in the first place. That way the encrypted set is small enough to defend and the reportable set is queryable.
What it cost: some product ideas simply became unavailable, and you have to say so out loud early — before someone promises a feature that would require reading data the server can’t read. It also means the coverage is a property of configuration rather than of the codebase, which is exactly the kind of security property that is hard to hold in your head. Knowing precisely where the envelope was switched on required reading config, and I’d fix that (see below) before anything else.
Cache the shared vocabulary, not arbitrary learner state
Tenant and standards metadata are read constantly and change deliberately, which makes them right for namespaced read-through caching. Learner state stayed authoritative in its own store.
What it cost: invalidation, handled by refresh-on-write and explicit reload paths. The temptation is always to cache one more thing; every time we did, the bug that followed was a stale read.
What changed
The product became a reusable integration surface rather than a collection of tenant-specific deployments. New organisations could be expressed as mappings and configuration, learners could resume work, institutions could inspect outcomes, and partner systems received signed, traceable results they could reconcile against their own records.
What I’d do differently
I’d make the integration contract schema-first, and ship contract tests a partner can run against themselves before launch. Most integration pain was discovered during integration, which is the most expensive possible time to discover it.
I’d move all post-submission work onto a single durable event backbone rather than several purpose-built paths that grew separately, and attach correlation IDs and delivery telemetry to every result so a partner’s “we never received it” can be answered in seconds rather than by reading logs.
And I’d keep an explicit coverage matrix for encrypted application payloads instead of relying on route-level enablement. The mechanism was there; knowing precisely where it was switched on required reading configuration, and that is not a good way to hold a security property.