An AI triage assistant clinicians trust

The challenge

The client — a clinical-AI company serving urgent-care and primary-care practices — had a real problem: patients fill out intake forms, but clinicians rarely read them carefully before walking into the room. Time pressure is the obvious reason. The subtler one is that intake forms are unstructured: paragraphs of free text, inconsistently formatted, with critical information buried in sentence three of the fourth field.

The company had explored a generic summarisation approach using a large language model. Early prototypes summarised fluently. But clinicians rejected them almost immediately. The core objection wasn't the quality of the summaries — it was the absence of traceability. "This says the patient reported no prior cardiac events, but where does that come from? I can't sign off on something I can't verify."

In a clinical context, a plausible hallucination is worse than no summary at all. The product needed to earn trust before it could earn adoption, and trust required a fundamentally different architecture than standard LLM summarisation.

What we built

We designed the assistant around retrieval-augmented generation with hard constraints on what the model was permitted to assert.

Every claim in the output was required to cite the specific intake field it came from. The system extracted structured attributes — chief complaint, symptom timeline, medication list, allergy flags, prior diagnoses — from the unstructured intake text using a combination of rules and a fine-tuned extraction layer. The LLM's job was to synthesize and present those extracted attributes, not to reason beyond them.

We introduced a validation layer that checked every output against the source fields before it was surfaced to a clinician. If the extraction confidence was below threshold for a given attribute, the field was flagged as "review required" rather than silently included. The clinician saw the summary and the underlying evidence side by side — the assistant never presented a claim without the source visible one click away.

The human-in-the-loop check was structural, not advisory. The UI surfaced a confirmation step before a triage summary was attached to a patient record. Clinicians could accept, edit, or reject any field in the summary. Edit and rejection events fed back into the evaluation pipeline, creating a continuous signal about where the model was underperforming.

The frontend was built in React with a focus on clinical workflow: keyboard-navigable, legible at the font sizes clinicians actually use on wall-mounted monitors, and fast enough to not add latency to an already pressured intake process.

The outcome

Triage time — measured from patient check-in to clinician entering the room — fell by 3.4× in the practices that rolled out the assistant fully. The primary driver was time to digest the intake: clinicians reported arriving at the summary faster than it took them to read the raw form.

Critically, the validated-output constraint held: zero unvalidated outputs were attached to patient records in production. The validation layer caught 23 low-confidence extractions in the first month and surfaced them for review rather than passing them through — the kind of edge case that would have been invisible in a standard summarisation approach.

Adoption was substantially higher than the client's previous AI tools, which the medical director attributed directly to traceability. Clinicians who understood how the summary was constructed were willing to rely on it; those who saw it as a black box were not. The architecture was the product strategy.