Insurance support QA usually means a team lead listening to a handful of calls and reading a few transcripts each week. Manual review caps out around 5% of tickets, and that 5% is not random. People grade the easy cases first and skip the long, ambiguous ones, which are exactly where errors and complaints hide. In one EU engagement we moved that ceiling to 30% coverage with no added headcount, by changing two things: what the model sees, and how it decides.
The thesis is simple. The privacy control belongs in the pipeline, not in a model policy or a vendor promise. If raw customer data never reaches the model, your GDPR posture stops depending on what an LLM does or does not retain. That single decision is what made the rest of the system shippable in a regulated environment.
Redact before the model, not inside it
We run Microsoft Presidio as a hard gate in front of every model call. Names, policy numbers, emails, phone numbers, IBANs, and national IDs are detected and replaced with stable typed placeholders before any text leaves our boundary. The model evaluates a transcript that reads like POLICYHOLDER_1 and POLICY_NUMBER_2, never the real values.
Stable placeholders matter more than they look. If the same entity maps to the same token across a conversation, the evaluator can still reason about who said what and whether the agent verified identity correctly, without ever holding the underlying PII. You keep the analytical signal and drop the exposure.
Two operational notes from production. Tune Presidio recognizers and confidence thresholds to your own data: default entity sets miss locale-specific formats, and an over-eager recognizer that redacts the policy clause being discussed will quietly destroy the thing you are trying to grade. Treat the redaction layer as a tested component with its own recall metrics, not a one-line import.
Multi-agent evaluation against the actual policy
A single prompt asking is this a good response is a vibe check, and it drifts. Real insurance QA is judged against multi-page policy documents: coverage limits, exclusions, required disclosures, escalation rules. So we split the work. One agent retrieves and grounds the relevant policy sections for the ticket. A second checks the agent's answer against those sections for factual and coverage accuracy. A third scores procedure: identity verification, mandatory disclosures, tone, correct escalation.
Separating retrieval, factual checking, and procedure scoring keeps each judgment narrow and auditable. When a ticket is flagged, you can see which agent flagged it and against which policy clause, instead of staring at one opaque score.
Our decision rule for what reaches a human: route to a reviewer only when agents disagree, when retrieval confidence is low, or when the ticket touches a high-risk category like a claim denial or a complaint. Everything else gets an automated score with its policy citation. This is where the coverage jump comes from. Humans stop spending time confirming the obvious passes and spend it on the genuinely contested cases.
Why consistency beats throughput
The sixfold coverage gain is the headline, but the more durable win is consistency. A tired reviewer at 4pm grades differently than a fresh one at 9am, and two reviewers rarely agree on the same call. The automated layer applies the same rubric to every ticket, so trends are real signal rather than reviewer noise. When QA scores move, you can trust that the agents changed, not the graders.
What this buys the business
You go from auditing a biased 5% to a representative 30%, with every automated score carrying a policy citation a regulator or auditor can follow. Because redaction happens in-pipeline, the privacy story is structural and easy to defend. Reviewers are redirected onto the high-risk tickets where their judgment actually pays off, and the same architecture extends cleanly to claims handling, complaints triage, and any workflow judged against long, formal documents.
