---
title: "95.2% recall is not proof that PII is gone. Redaction needs a reconciliation gate."
description: "AWS raised PII-redaction recall from 89.3% to 95.2% by matching repeated tokens against full-page output. The production lesson is stricter: document pipelines need measured residual-risk thresholds and reconciliation before release."
url: "https://devencelab.com/insights/2026/09/17/95-2-recall-is-not-proof-that-pii-is"
date: "2026-09-17"
section: "Insights"
tag: "Model Releases"
author: "Devence Lab"
reading_time: "2 min read"
site: "Devence Lab"
license: "Readable and quotable with attribution to the canonical URL."
---

# 95.2% recall is not proof that PII is gone. Redaction needs a reconciliation gate.

AWS raised PII-redaction recall from 89.3% to 95.2% by matching repeated tokens against full-page output. The production lesson is stricter: document pipelines need measured residual-risk thresholds and reconciliation before release.

AWS reported on 16 September that its Bedrock Data Automation redaction design raised recall from 89.3% to 95.2% after adding token matching to a field-specific blueprint. Precision moved from 97.0% to 96.5%. Those numbers make the production trade-off visible: finding more sensitive data can also redact more material that should remain.

The interesting part is not that a generative document service can draw black boxes. It is that AWS treats redaction as a measured pipeline rather than a single model decision. For regulated document flows, that should become the deployment pattern.

## A confidence score is not a release decision

AWS tested 12 documents across 47 pages, ranging from clean typed forms to handwriting and 100 DPI scans. The first blueprint pass could identify declared fields, but repeated PII in narrative text was sometimes missed. A second matching step reused the standard word-level output from the same API call to find those repeated tokens elsewhere on the page.

That matters because redaction failure is asymmetric. A false positive can damage document utility; a false negative can disclose personal data. Teams therefore need separate acceptance thresholds for precision and recall, tested against representative document quality rather than one aggregate accuracy score.

> Redaction is complete only when the pipeline can account for what it received, what it removed and what still needs human judgement.

## Ground truth belongs in the release process

The AWS example evaluates output against human-redacted ground truth before promoting the blueprint. That is the control worth copying. A blueprint or prompt change should be versioned like code, evaluated against a fixed regression set and prevented from reaching production when recall falls below the organisation's tolerance.

The regression set also needs ugly inputs. AWS deliberately included fax artefacts, handwriting, rescanned pages and low-resolution scans. A benchmark built only from clean PDFs validates the easiest part of the workload and hides the documents most likely to leak data.

## Reconciliation is the final security control

AWS's serverless workflow finishes with a job report and a reconciliation check. A fully reconciled run means every submitted document produced an output; document-level failures are recorded while successful peers continue, and sustained page failures can be redriven without reprocessing completed work.

Production teams should extend that idea to privacy assurance. Record the blueprint version, input count, failed pages, confidence distribution, redaction counts and human-review exceptions for every batch. Then make downstream release conditional on reconciliation and the agreed quality threshold. The model detects PII; the surrounding control plane decides whether the result is safe enough to leave the boundary.

## Sources

- [Build a serverless PII redaction pipeline with Amazon Bedrock Data Automation](https://aws.amazon.com/blogs/machine-learning/build-a-serverless-pii-redaction-pipeline-with-amazon-bedrock-data-automation/) - AWS Machine Learning