The audit that finds the bad news.
A QA audit of a synthetic labeled dataset — filter it by batch, defect type, and reviewer. Every number carries its n and its method.
INTERACTIVE SYNTHETIC DEMONSTRATION · 60 AUDITED UNITS · FILTERABLE BY BATCH, DEFECT TYPE, REVIEWER
This demonstration uses entirely synthetic data and simulated annotators. It shows how our evaluation process works — it does not show results on real data, real annotators, or real client work. No human annotators participated. Agreement figures shown are internal process measurements, not product accuracy, not guarantees, and not transferable to any real engagement. Primwright Enterprise does not currently offer these services commercially.
PRIMWRIGHT ENTERPRISE
DEMONSTRATION 7 OF 7
A QA audit of a synthetic labeled dataset — filter it by batch, defect type, and reviewer. Every number carries its n and its method, and the failures stay on the record: one batch failed the overturn gate, and one reviewer was wrong two times out of three.
01 FILTER THE AUDIT
Inspect the findings.
Showing: All batches · n=60 audited of 480 synthetic units. Synthetic data — simulated annotators.
DATASET: Northwind Desk labeled bank (synthetic)
AUDITED SAMPLE: 60 of 480 units
DEFECTS FOUND: 24 [ACTUAL — internal, simulated; synthetic data]
SAMPLING: Stratified · blind re-annotation
Filters (buttons; no data input required):
- Batch: All batches · Batch A · Batch B · Batch C
- Defect type: All types · Wrong class · Boundary misread · Guideline drift · Missed defect (eval) · Overturned label · Provenance gap · Language-slice error
- Reviewer: All reviewers · Q1 — QA reviewer · Q2 — QA reviewer · Q3 — QA reviewer
Defects by type
- Wrong class — 6
- Boundary misread — 5
- Guideline drift — 3
- Missed defect (eval) — 4
- Overturned label — 3
- Provenance gap — 1
- Language-slice error — 2
Definitions
| Type | Definition | Count |
|---|---|---|
| Wrong class | Label contradicts the taxonomy's precedence rules. | 6 |
| Boundary misread | Correct neighborhood, wrong side of a documented edge case. | 5 |
| Guideline drift | Label follows an older guideline version than the batch's governing version. | 3 |
| Missed defect (eval) | For evaluation units: a planted factual defect the judge did not cite. | 4 |
| Overturned label | Adjudication reversed the production label. | 3 |
| Provenance gap | Unit missing its guideline-version or judge record. | 1 |
| Language-slice error | Label correct for one language slice but wrong for the other. | 2 |
Error-rate estimate by batch
Defect rate per batch with Wilson 95% interval. The method is stated so the estimate is checkable — the audit is itself auditable.
| Batch | Audited | Defects | Rate | Wilson 95% |
|---|---|---|---|---|
| Batch A | 20 | 5 | 25% | 11.2% – 46.9% |
| Batch B | 20 | 7 | 35% | 18.1% – 56.7% |
| Batch C | 20 | 12 | 60% | 38.7% – 78.1% |
The bank was seeded with defects to demonstrate the audit — this error rate describes the synthetic bank, not any real dataset. Wide intervals are honest: n=20 per batch cannot support a precise estimate, and the dashboard says so.
Overturn gate — the bad news, on the record
Overturn gate: provisional bar 10% [PROPOSED — unvalidated standard].
Overturn rate measures reviewer–majority disagreement, not reviewer correctness.
| Batch | Overturn rate | Verdict |
|---|---|---|
| Batch A | 5.0% | Pass |
| Batch B | 10.0% | Pass |
| Batch C | 15.0% | FAIL |
Failed gate action: BATCH C FAILED THE GATE — Batch C exceeded the provisional 10% overturn bar. Per procedure the batch was re-sampled at double depth before delivery — the re-sample is shown, and the original failure stays on the record. Failing gates are reported, not hidden.
Reviewers — including the fallible one
- Q1 · QA reviewer [simulated] — 2 overturns raised · 2 upheld — "Raised 2 overturns; adjudication upheld both."
- Q2 · QA reviewer [simulated] — 3 overturns raised · 1 upheld — "Disagreed with the majority 3 times; right only once when checked against gold. Reviewers are fallible — the audit says so."
- Q3 · QA reviewer [simulated] — 1 overturn raised · 1 upheld — "Raised 1 overturn; upheld."
Example findings (6 shown)
F-01 — MINOR — Batch A · Boundary misread
"Billing-triggered lockout labeled billing_inquiry; precedence rules place it in account_access."
Disposition: Overturned to account_access; example added to the edge-case appendix.
F-02 — MINOR — Batch B · Guideline drift
"Two units labeled under guideline G1.0 after G1.1 took effect for the batch."
Disposition: Re-labeled under G1.1; provenance records corrected.
F-03 — MAJOR — Batch B · Missed defect (eval)
"Judge did not cite the self-contradictory retention claim in a response evaluation unit."
Disposition: Unit re-scored; judge re-briefed on contradiction defects.
F-04 — MAJOR — Batch C · Wrong class
"Security-relevant phrasing labeled technical_issue; precedence rules require security_concern."
Disposition: Overturned; batch C flagged for re-sampling (see overturn gate).
F-05 — MINOR — Batch C · Provenance gap
"One unit missing its judge-identity record — provenance gap."
Disposition: Record reconstructed from the batch log; gap logged as a process defect.
F-06 — MINOR — Batch C · Language-slice error
"Arabic item labeled with the English-slice reading of a boundary example."
Disposition: Overturned; in-language qualification rule restated in the batch briefing.
The sampling method, in full
Stratified random sample: 20 units per batch (60 of 480 synthetic units, 12.5%), stratified by class and language slice. Blind re-annotation by two independent simulated judges; disagreements adjudicated against sealed gold. Error-rate estimate uses the Wilson 95% interval on the sampled defect count. The audit is itself auditable: sample IDs, strata, and the gold seals are listed in the manifest.
Honest notes
Overturn rate measures reviewer-majority disagreement, not reviewer correctness — a distinction made on every report.
The audit measures quality; it does not certify it.
Findings generalize only to the sampled units under the stated method.
Batch C failed the provisional overturn gate and was re-sampled; the failure remains on the record.
◈ METHOD, STATED HONESTLY
What this demonstrates — and what it doesn't.
IT DEMONSTRATES:
- An audit that is itself auditable: stratified sampling, blind re-annotation, adjudication against sealed gold, and an error-rate estimate with the method attached.
- The habit of reporting bad results: a failed gate stays on the record, and a reviewer is shown fallible against gold.
- That overturn rate measures reviewer–majority disagreement — not reviewer correctness — a distinction made on every report.
IT DOES NOT CLAIM:
- That this error rate describes any real dataset — the synthetic bank was seeded with defects to demonstrate the audit.
- That the audit certifies quality — it measures quality. Findings generalize only to the sampled units under the stated method.
- Any staffing, scale, or turnaround figure.
This demonstration uses entirely synthetic data and simulated annotators. It shows how our evaluation process works — it does not show results on real data, real annotators, or real client work. No human annotators participated. Agreement figures shown are internal process measurements, not product accuracy, not guarantees, and not transferable to any real engagement. Primwright Enterprise does not currently offer these services commercially.
→ NEXT STEP
We sell the habit of reporting bad results.
A QA audit measures your data; it does not certify it. Start with a scoping conversation — we will tell you what we can and cannot audit.