The audit that finds the bad news.

A QA audit of a synthetic labeled dataset — filter it by batch, defect type, and reviewer. Every number carries its n and its method.

INTERACTIVE SYNTHETIC DEMONSTRATION · 60 AUDITED UNITS · FILTERABLE BY BATCH, DEFECT TYPE, REVIEWER

This demonstration uses entirely synthetic data and simulated annotators. It shows how our evaluation process works — it does not show results on real data, real annotators, or real client work. No human annotators participated. Agreement figures shown are internal process measurements, not product accuracy, not guarantees, and not transferable to any real engagement. Primwright Enterprise does not currently offer these services commercially.

PRIMWRIGHT ENTERPRISE

DEMONSTRATION 7 OF 7

A QA audit of a synthetic labeled dataset — filter it by batch, defect type, and reviewer. Every number carries its n and its method, and the failures stay on the record: one batch failed the overturn gate, and one reviewer was wrong two times out of three.

01 FILTER THE AUDIT

Inspect the findings.

Showing: All batches · n=60 audited of 480 synthetic units. Synthetic data — simulated annotators.

DATASET: Northwind Desk labeled bank (synthetic)

AUDITED SAMPLE: 60 of 480 units

DEFECTS FOUND: 24 [ACTUAL — internal, simulated; synthetic data]

SAMPLING: Stratified · blind re-annotation

Filters (buttons; no data input required):

Defects by type

Definitions

TypeDefinitionCount
Wrong classLabel contradicts the taxonomy's precedence rules.6
Boundary misreadCorrect neighborhood, wrong side of a documented edge case.5
Guideline driftLabel follows an older guideline version than the batch's governing version.3
Missed defect (eval)For evaluation units: a planted factual defect the judge did not cite.4
Overturned labelAdjudication reversed the production label.3
Provenance gapUnit missing its guideline-version or judge record.1
Language-slice errorLabel correct for one language slice but wrong for the other.2

Error-rate estimate by batch

Defect rate per batch with Wilson 95% interval. The method is stated so the estimate is checkable — the audit is itself auditable.

BatchAuditedDefectsRateWilson 95%
Batch A20525%11.2% – 46.9%
Batch B20735%18.1% – 56.7%
Batch C201260%38.7% – 78.1%

The bank was seeded with defects to demonstrate the audit — this error rate describes the synthetic bank, not any real dataset. Wide intervals are honest: n=20 per batch cannot support a precise estimate, and the dashboard says so.

Overturn gate — the bad news, on the record

Overturn gate: provisional bar 10% [PROPOSED — unvalidated standard].

Overturn rate measures reviewer–majority disagreement, not reviewer correctness.

BatchOverturn rateVerdict
Batch A5.0%Pass
Batch B10.0%Pass
Batch C15.0%FAIL

Failed gate action: BATCH C FAILED THE GATE — Batch C exceeded the provisional 10% overturn bar. Per procedure the batch was re-sampled at double depth before delivery — the re-sample is shown, and the original failure stays on the record. Failing gates are reported, not hidden.

Reviewers — including the fallible one

Example findings (6 shown)

F-01 — MINOR — Batch A · Boundary misread

"Billing-triggered lockout labeled billing_inquiry; precedence rules place it in account_access."

Disposition: Overturned to account_access; example added to the edge-case appendix.

F-02 — MINOR — Batch B · Guideline drift

"Two units labeled under guideline G1.0 after G1.1 took effect for the batch."

Disposition: Re-labeled under G1.1; provenance records corrected.

F-03 — MAJOR — Batch B · Missed defect (eval)

"Judge did not cite the self-contradictory retention claim in a response evaluation unit."

Disposition: Unit re-scored; judge re-briefed on contradiction defects.

F-04 — MAJOR — Batch C · Wrong class

"Security-relevant phrasing labeled technical_issue; precedence rules require security_concern."

Disposition: Overturned; batch C flagged for re-sampling (see overturn gate).

F-05 — MINOR — Batch C · Provenance gap

"One unit missing its judge-identity record — provenance gap."

Disposition: Record reconstructed from the batch log; gap logged as a process defect.

F-06 — MINOR — Batch C · Language-slice error

"Arabic item labeled with the English-slice reading of a boundary example."

Disposition: Overturned; in-language qualification rule restated in the batch briefing.

The sampling method, in full

Stratified random sample: 20 units per batch (60 of 480 synthetic units, 12.5%), stratified by class and language slice. Blind re-annotation by two independent simulated judges; disagreements adjudicated against sealed gold. Error-rate estimate uses the Wilson 95% interval on the sampled defect count. The audit is itself auditable: sample IDs, strata, and the gold seals are listed in the manifest.

Honest notes

Overturn rate measures reviewer-majority disagreement, not reviewer correctness — a distinction made on every report.

The audit measures quality; it does not certify it.

Findings generalize only to the sampled units under the stated method.

Batch C failed the provisional overturn gate and was re-sampled; the failure remains on the record.

◈ METHOD, STATED HONESTLY

What this demonstrates — and what it doesn't.

IT DEMONSTRATES:

IT DOES NOT CLAIM:

This demonstration uses entirely synthetic data and simulated annotators. It shows how our evaluation process works — it does not show results on real data, real annotators, or real client work. No human annotators participated. Agreement figures shown are internal process measurements, not product accuracy, not guarantees, and not transferable to any real engagement. Primwright Enterprise does not currently offer these services commercially.

All demonstrations

Demo 6: Rubric development

→ NEXT STEP

We sell the habit of reporting bad results.

A QA audit measures your data; it does not certify it. Start with a scoping conversation — we will tell you what we can and cannot audit.

Request a Pilot

All Enterprise Services