Primwright
Measure what your AI actually does.
Primwright Enterprise helps organizations adopt AI responsibly and evaluate AI systems rigorously: data and evaluation services that measure model behavior with documented methods.
Engagements begin as scoped pilots — sized to work we can staff, measure, and stand behind.
- Engagement model: Scoped pilots — Small, bounded, priced in writing before work begins.
- Scope: Text only · EN / AR — Dataset QA, model-output evaluation, rubric evaluation, guideline development.
- How it starts: A fit conversation — No commitment — out of scope? We say so plainly.
- Boundaries: Stated on this page — What we do not do — §05 below.
01 Problems We See — "The problems are the practice."
Four patterns we see wherever AI meets real operations. Each one is the reason a corresponding discipline exists in our method — the answer is not a slogan, it is a procedure.
Problem 01 — You can't measure what you can't define.
Teams ship AI systems with no agreed definition of a good output. Dashboards report token counts and latency while nobody has written down what a correct answer looks like — so every evaluation is an argument about taste.
Our answer. The rubric or guideline is written and approved before any measurement happens. If good can't be defined, we say so — before you pay for measurement.
Problem 02 — Annotation is treated as a commodity.
Labels bought by the thousand with no written guideline, no evaluator qualification, and no QA — then used to train or evaluate models as if they were ground truth. The labels are the foundation; nobody inspected the foundation.
Our answer. Versioned guidelines, qualified evaluators, blind gold items, adjudicated disagreements, and a QA report with every delivery. Every label traces to the guideline version that governed it.
Problem 03 — Vendor demos don't reproduce.
A polished demo on cherry-picked inputs: no method, no error analysis, no statement of limits. What you saw in the sales call is not what you get in production — and there is no document that would let you check.
Our answer. Methods written down, disagreement reported instead of hidden, limits stated before you ask. Our own demonstrations run on synthetic data and say so, loudly.
Problem 04 — Evaluation is a one-time event.
A model is "evaluated" once at launch. Behavior drifts, data drifts, prompts change — and nobody re-measures. The evaluation ages out while the decisions made on it stay in force.
Our answer. Pilots produce reusable artifacts — the rubric, the guideline, the QA method — that you keep. Measurement becomes a practice, not a launch checkbox.
02 What We Do · Four Services — "Measurement, done honestly."
Two practices, one discipline: defined guidelines, qualified evaluators, measured agreement, and a QA report with every delivery. English and Arabic are working languages across all four services — bilingual capacity, not a separate offering.
Dataset QA — Independent Labeled-Data Audits
See the Dataset QA page for the full service description: the audit, not the labels — stratified sample, blind re-annotation, defect taxonomy, error-rate estimate with the method attached.
Model-Output Evaluation — Independent AI Output QA
See the Model-Output Evaluation page for the full service description: the human-agreement layer — scored against criteria you approve, defects classified with measured frequencies.
Rubric Evaluation
Absolute scoring of model outputs on anchored rubrics — every scale point defined in words with examples, so scores mean the same thing across raters and across time.
What you receive:
- Anchored rubric, versioned, with scale examples
- Scored evaluations with per-dimension rationales
- Inter-rater reliability report and adjudication log
Read the full Rubric Evaluation service page →
Guideline Development
The classification system itself — categories, definitions, hierarchy, worked examples, edge-case rulings, and the annotation guideline your team can execute against. If good can't be defined, we say so before you pay for measurement.
Offerings:
- Guideline development — "A proposed taxonomy for your problem — definitions, hierarchy, examples, and a labeled sample." (Guideline development)
What you receive:
- Proposed taxonomy: definitions, hierarchy, examples
- Versioned annotation guideline with edge-case rulings
- Labeled sample demonstrating the taxonomy in practice
Read the full Guideline Development service page →
03 How a Pilot Works — "Four stages. Written down."
Every engagement follows the same four stages. The proposal is written before work begins; the QA report is written before you accept. What is not written down did not happen.
- 1. Fit conversation — A short conversation to check fit — the right service, a plausible volume, and data we can handle. Out of scope? We say so plainly.
- 2. Scoped pilot proposal — A written proposal — task definition, guideline or rubric, QA plan, and explicit acceptance criteria — agreed before production work begins.
- 3. Execution with QA — Guideline → worker qualification → calibration → production → reviewer sampling → adjudication. Every delivered unit traces to the guideline version that governed it.
- 4. Acceptance and delivery — A QA report with agreement measures and an adjudication log, then your acceptance review. Retention and deletion follow the agreed terms.
04 Deliverables — "What a pilot produces."
A pilot is not a conversation — it is a scoped piece of work with artifacts you can inspect. These are the documents and records every engagement generates.
- Before work begins — Scoped pilot proposal: The written agreement that governs the pilot. Nothing proceeds on a handshake. (Task definition; Guideline or rubric, versioned; QA plan; Acceptance criteria)
- During execution — Traceable production: Work performed under the agreed guideline, with the chain of custody intact. (Worker qualification records; Calibration results; Reviewer sampling log; Adjudication decisions)
- At delivery — QA report: The evidence that the work was done as specified — inspectable, not asserted. (Agreement measures; Adjudication log; Defect/error analysis; Acceptance review)
- After delivery — Defined handoff: The engagement ends cleanly, with data handled as agreed. (Delivered artifacts; Retention terms honored; Deletion on agreed terms; Remediation guidance)
05 Trust & Boundaries · Stated Plainly — "What we do not do."
An honest practice states its limits before you ask. If your need falls outside these boundaries, we will tell you in the fit conversation — before anyone spends anything.
Your data stays yours.
Data-handling terms are agreed in writing before anything changes hands. We take no regulated data — no health, financial, legal-privileged, or otherwise regulated datasets. Retention and deletion follow the agreed terms, and your data is never used to train anyone's models — including ours.
Controls described, not asserted.
We hold no SOC 2, ISO 27001, HIPAA, or other security certifications, and we say so plainly. Our pilot-grade handling controls are described under NDA during scoping — so you can judge them, not take our word for them.
We decline work.
If your need falls outside our boundaries — or we cannot staff and measure it honestly at pilot scale — we say so in the fit conversation, before anyone spends anything. A practice that can't say no can't be trusted to say yes.
Boundary table:
- Scope: Text only — we do not offer image annotation at scale.
- Languages: English and Arabic — we do not offer other languages.
- Data: No regulated data — no health, financial, legal-privileged, or otherwise regulated datasets.
- Evaluation type: No safety/red-team evaluation and no 24/7 operations.
- Specialization: No domain-specialized annotation (medical, legal, financial) without separately qualified reviewers.
- Certifications: We hold no SOC 2, ISO 27001, HIPAA, or other security certifications. Pilot-grade handling controls are described under NDA during scoping.
- Guarantees: No guaranteed accuracy figures, turnaround times, or SLAs — acceptance criteria are agreed per pilot, in writing, before work begins.
06 Synthetic Demonstrations — "Watch the method work."
Three interactive demonstrations of our evaluation methods — vote, score, and filter against synthetic data with simulated annotators. They show the procedure, not results: no real data, no real clients, no claims. Every demo carries the synthetic label.
Dataset QA
You spent serious money on labels and the model still isn't improving. We audit your labeled dataset — stratified sample, blind re-annotation, defect taxonomy, error-rate estimate with the method attached — so you know whether the data is the problem before you spend again.
Model-Output Evaluation
You are about to put a chatbot or AI feature in front of customers. We independently review what it produces — scored against criteria you approve, defects classified with measured frequencies — so ship, delay, and cut decisions rest on evidence instead of hope.
Rubric Evaluation
Absolute scoring of model outputs on anchored rubrics — every scale point defined in words with examples, so scores mean the same thing across raters and across time.
What you receive: anchored rubric, versioned, with scale examples; scored evaluations with per-dimension rationales; inter-rater reliability report and adjudication log.
Guideline Development
The classification system itself — categories, definitions, hierarchy, worked examples, edge-case rulings, and the annotation guideline your team can execute against. If good can't be defined, we say so before you pay for measurement.
Offerings: guideline development — "A proposed taxonomy for your problem — definitions, hierarchy, examples, and a labeled sample." (Guideline development)
Start a Pilot
Start with a fit conversation. Tell us about your project. We review every inquiry against our scope — if it's a fit, we schedule a scoping conversation. If it isn't, we say so plainly. No commitment, no sales process.
Request a pilot