How to Run A/B Tests to Validate AI-Powered Legal Deadline Extraction

The two variants can differ by language model, prompt, retrieval context, output schema, temperature, validation rule, or human-review threshold. For a meaningful result, change one major variable at a time whenever possible. Comparing Model A with Model B while also changing the prompt and the deadline calendar makes the result difficult to interpret.

A practical baseline is a fixed input set and two processing routes. Both routes receive the same publication text, process metadata, court information, and any relevant procedural context. The only difference should be the extraction configuration under evaluation.

Use a test corpus that reflects the team’s real work. Include routine notices, orders with multiple actions, publications involving business-day counting, duplicate publications, incomplete text, appeals, different courts, different parties, and documents where no deadline exists.

Negative cases are essential. If every test item contains a deadline, the experiment measures extraction ability but not the ability to avoid inventing deadlines. A safe system must identify “no actionable deadline found” when the publication does not create an obligation.

The reference answer, sometimes called the gold standard, should be created through independent human review. Ideally, two legally qualified reviewers classify a sample separately, then resolve disagreements using a written rulebook. This reduces the chance that the benchmark simply reflects one person’s interpretation.

A useful record for each publication includes the process identifier, court, publication timestamp, document type, source text version, expected action, expected deadline, counting convention, responsible party, and whether human review is mandatory. Remove unnecessary personal information from exported datasets and follow the organization’s privacy controls. The Brazilian General Data Protection Law should guide decisions about lawful handling, access, retention, and protection of personal data.

Accuracy alone is too broad for deadline extraction. The most important metric is usually deadline recall: the percentage of true actionable deadlines that the system identifies. Missing one deadline can be materially more serious than producing a formatting error in ten summaries.

Precision measures how often extracted deadlines are actually supported by the publication. Low precision creates review overload and may cause teams to distrust the system. A useful operational measure is the false-deadline rate, calculated as unsupported or invented deadlines divided by all publications processed.

Date accuracy should be scored separately from event recognition. A model can correctly identify that a response is required while calculating the wrong date because it used the publication date instead of the legally relevant trigger. Reviewers should classify whether an error changes the action date, changes the responsible party, or merely affects presentation.

Track action accuracy as well. “File a response,” “provide documents,” “pay a fee,” and “acknowledge receipt” are not interchangeable tasks. The extracted action should be specific enough for a lawyer or legal operations professional to understand what must happen next without reopening the entire publication.

Confidence calibration is another useful measure. If a system labels 90 percent confidence, approximately 90 percent of comparable outputs should be correct within the defined category. Poor calibration is dangerous because it can make weak outputs look safe. Use confidence to prioritize review, not to replace legal judgment.

Operational metrics complete the picture: review time per publication, percentage routed to manual review, queue backlog, processing latency, cost per 1,000 publications, duplicate rate, and escalation rate. For example, a model that improves recall by one percentage point but doubles review time may not be the best choice for a department processing 20,000 publications per month.

For statistical reporting, show the numerator, denominator, segment, and confidence interval. A result of 98 percent recall based on 50 deadlines is not equivalent to 98 percent recall based on 2,000 deadlines. The NIST AI Risk Management Framework provides a useful risk-oriented reference for documenting measurement, monitoring, and governance rather than treating model performance as a single score.

The most common mistake is testing only successful examples. Teams often select clear publications with explicit dates because they are easy to label, then discover that performance drops on short orders, scanned documents, multiple obligations, and notices that refer to earlier acts.

Another error is using the wrong unit of analysis. If a single publication creates three separate actions, scoring the entire publication as one correct or incorrect item hides partial failures. Record each actionable obligation separately, while retaining the publication as the source context.

Do not let the model judge itself. A generated explanation can sound convincing even when it cites no supporting text. Human reviewers should verify the extracted field against the source and mark unsupported reasoning as an error or uncertainty, depending on the agreed rubric.

Avoid changing the production workflow during the experiment. If one variant is sent to the calendar and the other remains in a review queue, the comparison includes different levels of human control. Both variants should be assessed under equivalent conditions.

Finally, do not treat a provider comparison as a permanent ranking. Model behavior changes, legal publication formats evolve, and your prompt may favor one kind of document. Revalidate the selected route periodically and whenever the surrounding workflow changes.

A defensible conclusion sounds specific: “Variant B achieved 99.1 percent recall for the tested civil deadlines, 97.8 percent precision, and a 24 percent review rate over 1,200 publications, with no unresolved high-severity errors.” That is more useful than saying that one model was generally smarter.