LLM Evaluation Kit for Legal Teams: 40 Annotated Publications, Metrics, and Scoring

Summary quality should not be reduced to a single language score. Legal reviewers need factual completeness, factual consistency, actionability, and review effort. A useful rubric can score each dimension from 0 to 2: incorrect or absent, partially correct, and correct and sufficient.

For example, a publication summary may receive separate scores for the procedural event, parties or case context, consequence, required action, and urgency. Add a critical-error flag for any omission or invention that could change a legal decision, even if the overall prose is readable.

Deadline extraction requires classification metrics. Precision answers: when the model reports a deadline, how often is it valid? Recall answers: of all publications with a deadline, how many did it find? The F1 score combines both, but legal teams should not let a balanced average hide a dangerous recall gap.

Use the formulas directly in the spreadsheet. Precision equals true positives divided by true positives plus false positives. Recall equals true positives divided by true positives plus false negatives. F1 is the harmonic mean of precision and recall, calculated as 2 multiplied by precision and recall divided by their sum.

Treat date accuracy as a separate measure. A model that identifies a deadline but shifts the date by one business day is not equivalent to a model that correctly identifies the date, and both should be distinguished from a model that correctly says the publication contains no deadline.

Useful operational metrics include average review time, percentage accepted without editing, percentage escalated to a lawyer, latency, structured-output failure rate, and cost per accepted result. Report results by category, such as explicit deadline, implicit deadline, no deadline, and ambiguous publication.

For legal and privacy governance, document how the test data was anonymized, who accessed it, and where outputs were stored. The NIST AI Risk Management Framework is a useful general reference for organizing risk identification, measurement, and monitoring, while Brazil’s LGPD text on the official government legislation portal should inform personal-data handling decisions.

The Model Context Protocol, or MCP, can connect an AI assistant to controlled tools and data sources. In a legal publication workflow, that may mean allowing ChatGPT, Claude, or Gemini to retrieve a publication, inspect structured process information, or return a proposed action without forcing staff to copy and paste between systems.

For a fair comparison, expose the same tool capabilities to each model. Record whether the model used a tool, which publication identifier it requested, the number of calls, the input and output usage, latency, and any failed or repeated request.

The test should distinguish model quality from connector quality. If one model receives incomplete context because a connector returned an error, classify the event as an integration failure and retain it in a separate operational metric. Do not silently score missing context as a reasoning failure.

Calculate cost per accurate result rather than cost per request. One practical formula is total model and infrastructure cost plus human review cost, divided by the number of outputs accepted for their intended use. Include correction time because a low-priced model that requires extensive editing may be more expensive in practice.

An illustrative worksheet might show a model producing 40 outputs, with 30 accepted, 8 corrected, and 2 escalated. If usage costs total $4 and review costs total $36, the operational cost is $40, or $1.33 per accepted output. The figures are illustrative; replace them with provider invoices, internal hourly rates, and measured review time.

Pocket Jurídico can serve as the publication and process layer in this test, including MCP connectors for AI tools and API connections to internal legal software, ERP, or CRM systems. This allows the team to compare not only model responses, but also whether the answer fits the existing feed, calendar, assignment, and audit workflow.

Start with one workflow and one owner. A litigation team might begin with daily publication summaries and deadline suggestions for a defined group of cases, while a corporate legal department might select one outsourced law firm or one business unit for the pilot.

Create a written output contract before connecting the model. Require fields such as process number, publication date, movement, deadline status, deadline date, counting basis, suggested action, responsible person, and review status. If a field cannot be supported by the publication, the model must return “not identified” rather than guessing.

Next, connect the evaluation to the real feed. A monitored publication should retain its source text, process context, documents, comments, and review history. This is more useful than evaluating isolated text because it reveals whether the proposed AI output reduces work or simply moves it to another screen.

Measure the baseline before enabling AI. Record how many minutes staff spend checking diaries and courts, classifying publications, identifying deadlines, and creating tasks. Then compare that baseline with AI-assisted review, separating time saved from time spent correcting model output.

Run the 40-publication kit, review category-level errors, and select a narrow production threshold. Do not choose a model solely because it produced the most fluent summaries. The winning configuration is the one that meets the risk threshold at an acceptable cost and integrates cleanly with the team’s daily process.

After launch, sample outputs weekly at first. Re-test after changing the prompt, model, connector, court coverage, or document format. Keep a versioned record of the prompt, model configuration, scoring results, and human decisions so that performance changes can be explained.

Teams already using a legal management platform do not necessarily need to replace it. A monitoring and AI layer can feed existing systems through APIs, while comments, alerts, calendar events, and publication history remain connected to the process. That reduces duplicate entry and makes adoption easier for offices managing high volumes or multiple outsourced firms.