Legora can't read a single trace of its legal agent's work. It tests in simulated worlds and sorts failures into 10 to 20 types

Most teams improve agents by reading traces. Legora can't read any, and its workaround is useful for anyone handling confidential client data.

What happened

On the Latent Space podcast, Legora CTO Jacob Lauritzen explained how the legal AI company tests its agent when almost everything users upload is confidential and much of it belongs to law firms' clients. Three engineers and three lawyers build test environments that simulate the work customers do. One example is a 10,000-file data room where the agent runs due diligence and must produce a report that catches the right red flags. Results are plotted on a heat map by practice area and task, because a new model often gets better at some legal work and worse at other kinds.

In production, Legora classifies failures into 10 to 20 known failure modes and runs cheap LLM-based checks behind feature flags. The episode notes also mention paraphrasing traces. For hallucinations, everything the agent writes gets a citation that highlights a bounding box in the source PDF. Its Tabular Review feature supports up to a million cells but starts with 10 rows so users can fix prompts before running the rest. Lauritzen also described a citation problem where the model was right 70% of the time, then 99% after one model release.

My take

Small automation builders face this same problem with fewer resources. You build an AI step that reads a client's support tickets or contracts, and you often shouldn't be reading the live data yourself.

What I take from Legora:

• Build a small fake dataset that looks like the client's real work, and test every prompt or model change against it.
• Name the ways your automation fails. Ten labels like "wrong contact matched" beat a pile of vague bug reports.
• Run the first 10 records, check them, then run the batch. That habit alone saves a lot of cleanup.
• Make every AI output point back to its source record so a human can check it fast.

Re-run your test set when a model update ships. A change can fix one task and quietly break another.