Littman

The problem: the baseline failure rate is catastrophic

Start with the number that makes this hard. Stanford's RegLab found that general models hallucinate on 58–88% of specific legal queries. Even the purpose-built legal tools from LexisNexis and Westlaw — teams with far more money and data than me — hallucinate 17–33% of the time. So "AI for legal work" isn't a interface problem you design your way out of. It's a reliability problem, and the reliability bar is set by the fact that a firm can't act on an answer it can't check — and in law, a confidently wrong answer surfaces weeks later, in a dispute, after it's already compounded.

The thesis

Turn a firm's own standards into structured, cited, auditable decisions — and know when you can't.

Littman ingests a firm's documents, playbooks, and precedents and exposes them as an assistant, a knowledge layer, a document vault, and workflows. But the output I care about isn't an answer. It's a decision — approve, flag, escalate — that carries the rule, the fact, and the citation that produced it, or that abstains and routes to a human when the evidence is thin.

The model is never the hard part

The thing I keep having to explain: the model call is the easy 10%. The value is the plumbing around it that makes an answer provable. Littman is deliberately a stack of independent checks, because any single check shares failure modes with the thing it's checking. Here's the stack, with the hard part named at each layer.

Ingestion — it's layout, not characters

Modern OCR nails glyphs. It fails on structure: multi-column pleadings, footnote-heavy briefs, nested exhibit tables, redaction bars. And the dominant failure isn't omission — it's confabulation. A parser will invent a plausible cell value in a damages table rather than admit it couldn't read it, and that fabricated number is invisible downstream because it's syntactically perfect. So the first move is to treat parsing as untrusted input and carry a per-block confidence and a pixel-level provenance (page, bounding box) from stage one, so a citation can later point at the exact place on the page.

Chunking — legal text has long-range dependencies

A statute's §(a)(2)(iv) defines a term used in §(a)(2)(v). A fixed 512-token window splits the definition from its use, and you retrieve a chunk that looks relevant and is meaningless without its structural context. The answer is structure-aware chunking plus parent-document retrieval — index small precise pieces, return the enclosing clause — and Anthropic's contextual-retrieval trick of prepending a short situating blurb to each chunk before you embed and index it. That one move cut retrieval failure ~35% in their numbers, ~49% combined with contextual keyword search.

Retrieval — legal queries are lexical and semantic at once

"Bluebook cite for §4.2" demands exact match; "does this indemnity survive termination?" demands semantics. Either alone drops about half of real queries. So: BM25 and dense retrieval in parallel, fused with Reciprocal Rank Fusion, then a cross-encoder reranker for precision — the mechanic you can watch in the retrieval visualizer. The honest failure mode lives here: recall caps everything. If the governing clause never makes the candidate set, no downstream verification can save the answer — and that's exactly where "confidently wrong" is born.

Extraction — valid JSON is not correct JSON

Constrained decoding guarantees schema-valid output 100% of the time, and that output can be perfectly valid and completely wrong. Since the extracted facts (parties, dates, governing law, dollar thresholds) are what the decision engine runs on, a wrong field is a wrong decision with a clean audit trail pointing at it. So every extracted value has to carry an evidence span, uncertain fields are optional so the model abstains instead of confabulating, and business rules catch the impossible (a termination date before the effective date auto-flags).

The decision engine — the neuro-symbolic core

This is the layer that makes Littman defensible. The LLM is an observation engine that reads unstructured documents and proposes structured facts. A symbolic engine — versioned decision tables encoding the firm's playbook — applies those facts deterministically. Because rule execution is deterministic given the facts, the remaining variance is confined to the extraction stage. You've collapsed an open-ended reliability problem into a bounded one you can test exhaustively. Every approve/flag/escalate is a traversal of named rules, which is also a human-readable justification. The decision-trace sandbox is a synthetic model of exactly this.

Faithfulness & abstention — three properties, not one

"Cited," "supported," and "correct" are three independent things, and legal tools fail at each separately: a citation can exist, point at a real source, and still not support the sentence attached to it. So a dedicated verification pass, separate from generation, decomposes the answer into atomic claims and runs entailment of each against its cited span; an unsupported claim gets stripped or flagged. The metric worth reporting is attribution precision, not answer accuracy. And because models are badly calibrated, the three-way approve/flag/escalate is treated as a selective-prediction problem with conformal risk control — a distribution-free ceiling on wrong-approvals — rather than trusting a softmax that says "90%."

Permissions & the eval harness — the boring parts that are the moat

In a firm, the vector index itself is a data-leak surface: a semantically-relevant chunk from a walled matter must never surface, so access is enforced pre-retrieval, tenants are isolated, and every retrieval and decision writes an immutable trace. And the real defensibility isn't the model — anyone can wire up the models. It's the evaluation harness: attorney-adjudicated cases with cited ground truth, versioned like code, where every production failure becomes a permanent regression test. That harness is what lets Littman eventually make the one claim the incumbents can't back up — a measured, auditable error rate, per layer, not a vibe.

Status & what's next

Building. The near-term work is where the difficulty actually concentrates: tightening extraction so the rules operate on clean facts, calibrating confidence so abstention has a real guarantee behind it, and turning the audit trace into something a reviewer wants to read instead of a compliance afterthought. Littman lives at littman.ai.