Research
The open questions I keep poking at.
Active explorations — open questions, experiments, and rough prototypes I'm working through right now. These are directions, not results: there are no benchmarks to report and nothing published to cite. Each entry pairs a question I find non-obvious with the first experiment I'd run to attack it.
- Experiment01
Attribution precision, not answer accuracy
How often does a legal answer cite a real source that doesn't actually support the sentence it's attached to?
- Why
- "Cited," "supported," and "correct" are three independent failure axes — a system can score well on one while quietly failing another, so a single accuracy number hides the failure that matters most in law.
- Approach
- An NLI-based harness that decomposes each answer into atomic claims, aligns every claim to its cited span, and reports an attribution-precision curve rather than one headline score.
RetrievalEvalLegalRelatedLittman - Prototype02
Distribution-free approve / flag / escalate
Can you put a guaranteed ceiling on wrong-approvals — at most 3%, say — while abstaining as rarely as possible?
- Why
- It turns "confidence" from a vibe into a statistical guarantee, and makes the abstain rate an explicit, tunable business cost instead of an accident of the model.
- Approach
- Calibrate abstention thresholds with split-conformal prediction on a labeled set, then stress-test whether the guarantee still holds under deliberate distribution shift.
CalibrationDecision systemsRelatedLittman - Prototype03
On-device distilled models for legal triage
How small can a model get and still do a competent first pass at privilege and PII detection — without any document leaving the building?
- Why
- Confidentiality often forbids cloud calls outright, so the real constraint is not accuracy in the abstract but accuracy under a hard "nothing leaves the device" rule.
- Approach
- Distill a teacher's privilege-flagging behavior into a 1–3B student, quantize it, and map the accuracy/latency/privacy frontier — pairing the hard tail cases with selective escalation.
DistillationPrivacyEdgeRelatedLittman - Experiment04
Prover–verifier decomposition for auditable workflows
When being auditably right matters more than being fast, does a cheap generator paired with an independent adversarial verifier beat a single strong model?
- Why
- The bet is architectural: separating generation from verification should produce loud, catchable failures instead of silent, confident ones.
- Approach
- A plan–execute–verify–replan loop where the verifier is a different model family — decorrelated blind spots — scored on injected-error catch rate against a self-critique baseline.
AgentsVerification - Open question05
The half-life of a detector
Once a detector ships — for cheating, fraud, adversarial inputs — how fast does the adversary route around it, and can that decay be modeled as a curve?
- Why
- Most ML security treats robustness as a static property, but the real object is a two-player, co-evolving game with a measurable "detector half-life."
- Approach
- A bilevel attacker/detector loop on a toy evasion task: ship successive versions and measure time-to-evasion as a function of how opaque the detector is.
SecurityAdversarialSystems - Exploring06
A CVaR go / no-go layer for tail-risk decisions
Should risk-sensitive go/no-go decisions optimize the tail — the worst 5% of outcomes — instead of the average?
- Why
- Expected-value decision-making is exactly the wrong tool when the tail is fat and a single catastrophic outcome is unacceptable; averaging over a rare disaster is how you talk yourself into it.
- Approach
- Build VaR- versus CVaR-optimized go/no-go rules over weather, mechanical, and duty-time features, and show where the two recommendations diverge on heavy-tailed cases.
RiskAviationDecision systemsRelatedFlightReady - Prototype07
Latency-budgeted edge perception with calibrated early-exit
Under a hard deadline — say 30ms on-device — how do you spend a fixed compute budget to maximize accuracy, and know the moment to escalate to a bigger model?
- Why
- It reframes inference as a scheduling problem with an SLO, where the real decision is not which model to run but when to stop and when to call for help.
- Approach
- A two-tier cascade — a tiny on-device model backed by a large cloud model — gated by a conformal confidence threshold, plotted as accuracy versus p99 latency versus escalation rate.
EdgePerceptionCalibrationRelatedEagleGrid - Open question08
Tamper-evident decision provenance
Can an AI decision be made cryptographically auditable — so you can prove, after the fact, exactly which document versions, retrieved spans, prompt, and model version produced a given approve or flag?
- Why
- Audit logs today are trust-me databases; a Merkle or transparency-log structure turns provenance into compact, verifiable tamper-evidence instead.
- Approach
- Hash each decision's full input bundle into an append-only log, then demonstrate a compact inclusion-and-consistency proof a regulator could check without trusting the vendor.
ProvenanceAuditCryptoRelatedLittman - Exploring09
Eval harnesses as a moat
What if the defensible asset isn't the model but a self-reinforcing, expert-labeled eval set — one that gets stronger every time the system fails in production?
- Why
- It inverts the usual framing: the moat becomes measurement infrastructure, not weights — and it compounds over time instead of depreciating.
- Approach
- A closed loop — production failure to triaged golden case to PR-gating regression test — with per-layer metrics and a drift monitor that fires when a calibration guarantee silently degrades.
EvalInfrastructureRelatedLittman - Experiment10
Calibration under distribution shift
A model calibrated on last year's contracts — is it still calibrated on a new practice area, and can you catch the drift before it costs you an approval?
- Why
- This is the failure mode conformal methods quietly assume away: the guarantee holds only while production looks like the calibration set. A new practice area can void it silently.
- Approach
- Track expected calibration error and a drift statistic per segment; fire an alarm when a segment’s reliability degrades past a threshold, before the wrong-approval rate moves. There is a live calibration explorer in the lab.
CalibrationEvalRelatedLittman - Open question11
Retrieval that knows what it is missing
Can a retriever estimate its own recall — flag "the governing clause probably is not in this candidate set" — before the generator confidently answers anyway?
- Why
- Recall caps everything downstream, and "confidently wrong" is born exactly when the right passage was never retrieved. A retriever that abstains beats a generator that guesses.
- Approach
- A coverage estimator over the candidate set (embedding-space density plus a learned gate); route low-coverage queries to human search instead of to the model.
RetrievalCalibrationRelatedLittman - Exploring12
Speculative decoding for unit economics, not latency
Can a small domain-tuned drafter plus a large verifier cut the cost of a legal-extraction pass without changing the output distribution?
- Why
- Everyone frames speculative decoding as a latency trick; for a solo builder the interesting lever is unit economics at scale — the accepted-token rate is a cost curve, not a speed curve.
- Approach
- Measure accepted-token rate on legal extraction as a function of drafter quality; find the point where a domain-tuned drafter pays for itself.
InferenceSystems - Prototype13
A git diff for judgment
When a firm changes one standard, can you show exactly which past decisions would have flipped — the blast radius of the change — before you ship it?
- Why
- Playbooks are versioned rules, so a change has a computable blast radius. Proving it beforehand turns "trust us" into a diff a partner can review.
- Approach
- Re-run the rules engine over a decision corpus under the old and new rule sets; report the exact set of flipped decisions, each with the rule that moved it.
Decision systemsLegalRelatedLittman - Experiment14
The economics of an eval
How many expert-labeled examples do you actually need before a distribution-free guarantee is trustworthy — and what is the marginal value of label number 2,000?
- Why
- Labeling is the real cost of a legal-grade eval set. Treating it as a budget problem — a learning curve on guarantee tightness versus label count — is oddly underexplored.
- Approach
- Plot guarantee tightness against label count on a held-out set; find the knee where the next hundred labels stop buying much, and spend elsewhere.
EvalCalibrationRelatedLittman - Experiment15
Structured extraction as a typed contract
If extraction output is a typed contract, can property-based tests over the schema catch the class of silent errors — valid JSON, wrong value — before the rules ever run on it?
- Why
- Constrained decoding guarantees valid JSON, which is exactly why the dangerous errors are the well-formed ones. Treating extraction like a typed API invites property testing.
- Approach
- Generate adversarial documents that satisfy the schema but violate invariants (a termination date before the effective date); measure how many the validators catch.
RetrievalEvalRelatedLittman - Open question16
The chemistry-to-enjoyment ceiling
How much of an individual’s wine enjoyment is even predictable from chemistry — and where does the signal run out?
- Why
- Instrumental chemistry predicts some composition-linked attributes moderately well and many aromas weakly, and hedonic liking is partly constructed from expectation and context. There is a ceiling, and honestly locating it matters more than chasing accuracy past it.
- Approach
- On a public chemistry + descriptive-panel + liking dataset, fit models for each hop — analytes → sensory axes → individual liking — and report the variance explained at each stage as a ceiling curve, not one headline accuracy number.
ChemometricsSensoryPreferenceRelatedOENRA - Experiment17
A palate from as few ratings as possible
How few wines does a person have to rate before an eight-dimension palate model predicts their enjoyment as well as it ever will?
- Why
- A preference model that needs two hundred ratings is not a product. The cold-start economics — how fast confidence tightens per rating — decide whether the whole idea is usable.
- Approach
- Simulate elicitation that actively samples the wines which most reduce posterior uncertainty over the palate’s per-dimension weights, and measure how match confidence narrows per rating versus random sampling.
PreferenceActive learningBayesianRelatedOENRA - Prototype18
Confidence intervals that actually cover, per dimension
Can each sensory-dimension estimate carry a distribution-free interval that genuinely covers — including on wines identified only from a label?
- Why
- The promise is that confidence widens when data is sparse. That promise is only honest if the intervals have guaranteed coverage, not just a wider-looking band drawn for effect.
- Approach
- Conformalize the per-dimension regressors (split-conformal / conformalized quantile regression), then verify marginal and completeness-stratified coverage on held-out and deliberately shifted wines.
CalibrationConformalSensoryRelatedOENRA - Open question19
Inverting the sensory model for blend design
Can the forward chemistry-to-sensory model be run backward — from a target sensory profile to the component fractions that would hit it?
- Why
- For a winemaker the useful direction is inverse: not “what does this blend taste like” but “what blend lands on this target.” The forward model makes that inverse problem tractable.
- Approach
- Treat blend fractions as the decision variable, the forward model as a constraint, and search for fractions minimizing distance to a target eight-dimension profile under composition bounds; validate against known blends.
OptimizationChemometricsBlendingRelatedOENRA
A note on honesty
Everything above is an exploration, not a claim. These are open questions, experiments mid-flight, and prototypes with rough edges — no benchmarks to report, nothing peer-reviewed to cite. When one of them turns into something real, it graduates into a project or a written note, with its limitations stated plainly.