# Evaluation: what the MVP shows, and what it doesn't yet

Dataset 2026.09.29 (CourtListener cross-check added in 2026.09.29.1, citation-audit fixes in 2026.09.30), NYC sidewalk-defect liability: 21 propositions, 5 authorities, 13 cases, 19 sources, 53 relationships. Written at the end of the first build, before any expansion.

> **Update, dataset 2026.09.30.1.** A second subject, traffic accidents, has since been added (see [below](#update-traffic-accidents)). The sections that follow still describe the first build. On 2026-10-25 the benchmark dropped five questions: two "should I sue" questions (legal advice), a dog-bite question, and two fictional fixture questions. It now has 28 questions. The metric tables below are the results from the earlier runs.

The project set out to prove seven things first. This note says, for each, what the MVP demonstrates, how, and what has not been tested. The short version: the machine-facing claims (retrieval, provenance, handoff) are demonstrated in controlled conditions. The human-facing and outcome claims (trust, better research, developer adoption) are not yet tested at all, and they are the ones that decide whether this is worth expanding.

| # | Claim | Status | Main evidence |
| --- | --- | --- | --- |
| 1 | Humans trust the evidence presentation | Not tested | Pages exist; no one outside the project has used them |
| 2 | Agents can discover the service | Partly shown | A scripted agent finds MCP from `/llms.txt`; no LLM agent or public deployment yet |
| 3 | Agents can retrieve structured evidence | Shown, on our own questions | 17-question benchmark, 100% recall, in CI |
| 4 | Evidence retains provenance | Shown for integrity; partly for source accuracy | 100% provenance completeness; 28 of 28 automatically checkable quotations found verbatim |
| 5 | Evidence can be passed between agents | Shown | Handoff demo: 13 of 13 objects verified by an agent sharing no code; tampering caught |
| 6 | Agents produce better-grounded research with it | Not tested | The scoring harness exists; the with/without experiment has not been run |
| 7 | Developers are willing to integrate the API | Not tested | OpenAPI, MCP configs, quickstart; no external developer has tried it |

## Benchmark

`npm run eval` builds a packet for each question in `eval/questions/` and checks it against written expectations. It then re-checks every object through the public HTTP interface: it fetches each canonical URL, recomputes the content hash, re-derives applicability from the stored dates, and resolves each legal citation. CI fails below `eval/thresholds.yaml`.

The 17 questions covered the rule before and after § 7-210 (including the effective date and the day before), tree wells, the owner-occupied exemption, City-created defects, Big Apple maps, notice-of-claim deadlines, trivial defects, constructive notice, the nondelegable duty, an out-of-scope car accident, "should I sue", an uncovered jurisdiction, and two fixture questions for conflicts and temporal uncertainty. The "should I sue" question and the two fixture questions are no longer in the benchmark. Conflict and temporal-uncertainty behavior is still covered by the unit tests.

| Metric | Result | Threshold |
| --- | --- | --- |
| Authority recall (authorities and cases a question must cite) | 100% (14/14) | ≥ 95% |
| Proposition recall | 100% (28/28) | ≥ 95% |
| Citation accuracy (every `cite_as` block matches its URL and resolves) | 100% (364/364) | 100% |
| Temporal accuracy (every returned rule applies on `as_of`) | 100% (75/75) | 100% |
| Temporal questions (date-bounded questions fully right) | 100% (10/10) | 100% |
| Provenance completeness | 100% (292/292) | 100% |
| Unsupported-claim rate (propositions with no grounding authority or case in the packet) | 0% (0/59) | 0% |
| Forbidden hits (must-not objects, advice phrases, known fabricated citations) | 0 | 0 |
| Conflict and limit detection | 100% (4/4) | 100% |
| Status accuracy | 100% (16/16) | 100% |
| Determinism (identical packet hash on repeat) | 100% | 100% |
| Latency, p95 | about 3 ms in-process, 7 ms over local HTTP | ≤ 250 ms |

What these numbers do not mean:

- **The questions were written by the same person who wrote the data and the topic vocabulary.** Question interpretation is deterministic topic matching, and during the build three phrasings initially missed rules until synonyms were added. A held-out set written by other people would be a fairer test and would likely score lower.
- **The benchmark no longer includes the fictional conflict question.** The seed data has no known conflicting authorities, so benchmark conflict detection measures `limits` relationships. Fixture conflicts and temporal notes are covered by the unit tests.
- **The dataset is tiny.** Latency and selection behavior at 21 propositions say little about 2,000.

### Update: traffic accidents

Dataset 2026.09.30.1 adds traffic accidents: 23 propositions, 14 authorities (including Part EE of chapter 58 of the Laws of 2026 and the federal Graves Amendment), and 6 Court of Appeals and Appellate Division cases. The whole dataset now has 44 propositions, 19 authorities, 19 cases, 41 sources, and 92 relationships.

The accident questions were written before the records (q18 to q33). They cover the rear-end rule, the serious injury definition on each side of the 2026 amendments, comparative fault in 2025 and 2026, rental companies before and after the Graves Amendment, owner permission, crosswalks, a crash with a City vehicle, no-fault notice, the time to sue, and rideshare. The "should I sue" question and the out-of-scope dog-bite question were removed on 2026-10-25. q11, the car accident question, used to check that the dataset said it had no coverage; it now checks that the answer contains only traffic rules and no sidewalk rules.

| Metric | Result (33 questions) |
| --- | --- |
| Authority recall | 100% (37/37) |
| Proposition recall | 100% (54/54) |
| Citation accuracy | 100% (649/649) |
| Temporal accuracy | 100% (180/180) |
| Temporal questions | 100% (17/17) |
| Provenance completeness | 100% (508/508) |
| Unsupported-claim rate | 0% (0/118) |
| Forbidden hits | 0 |
| Conflict and limit detection | 100% (6/6) |
| Status accuracy | 100% (31/31) |
| Determinism | 100% (33/33) |
| Latency, p95 | about 3 ms in-process |

Two engine changes came with the second subject. Topics now carry a domain, and a question that names a subject's topics gets only that subject's rules, plus rules on shared topics such as notice of claim, time limits, and comparative fault. Without this, "Who is liable for a car accident?" also matched the sidewalk liability rules. Topic weights are likewise counted only over the propositions in the question's domains, so adding a subject leaves the other subjects' answers unchanged; the sidewalk benchmark and page questions return the same packets as before the expansion. And because the 2026 amendments depend on when a lawsuit is commenced rather than when the accident happened, those records tell readers to use the commencement date as `as_of`.

The caveats above apply with more force: the same person wrote the accident questions, records, and topic terms, and some phrasings needed new synonyms before they matched. The New York Senate's statute pages refuse automated requests, so the statute passages were compared by hand with a second published copy and, where amended, with the bill text; the bill, the Department of Financial Services pages, the U.S. Code, and all six cases' CourtListener opinion text are checked automatically by `npm run verify`.

## 1. Humans trust the evidence presentation: not tested

What exists: every object has an HTML page with the verbatim passage, a link to the source, the retrieval date, a status badge, separate trust fields (no combined score), and a plain statement that no record has yet been reviewed by a lawyer. Packets open with their status and limitations.

What's missing is any person outside the project using it. "Trust" should also mean *calibrated* trust: people should rely on verified records and hesitate on the three unverified candidates, not trust everything equally.

Next test: moderated sessions with 5 to 8 target users (NY lawyers, paralegals, legal journalists). Give them tasks ("Was the City liable for this 2002 fall?"), then check whether they can find the source text, tell a verified record from an unverified one, and notice the date limits, and ask how much they would rely on it.

## 2. Agents can discover the service: partly shown

Shown: the handoff demo's Agent A starts from the origin alone, reads `/llms.txt`, finds the MCP endpoint, and connects with the official MCP client. The service also publishes `/.well-known/service.json`, OpenAPI 3.1, JSON-LD, `Link` headers, a sitemap, and Markdown versions of every page.

Not shown: Agent A is a script that parses one line of `llms.txt`, not a language model deciding what to do. The service isn't deployed, so search engines, MCP registries, and crawlers can't find it.

The evaluation caught one real discovery failure. The original default port, 4190, is on the Fetch standard's blocked-port list, so Node's `fetch`, browsers, and the MCP HTTP client all refused to connect, while curl and the in-process tests worked fine. The default is now 4191, and the server warns about blocked ports. Failures like this only show up with real clients, which is the argument for testing with real agents next.

Next test: give two or three LLM agents (for example Claude and GPT with web fetch) only the public URL and a sidewalk question, and record whether they find and use MCP or REST without further hints.

## 3. Agents can retrieve structured evidence: shown, on our own questions

Five MCP tools with input and output schemas (`build_evidence_packet`, `search_evidence`, `get_evidence`, `get_relationships`, `resolve_citation`), the same data over REST with OpenAPI, and packets that are deterministic and schema-validated. The benchmark above passes at 100%, over MCP and REST alike, in-process and against a running server.

The limit is interpretation. The service matches topics, not meaning. That is deliberate (no model in the request path, so results are reproducible), but it means unusual phrasings can miss. Packets say so through `interpretation.matched_topics` and `NO_COVERAGE` rather than guessing.

## 4. Evidence retains provenance: shown for integrity, partly for source accuracy

- Every proposition, authority, case, and relationship in every benchmark packet has at least one verbatim passage, tied to a source record with a URL and a retrieval date (292 of 292 checked). Getting there took data fixes: five relationships had no supporting passage, and the benchmark flagged them.
- Every `cite_as` block matches the content at its URL (364 of 364), and each passage carries its own SHA-256.
- `data/published.lock.json` holds the hash of every published `id@vN`. `npm run verify -- --offline` runs in CI and fails if any of them changes without a version bump. `npm run bump` archives the old version verbatim first. `npm run build` produces a byte-identical static export for the same data.
- `npm run verify` found every quotation from the sources that allow automated fetching verbatim (28 of 28, across Cornell LII and one other host), after teaching it to skip reporter page markers.

What is not automatically checked: the official publishers (American Legal for the Admin Code, nysenate.gov, and nycourts.gov) refuse automated requests, so 12 sources were checked by hand and through search snippets, as each record's `verification.methods` says. The two 2026 decisions (*Landberg*, *Mantinaos*) rely on secondary copies and stay `UNVERIFIED_CANDIDATE`. On 2026-10-25 the evidence records were reviewed for source and accuracy and `human_review_status` was set to `reviewed`. `VERIFIED` still means checked against a source (see [trust](https://agents.newyorkeverydaylawreference.com/docs/trust)).

CourtListener cross-check: 12 of 13 case citations resolve to a CourtListener opinion whose case name and decision date match the record, including both 2026 decisions, and every quoted passage from those 12 cases appears in CourtListener's opinion text. *Xiang Fu He v Troon Mgt.* (34 NY3d 167) is not on CourtListener, but 66 opinions there cite it, so it rests on its official source. `npm run verify` fails a verified record if its citation resolves to a different case, if none of its quotations appear in the linked opinion, or if CourtListener neither has it nor has any opinion citing it. Those are the shapes fabricated or mis-attached citations take.

## 5. Evidence can be passed between agents: shown

`npm run demo`, also run in CI: Agent A gets a packet over MCP and hands over only `cite_as` blocks and statements. Agent B imports nothing from this project. It re-implements canonical JSON and SHA-256 from the docs, verifies all 13 objects and the packet hash, and asks `/resolve` whether each is current. When one statement is altered and re-hashed, `/resolve` answers `hash_mismatch`.

The demo also surfaced a semantic point. `/resolve` answers as of today, so a correct claim about 2002 comes back `superseded`. Agent B now accepts that when the replacement took effect after the claim's date, and the citation docs explain how.

Limits: both agents are scripts in one language on one machine. A hash proves the object is what the origin published, not that the origin is honest. Signed releases are not implemented. The integrity anchors today are git tags (`dataset-2026.09.29`) and the lock file.

## 6. Agents produce better-grounded research using it: not tested

This is the central claim, and it has not been measured. The benchmark scores packets, not agents.

Next test, and the cheapest of the three: run LLM agents on the benchmark questions twice, with and without the MCP tools, and score the answers with the same checks. Did they cite the right authorities? Did any citation fail to resolve (for example the fabricated "13 NY3d 791")? Did they get 2002 against 2024 right? Did they give advice? The scoring code in `eval/run.ts` can be reused. What's missing is the agent runner and answer-to-citation extraction.

## 7. Developers are willing to integrate the API: not tested

What exists: OpenAPI 3.1, JSON Schemas, copy-paste MCP configs for Cursor and Claude, a quickstart, curl and JavaScript examples, problem+json errors with request IDs, and usage events recorded per request. The agent server requires a paid access token before it returns a packet. The benchmark calls the app with that gate open.

No developer outside the project has tried it, and the repository is private. The license question is settled (MIT for code, CC BY 4.0 for the editorial data), so the remaining blocker is a public deployment.

Next test: 3 to 5 developers integrate it into something they already build. Measure time to first packet and where they got stuck, and ask whether they would keep using it.

## Recommendation

Don't add jurisdictions or topics yet. In order:

1. Run the with/without agent experiment (claim 6). It is cheap, and it tests the core hypothesis.
2. The evidence records were reviewed for source and accuracy on 2026-10-25 (`human_review_status: reviewed`). A later lawyer review of the complaint guide is still open.
3. Deploy publicly, and run the discovery (2) and developer (7) tests.
4. Run the human-trust sessions (1) on the deployed pages.

Expand only if the agent experiment shows fewer fabricated or out-of-date citations with the service than without it.