Pinata evals

This is the human-readable catalog of how Pinata is evaluated, rendered live at /reqs/evals. The formal, executable definition of done is the mission validation contract (validation-contract.md, assertion IDs VAL-*); this catalog explains the same intent in plain language so anyone can add scenarios. When the two disagree, the contract wins.

How evaluation works

• One gate. npm run validate runs lint and integrity, typecheck, unit and component tests, the deterministic docs check, the production build, and Chromium end-to-end tests — the same command locally and in CI.
• Real integrations. Milestone validation exercises the real Browserless, Turso, private Blob, and Vercel surfaces. Mocks support focused unit tests; they are never the definition of done.
• Live checkpoints. After each milestone's automated validation passes, the owner drives a headed shared-browser session while the agent inspects the same DOM, console, network, and persisted state.
• Escapes become assertions. Any issue the owner reports is written down as a given / when / expected / observed contract assertion before a fix is implemented.

Chickpea scenarios

The primary real-world target is Chickpea (chickpea.co) plus its explicit /pricing, /about, and /privacy URLs, captured on desktop and mobile. Representative scenarios:

1. Full-page captures include lazy-loaded content near the bottom of the page, on both viewports.
2. Deep zoom on a dense pricing-table cell stays readable, and a pin dropped on that cell persists to the exact natural pixel after reload.
3. Choosing the right nearby element from a dense table ranks the cell ahead of its row and table wrappers.
4. The closed mobile menu's hidden descendants never appear as selectable metadata, while a visible header element does.
5. Desktop and mobile annotations of the same page stay fully independent.
6. The explicit URL array preserves its submitted order in the page tree.
7. A failed viewport capture does not discard its successful sibling, and retry replaces only the failed attempt.

Dogfood scenarios

Pinata captures and annotates its own requirements hub. The dogfood URL array, in order:

1. /reqs
2. /reqs/architecture
3. /reqs/milestones
4. /reqs/decisions
5. /reqs/evals

Representative scenarios:

1. Navigate the hub from the landing page without typing a URL; every route refreshes cleanly; an unknown /reqs/* address lands on a bounded 404 with a working link home.
2. Decision cards match docs/decisions/decisions.json exactly, including provenance quotes and supersession links — there is no second decision dataset.
3. Capture the five routes above as a Pinata project and annotate a requirement from inside Pinata itself.
4. Hostile markup or unsafe links in a document render inert; external links identify their destination and withhold the referrer.
5. The hub stays usable at phone width, at 320 CSS pixels with 200% zoom, and with reduced motion preferred, with no horizontal scrolling of the page.

Landing page scenarios

How a first-time visitor meets Pinata (VAL-LANDING-001, VAL-LANDING-002, VAL-LANDING-003, D066/D067):

1. An anonymous visitor opening / sees the pinata mark — the same inline SVG mark as the favicon — directly above the URL capture entry (a required root URL, an optional add-more-URLs control, and one visible primary action), plus a brief value proposition. The page renders with no console errors, no /api/* traffic, and no external image requests.
2. Below the hero, a fully static example shows a marked-up capture: a screenshot region with two numbered pins, a two-entry comment thread, and a visible DOM metadata panel. It renders from bundled fixture data with no database or API dependency and depicts saved comments only — no reply control.
3. An anonymous visitor who submits the capture entry is routed to the sign-in prompt (focus lands on the password field) without any project write; after signing in, the editor lands back on / with the entered URLs retained in the always-active form, and creating the project issues exactly one POST /api/projects answered 201, with the project listed on the same page.

Editor entry scenarios

How Lucas gets from the password prompt to a project. Representative scenarios (VAL-AUTH-008, VAL-AUTH-009):

1. While the project list loads, the region reports itself busy and says so in text; loading, empty, populated, and failure never render at once.
2. With zero projects, the named list says it is empty and offers exactly one create action; submitting once shows the project exactly once, already in the workspace, with no address-bar navigation.
3. When the list read fails, the failure is announced, sign-out stays available, any half-typed create form keeps its safe input, and one press of Try again issues exactly one read; a press while that read is in flight does nothing.
4. Reload and Back/Forward after a create never resubmit the form and never create a second project.
5. Login, list, and create controls all have accessible names, associated errors, visible focus, and no keyboard trap; a wrong password announces a generic error and clears the password field.
6. End-to-end runs delete their own rows in teardown, not in a trailing test, so an aborted run cannot leak run-scoped projects, pages, captures, or idempotency keys into the real database.

Role and thread scenarios

1. A founder link opens a read/reply-only view; editing controls are absent visually and to assistive technology.
2. Founder replies and editor follow-ups interleave chronologically and cannot be edited or deleted by anyone, enforced below the application by database triggers.
3. Rotating a link invalidates the old link and every old session; revoking ends access without deleting project history.
4. A stale reply composed before logout, rotation, or revocation is rejected and adds nothing.

Milestone 1 live checkpoint (2026-09-09/10)

The first live checkpoint: the owner drove their own browser against the local production build on 127.0.0.1:3100 (commit 71c8ee2, run id checkpoint-m1-2026-09-09T211026Z, server running with the D052 local auth-bypass flag) with the seeded Chickpea project — root plus /pricing, /about, /privacy, Desktop and Mobile, nine ready captures and one failed Mobile-root attempt. The session spanned roughly 2026-09-09T21:30Z to 2026-09-10T02:40Z with intermittent engagement across three mid-session fix cycles; the owner did not state active time. Per the escapes-become-assertions rule, each finding below was written down before its fix was implemented. Overall outcome: conditional acceptance — capture and organization validated; the product is not yet usable because pins and pin comments do not exist (the very next milestone, D051). Verbatim closing statement:

okay, i thought we had something usable. this doesn't let us create the most fundamental thing - creating a pin with an annotion. definitely move onto that next.

M1-LIVE-1 — the stage must show the actual screenshot

• Given a project with ready captures in the editor workspace
• When the owner selects a capture and interacts with the stage
• Expected the selected capture's real screenshot renders in the stage
• Observed the stage was a text placeholder and clicking its controls did nothing. Verbatim: "okay. so, a thing is captured, but I click on the buttons and nothing happens. please review / explain."
• Outcome rejected as shipped; fixed mid-session (editor-capture-image-stage, commit b21385a), now formalized as VAL-CAPTURE-015.

M1-LIVE-2 — the entire capture is in view on first open

• Given a ready tall capture (the Chickpea pages run ~9k–13k px)
• When the owner opens it in the stage
• Expected the whole capture is visible at once, without scrolling
• Observed the capture rendered at natural size and ran far off screen. Verbatim: "the content is being successfully fetched/screenshotted! but it should be presented such that the entire page is in view. see screenshot."
• Outcome fixed mid-session (capture-stage-fit-view-and-hints, commit 0a27d77): fit-to-view is the default with a labeled natural-size toggle; the contract's VAL-CANVAS-002 now carries this as a user-directed requirement.

M1-LIVE-3 — the pin-drop mechanism must be discoverable from the page

• Given a ready capture on screen
• When the owner looks for how to drop a note on it
• Expected the page itself explains the interaction, and the mechanism exists
• Observed no pin mechanism exists yet and nothing on the page said so. Verbatim: "what hte heck is the interactin mechanism to drop a note? I can't figure it out? add instrucitons on teh page itself to make it self-documented"
• Outcome open requirement, captured as VAL-CANVAS-009 (self-documenting canvas interactions) for the canvas/pins milestone; the interim stage gained a plain-language hint in the meantime.

M1-LIVE-4 — wide captures must pan horizontally at natural size

• Given a wide Desktop capture viewed at natural size
• When the owner tries to scroll right to see the rest of the page
• Expected the stage pans on both axes; the document never overflows horizontally
• Observed the image overflowed the page and nothing scrolled horizontally. Verbatim: "this still goes off the page when expanded. i can't scroll right either - the page scroll seems fixed horizontally - so i can't view the entire page that's been captured." In the same exchange the "canvas update" hint jargon was impenetrable: "what does \"pins and comments arrive with the canvas update. Use natural size to scroll into fine detail.\" mean? i literally can't figure out how to create a pin and comment on it. tell me what to do."
• Outcome fixed mid-session (capture-stage-natural-scroll-and-plain-copy, commit 71c8ee2): the scroll region is bounded on both axes (minmax(0, 1fr) tracks, min-width: 0 on the detail and scroll containers) and the hint now says plainly that pinning and commenting are not available in this build yet.

M1-LIVE-5 — owner-driven retry recovers a capture end to end

• Given a failed capture attempt sitting next to ready siblings
• When the owner presses the variant's Retry control themselves
• Expected one fresh attempt is dispatched through the real provider and appears as a new ready version
• Observed exactly that: the owner triggered "Retry Desktop capture" unaided and the retry produced a ready Version 2 end to end through the real Browserless provider, with no worker intervention.
• Outcome accepted behavior — positive evidence for the retry and dispatch-driver design (VAL-CAPTURE-008, D049).

Published boundaries

Every runtime boundary is exported exactly once from src/lib/boundaries/ (policy version 2026-09-23.3, constant POLICY_VERSION). Unit tests import the same constants and compare them against this page, docs/ARCHITECTURE.md, and the deployed /reqs routes; any drift between code, docs, and deployed content fails the gate, and duplicating one of these literals anywhere else in the application is a defect.

Editor session

URL and input limits

URL normalization fixtures

Normalization trims surrounding whitespace, rejects rows over MAX_URL_BYTES, then parses once with WHATWG URL, lowercases scheme and host, converts IDNA hosts to punycode, strips a single trailing host dot, removes the default :443 port and the fragment, resolves dot segments, treats backslashes as path separators, drops an empty query, and turns an empty path into /. Path case, percent-encoding (so %7E and ~, and encoded separators such as %2F, stay distinct), and query order/duplicates are preserved. Canonicalization happens before any address check, so every IP literal — including numeric spellings and private, link-local, carrier-grade, multicast, and metadata addresses — is rejected as ip-literal, while hosts that cannot exist publicly (single-label names and the reserved localhost, local, internal, intranet, home.arpa, invalid, test, and onion suffixes) are rejected as not-public. Public cross-origin HTTPS rows are allowed. These exact fixtures pin the behavior:

/pricing and /pricing/ normalize to different pages; https://example.com and https://example.com/ normalize to the same page. Rejections happen before any capture attempt or project row exists.

Capture dimensions, time, and bytes

Capture destination admission

A capture target is admitted only when it canonicalizes to an absolute public HTTPS URL and resolves publicly. Canonicalization runs first, so every IP spelling is rejected as a literal before any range check. DNS then follows a bounded CNAME chain and queries both A and AAAA: the host is admitted only if every answer parses and every answer is public. A timeout, a server failure, an ambiguous answer from either family, an empty result, a loop, or a single non-public answer rejects the host. Rejections never disclose a resolved address.

Every top-level redirect hop is revalidated under the identical scheme, credential, port, canonical-host, DNS-answer, and address-range rules, up to MAX_REDIRECT_HOPS; a chain that stays public completes and the capture stores both the requested and the final public URL.

Browserless runs in a different network, so application-side DNS proves nothing about what the remote browser resolves. The capture function therefore installs a request guard that revalidates every top-level navigation and refuses subresource requests to credentialed hosts, reserved hosts, non-HTTP(S)/WebSocket schemes, and all IP-literal hosts, while ordinary public subresources continue to load. The guard never disables web security, TLS validation, sandboxing, or the provider's private-network blocklist.

These prefixes are never a capture destination:

Capture attempts, concurrency, and staleness

Concurrency admission is durable: capture_leases holds exactly MAX_ACTIVE_CAPTURES slot rows in Turso, and claiming a slot is one atomic conditional upsert, so the limit holds across browsers and application instances. When every slot is held, dispatch fails as quota-exceeded (HTTP 429) without consuming the attempt — it stays pending and any later authorized client can resume it. A lease releases when its attempt reaches a terminal state and is reclaimed in place once it expires; a late release can never free a slot another attempt already reclaimed. Polling issues hierarchy GETs only — it can never create, duplicate, or mutate an attempt — and stops on any terminal or computed-stale state.

DOM manifest schema

Supported motion and tolerances

Capture freezes or pauses what it safely can from the same stabilized layout state as the screenshot, and warns on the rest:

Exactly what each policy does, so the result is reproducible rather than merely still:

• frozen — capture-only CSS sets animation: none and transition: none, so the element renders its base style instead of whichever frame the clock happened to be on. Freezing in place would be still but not reproducible.
• hidden — caret-color: transparent, which removes the blinking caret without moving or resizing the field.
• paused — every document.getAnimations() animation is paused and rewound to time zero; every <video> is paused. A rewound Web Animation is reproducible; a paused video frame is not, so video regions are excluded from reproducibility comparisons.
• first-frame — an animated image is covered by a canvas holding the frame createImageBitmap decodes, which is the format's default (first) frame. When the bytes cannot be re-read (opaque cross-origin response), the capture warns instead of claiming a freeze.
• unsupported-warn — a visible <canvas> may be repainted by script at any moment; capture cannot freeze it and says so in a warning.
• as-rendered — sticky and parallax layers are captured where they sit after the scroll returns to the top, and the capture warns that their position is scroll-dependent.

Every warning code is derived from this matrix (motion-unsupported:canvas-js, motion-as-rendered:sticky-parallax, motion-first-frame:animated-image), so no capture can invent a motion warning outside the published cases.

Capture outcome catalog

Every capture outcome maps to exactly one row: its persisted status, whether retry is offered, the HTTP status of the API response (— marks a computed or persisted state rather than a response), whether it consumes a persisted attempt, and whether it surfaces as a warning on a ready capture. Public messages and remediation text live in the catalog and are bounded by MAX_PUBLIC_MESSAGE_BYTES | 256 bytes.

Private asset delivery

Screenshots are stored in a private Vercel Blob store; the only way bytes reach a browser is GET/HEAD /api/captures/<captureId>/asset, which verifies the actor's authority on every request — including requests that end in 304, 206, or an empty HEAD — and never redirects to or names the provider. A capture id resolves through its project: a ready capture of a live project with a complete storage record is served; every other case (anonymous, expired, tampered, or ended authority; nonexistent, pending, capturing, failed, deleted-project, or integrity-failed captures) receives the one bounded generic denial with no bytes. Served bytes are revalidated against the persisted content type, byte length, and SHA-256 before anything is returned, and the strong ETag is exactly that SHA-256. A single bytes=<start>-<end?> range is honored (206 with Content-Range); suffix, multi-range, reversed, and non-numeric ranges are rejected without bytes, and an unsatisfiable range answers 416 with the published length. If-None-Match (weak forms and * included) and If-Modified-Since are answered from the persisted hash without a provider read.

Geometry minimums

Login and reply quotas

Both quotas are enforced in the durable store, so they hold across tabs and application instances, and both recover after exactly the published window.

Login enforcement semantics (D098, superseding the single shared bucket of D026): every attempt is counted in two rate_limit_buckets rows before the password is checked — a per-client row keyed by the SHA-256 digest of the editor-login scope and the client address (Vercel's x-real-ip, else the first x-forwarded-for entry, else a fixed unknown), limited to LOGIN_MAX_FAILURES, and a global row keyed by the scope alone, limited to LOGIN_GLOBAL_MAX_FAILURES. Keys are digests; a password or secret is never part of one. Reserving before verifying means parallel requests cannot multiply the guesses a window allows, and the per-client row means one guessing address cannot lock the editor out, while the global row still bounds guessing spread across many addresses. The window is fixed at the first attempt in it; throttled attempts receive the same generic 429 with a bounded Retry-After header, never reach the password check, and never extend the window. A correct attempt succeeds immediately once window_started_at + LOGIN_WINDOW_MS has passed; a successful login clears its client row and returns its one slot to the global row.

Feedback, annotation, and interaction limits

Performance protocol and budgets

Quality attributes under test

• Warm, playful, focused design; the whole product demos live in under a minute.
• No horizontal page overflow, clipped controls, or hover-only content at any supported width.
• Keyboard-operable controls with visible focus; reduced-motion preference respected.
• Bounded, actionable error states that never leave silent partial project state.
• No application secret or founder capability in any public response, asset, log, or committed file.