Day 3 — Poison PDFs die in a subprocess

Landed: 2026-07-03 · Verified: 9 PDF integration tests green on real fixtures; the worker survives a poison file mid-batch

Non-negotiable. A poison PDF must not take down the ingestion worker — it has to die in an isolated subprocess and land in a dead-letter queue with a reason, never crash the service.

The goal

Route each PDF page through the cheapest path that works: native text extraction first, OCR only for scanned pages, vision only for image- or vector-heavy pages. And parse every PDF inside a timeout-bounded subprocess, so a file that kills the underlying C library kills a child process, not the worker.

This day was deliberately a spike: prove the riskiest part of reading a customer PDF, not build the full ingestion pipeline (that is Day 4).

The story

Plan

Built

Tested

What broke

The fix

Green

1. PlanProve the riskiest part of reading a customer PDF: a per-page cheapest-path router, and a crash boundary so a poison file can never take the worker down.
2. BuiltA per-page router — native text first, gated OCR (Tesseract) only for scanned pages, vision only for image-heavy pages, with an image-hash dedupe before any vision call. All parsing inside a timeout-bounded child process; a poison file lands in a DLQ-shaped dead result with a distinct reason (malformed / encrypted / timeout), terminal and never retried.
3. Tested9 PDF integration tests over real committed fixtures (scanned, diagram, poison, encrypted), plus a batch proof: healthy file, poison file, healthy file — the parent process survives the poison and keeps working.
4. What brokeThe reviews and follow-up work caught 4 real things: a duplicated test helper in the diff; the configured model id did not actually exist (404 everywhere); embeddings at the chosen dimension came back un-normalized; and the type checker had never covered the test suite — 4 latent type errors were hiding there.
5. The fixThe helper was shared and the failure-reason construction centralized; the model id was corrected with a regression test pinning it; normalization was made mandatory and recorded as an architecture decision; the type-check scope was widened and all 4 errors fixed properly — no casts, no ignores.
6. Green9 PDF tests in 18.29s. 32 integration tests pass. Unit tests grew 84 → 89 across the window; the type checker now covers 74 files, up from 46.

What actually shipped

• A per-page router. Native text extraction first; OCR (Tesseract) only when a page looks scanned (the heuristic is named constants, not magic numbers); vision only for image- or vector-heavy pages. A SHA-256 hash of the page image is checked before any vision call, so the same image is never described twice — cost discipline built in from the first day it could matter.
• A subprocess crash boundary. Every PDF is parsed in a child process under a hard timeout. A poison file kills only the child; the parent raises a typed error with a distinct reason — malformed, encrypted, or timeout — and keeps going.
• A DLQ-shaped result. A poison file lands in a dead result carrying its reason: terminal, never retried, and nothing gets written to the Day-2 tables for it.
• Real committed fixtures — a scanned page, a diagram page, a poison file, an encrypted file — plus a reproducible generator, so the proof re-runs anywhere. CI installs Tesseract and runs the whole spike in the integration job.

Also landed in the Day-3 window

The days after the spike merged a series of smaller PRs that count as Day-3 work:

• One rules file, two new review skills. The agent working rules were consolidated into a single source of truth, and the self-review gate grew from three skills to four (a question-pass that interrogates every choice and forces every stub into a tracked ledger). A codebase map and a stub ledger were seeded — every placeholder in the code now has a close condition, a real test that will prove the real thing, and a fail-closed guard.
• Per-component secrets. The backend reads exactly one env file, anchored so the working directory can never change which file loads; the cloud credential is typed, registered with the secret-on-server guard, and the key file is git-ignored by pattern so it cannot be committed.
• Decisions live in one log. The architecture decision records were reconciled into the single decision log; stale docs that described a superseded pattern were corrected or marked historical.
• The model config was locked in — after a real pre-flight. The generation model, location, and embedding dimension are code-defaulted and env-overridable, with a regression test pinning the defaults. What that pre-flight caught is told in the story above.
• The type checker now covers the tests. CI had been type-checking only the source tree, so errors in tests could never fail the build. The scope was widened to tests and scripts, and the 4 pre-existing errors found there were fixed properly.

Honest wrinkles worth telling

1. The live vision call is deliberately sequenced. Day 3’s job was the crash boundary and the routing, and that is what shipped: the decision logic that flags a page for vision, the image-hash dedupe so the same page image is never described twice, and the model name read from a single source in settings — all built and tested. The prerequisites are settled facts: Vertex is the chosen access path to the Gemini models, the credential is wired through the app’s settings, and a live probe confirmed exactly which model ids are actually reachable (the full story of what that probe caught is in the steps above). The one live model call lands when ingestion runs end to end. Until then the seam sits in the stub ledger under a fail-closed contract: a real run must raise rather than ever invent a description.
2. The poison fixture is a catchable parser error, not an OS-level segfault. Crash isolation is still proven — the timeout path forcibly kills the child process and the parent survives — but this page won’t claim “segfault-tested,” because it wasn’t.

Architecture snapshot

What Day 3 added: the PDF reading spike

Per-page router

native text → gated OCR (Tesseract) → vision; scanned-page heuristic as named constants; SHA-256 image dedupe before any vision call

Subprocess crash boundary

all parsing in a child process under a hard timeout; poison kills only the child; typed reasons: malformed / encrypted / timeout

DLQ-shaped dead result

terminal, carries its reason, never retried; no rows written for a poison file

Vision describer seam

routing, dedupe, and settings-sourced model name built and tested; the one live call sequenced for when ingestion runs end to end

Tenant isolation layer (Day 2)

untouched — Day-3 code never sees a tenant id; extracted content flows into it on Day 4

Full ingestion pipeline (Day 4)

not yet built — chunking, embedding, and indexing consume this spike next

solid box = built this daydashed box = rest of the system (context, not new)

Proof

9 PDF integration tests: green in 18.29s (real fixtures, real Tesseract, real subprocess)
Crash isolation batch: good → DONE, poison → DEAD (malformed), next → DONE — parent survived
Quick-gate: green (84 unit tests at the spike; 89 by the end of the window)
Integration: 32 tests pass
Type-check: clean over 74 files (was 46 before the scope widening)

Screenshot placeholder — reserved for a future GitHub Actions run screenshot of the PDF spike job, if the owner wants one alongside the pasted output above.