Caldrop: measure before you build

A product idea hinged on one question we could not honestly answer from a desk: is this even buildable at scale, and what would it cost? The tempting move is to assert plausible numbers and start coding. Instead we ran a bounded spike, produced measured figures for coverage, cost, and error, and named precisely what a small sample could not settle. The result is a go decision you can audit. This is the clearest example of the one habit that runs through all of our work: measure, do not assert.

The question

Can you programmatically resolve and parse the current academic calendar for every US public school district at scale, and at what coverage rate, cost, and dangerous-error rate? Building first to find out would have burned weeks. So we scoped a spike, not a product: just enough to turn three unknowns into measured numbers, and to be honest about which ones the sample could not yet answer.

The method

We built the real national spine, proved the full resolve-to-parse loop on a deliberately diverse sample, then modeled the cost in token math so the answer would hold regardless of the one input we could not yet pin down.

1. The spine, and the surprise inside it

Phase 0 built the district directory from the federal NCES Common Core of Data, served through a free public API. That gave a hard count to work against: 19,714 total agencies, 13,407 regular operating public districts, and 718 in New York. The first surprise arrived immediately. The directory carries no website field, so essentially every district has to be search-resolved. That is the realistic hard case, not a shortcut, and we chose to confront it.

The second surprise is where the difficulty lives. 48% of districts (6,456) enroll fewer than 1,000 students; 38% sit between 1k and 5k; only 283 are above 25k. Whatever breaks at scale will break in the small-district long tail, which is exactly where templating and clean web infrastructure thin out.

2. The resolver, and an honest funnel

The resolver issues a search query per district and ranks the returned URLs with a heuristic judge that down-ranks aggregator sites. Its cost is the one expensive input in the whole system, so we did not estimate it. We measured it: $0.0015 per district, ten districts for one and a half cents.

The funnel it produced is the most honest number in the study, because it refuses to flatter the approach:

A naive "find a PDF" resolver clears only 4 of 10. Getting to high coverage means building an HTML parse path and a second-hop step. That is the difference between roughly 40% coverage and a real product, and we named it rather than hiding it behind the cases that worked.

3. The parser, and the load-bearing surprise

We parsed four calendars spanning every artifact subtype we hit. The cheap path works: a text-layer "legend" PDF parses with pdftotext at essentially zero cost and got the first and last day right every time. The expensive path also works: an image-only scanned PDF parses correctly through vision.

The surprise sat between them and would have sunk a naive cost model. One district's PDF has a text layer (488 text operations) but the actual dates live in color-coded grid cells, so plain extraction returns the legend and not the start and end dates. "Text-layer PDF" is not one thing. That single observation is why the cost model is built around a vision fraction rather than assuming the model is rarely needed.

The event schema is real

Each parsed calendar emits typed events with a per-event confidence score and source provenance. The first and last day of school are flagged as the load-bearing fields, the ones a parent always knows by heart and the ones a confirm step has to get right. One real parsed event, with credentials and internal paths removed:

{
  "district": "Cherry Valley-Springfield Central School District",
  "school_year": "2026-2027",
  "artifact_subtype": "pdf-image",
  "events": [
    {
      "title": "Classes Begin",
      "start": "2026-09-08",
      "end": null,
      "all_day": true,
      "type": "first_day",
      "applies_to": "all",
      "confidence": 1.0
    }
  ],
  "dangerous_events": ["first_day", "last_day"],
  "parse_confidence": 1.0
}

The cost model

Every figure here is reproducible token math against confirmed per-token pricing, plus the one measured input (the search cost). It runs with no network and its output matches the report to the cent. Crucially, it is shown across a range of vision fractions, so the conclusion survives the one thing we could not yet pin down.

The full national one-time pass costs roughly $82 to $157, and New York roughly $6. Cost does not constrain this product at any plausible scale. Which is the point: it removes the cheap objection and forces the decision back onto the two things that actually matter, coverage and the error rate.

Failure modes, observed and fixed

These are not theorized risks. Each one was hit during the ten-district run, and each has a fix. One fix was verified end to end during the spike.

The verified case: a New York district that first collided with a same-named district in another state was recovered by appending its directory geography to the query. The aggregator sites that already rank on these searches are a concrete, beatable competitor class, not a vague threat.

The decision

Why this is the centerpiece

A skeptical engineering leader should finish this page and conclude, without being told, that we de-risk before we build. The whole study turns on what we refused to do: we had the one number everyone wants (0 of 4) and we declined to dress it up as an accuracy claim. We measured what was measurable, cost, decisively, and we named what the sample could not yet prove, coverage and the error rate, as the next measurements rather than guesses. That is what "measure, do not assert" looks like when it is expensive to follow.

This is a bounded R&D spike, not a finished product. Every figure on this page traces to the spike's findings report dated 2026-06-01 or to a live run of its cost model. What the team built after the go decision is separate work, deliberately not folded into the spike's numbers.