Fifteen towns, fifteen ways of publishing the same kind of record.
PermitParser exists because a roofing permit filed in Branford and one filed in Wallingford describe the same kind of event (a homeowner spending money on their house) but live in two completely different systems, in two completely different formats, neither of which is built to be queried by anyone who isn't standing at a town hall counter.
01. A data platform showcase
Municipal data is the purest possible test of resilient ingestion: there is no API and no shared schema, and the vendors do not talk to each other. One town exports a PDF with a real text layer; the next scans theirs from paper. One publishes a table; another hides everything behind a JavaScript single-page app that explicitly asks not to be crawled.
Every piece of this system exists to survive that without a rewrite every time a new town looks slightly different: a raw/normalized table split so a parser bug never means re-scraping, a content-hash ledger so a re-run costs nothing, a portal-type interface so a new town on an already-supported vendor is a config row, not new code.
02. A lead-generation engine
A newly filed roofing permit, solar installation, or commercial addition is one of the warmest signals in local real estate: a homeowner who is, right now, already spending money on their property. Contractors, installers, and investors currently get this by checking fifteen town websites by hand, or not at all.
PermitParser publishes the same normalized data as public files and a search over them. Filter by municipality, status, date range, or a straight-line radius around a filing, so that signal is a link, not a weekly ritual of clicking through town portals.
Who this is for: a contractor or installer who works three or four of these towns and wants this week's filings in one list; an investor watching a category, a valuation band or a radius around a property; a town resident who would like to know what the commission decided on Tuesday without downloading the PDF.
Coverage grows one town at a time, and only where the portal permits automated access. If your town should be next, or you would use a weekly filtered export, say so: on LinkedIn.
Held to the standard of "would I ship this."
| Layer | Choice | Why |
|---|---|---|
| State | One SQLite file + content-addressed raw store | Hundreds of filings, not millions. A file the collector opens, updates and closes beats a database server that sits idle all day; it moves to a bucket between runs. |
| Serving | Static files, searched in the browser | Every filing is one JSON file under 200 KB. Text, town, date and radius (haversine) search run on your device, so there is no query server to keep up or rate-limit. |
| Validation | Pydantic v2 | PermitRecord is the hard boundary between whatever a source's HTML or PDF says and what the schema is allowed to store. |
| Scraping | httpx + BeautifulSoup, portal-type dispatched | Static-first. A JavaScript-only portal gets a browser step; a plain HTML table doesn't pay that cost for nothing. |
| PDF / OCR | pdfplumber, PyMuPDF + Tesseract fallback | Most municipal PDF exports have a real text layer. Some scanned agendas don't. Both paths are built and tested against fixtures, not stubbed. |
| Scheduling | One batch job that exits | Each run scrapes the towns that are due, publishes the files and stops. Nothing polls a queue every fifteen minutes to find out there is nothing to do. |
| Logging | structlog | Every line carries bound context (municipality, document hash) so a fleet of scrapers is debugged by grep, not by guessing which town failed at 3am. |
What this is not, yet
Branford's own building permits are not in this system.
Branford's real permit system is OpenGov Permitting & Licensing, a
JavaScript application with no server-rendered data to parse, and
its robots.txt explicitly disallows automated access. What's
live today is Branford's Planning & Zoning Commission agenda and
minutes archive, a separate, legitimately public, statically scrapable
source. It's a working pipeline against a public Branford source, just
not the one a JavaScript portal declined to allow.
The rest of the fifteen-town target list is not built yet. Each one needs its portal found and verified before a scraper is written for it. This project does not fabricate coverage it hasn't earned.
Every design decision has its reasoning written in
Docstrings that explain the "why," not just the "what." The same standard as every other engineering decision on this page.