How we measure detection accuracy

Faultlines emits two layers per repo — L1 engineering features (code-grounded, deterministic) and L2 product features (customer-facing, Sonnet-synthesised) — plus user flows attached to both. Every number on the landing comes from a fresh cold-scan compared against a hand-curated golden corpus.

Scan scope today

We publish full scans on 25 production OSS repositories in the landing carousel, plus dozens more private and dogfood scans not surfaced here. Of those 25:

• 16 trained corpus repos — hand-curated L1 + L2 + flow truth files, used to tune every Stage / extractor in the pipeline. Numbers below are computed against these truths.
• 7 unseen generalization repos — truth curated AFTER the engine was frozen, sourced from each product's public marketing surface. No engine change targeted these repos. Used to confirm we learned the problem class (route extractors, MVC, schema, workspace boundaries) rather than memorised the corpus.
• 2 internal dogfood scans — our own faultlines-app + a private soc0 repo we use as a stack-detection regression target. Not included in the precision/recall averages.
• Plus the deterministic pipeline runs on every repo. Stage 0 → Stage 6 (intake, audit, extractors, reconciliation, flow detection, metrics) needs zero training data — only the Sonnet L2 rollup is tuned against the trained corpus. So coverage, health, hotspots, impact, and behavioral coverage work identically on a repo we've never seen before.

What the numbers mean

Per-repo · trained corpus

Per-repo · unseen (generalization eval)

How the golden corpus is built

1. External triangulation. Marketing site, public docs sidebar, pricing page, llms.txt — what the maintainer tells paying customers the product does.
2. Code triangulation. File-system routing (Next.js App Router, FastAPI routers, Rails routes), schema files (Prisma / Drizzle / Django models), package manifests, MVC controllers, workspace member layouts.
3. Engineering-grain expansion. When the codebase ships per-subsystem folders (one per auth method, one per PAM resource, one per crypto operation), each becomes its own truth entry. We grade the engine for finding the granular reality of the code, not a marketing abstraction. This is a hard rule — we never retro-fit truth to match engine output (that would be cheating; the measurement becomes self-referential).
4. Aliases. For every truth entry we record the engine's plausible naming variants (Title Case, kebab-case, abbreviations). Matching is alias-aware and stemming-aware.
5. No README parsing. Hard rule per CLAUDE.md. README is the maintainer's pitch — too biased, too aspirational, too dependent on author's writing skill. External web surfaces (the live product's marketing site as customers see it) and code structure are the only inputs.

How each scan is run

• Cold-scan only. The per-repo assignments cache is scrubbed before every measurement run, so a prior result can't bias the next one. If you skip the scrub, evaluation becomes a re-measurement of the last scan's bias — useless.
• 9-stage deterministic-first pipeline (pipeline_v2). Stage 0 git + filesystem intake → Stage 0.5 LLM stack auditor → Stage 0.6 deterministic shape classifier → Stage 1 parallel extractors (route, MVC, schema, package, FastAPI, Rails-suite, Go-router, Rust-workspace, JS-library, Python-library, …) → Stage 2 reconcile → Stage 3 Haiku flow detection (with dynamic wall-time budget, scales with feature count) → Stage 4 residual LLM clusterer → Stage 5 post-process → Stage 6 metrics + behavioral coverage → Stage 6.5 deterministic Layer 2 → Stage 7 metrics → Stage 8 Sonnet analyst + shape-aware flow rollup.
• Shape-aware Stage 8. Each repo is classified into one of 10 shapes; a strategy registry picks the right flow-attribution algorithm. Turborepo monorepos use workspace-prefix matching; OSS libraries use Sonnet'smember_flows semantic attribution; backend monoliths use MVC controller grouping; the universal-residual fallback uses entry-point-in-paths + 50%-overlap as a safety net.
• LLM stack. Anthropic Claude — Haiku 4.5 for high-throughput stages (flows, residual), Sonnet 4.6 for the Layer 2 analyst. Auto- streaming on everymessages.create so long-output stages never silently fail.
• Deterministic scoring. Output JSON saved to ~/.faultline/feature-map-<slug>-<run-id>.json and scored by eval/detection_eval.py. Stemming + alias matching; consolidated matches (e.g.transactions ↔transaction) count toward recall but not as phantoms toward precision.

Behavioral test coverage (no test runs needed)

Most OSS repos don't ship lcov reports, so we built a deterministic 7-signal coverage estimator that works on any clone with zero setup.

• 7-signal composite (per-feature, per-flow). co_change (0.35) + bug_fix_test (0.25) + freshness (0.15) + density (0.10) + co_author (0.10) + ci_workflow (0.05) + commit_msg (0.05) — weights sum to 1, each signal clamped to [0, 1], composite clamped to [0, 1]. Per[behavioral-coverage-analyzer] skill in the repo.
• Confidence band. high (5+ signals active > 0.1), medium (3-4 active), low. A high-confidence 50% coverage means the same thing algorithmically as a high-confidence 90% — different repos, different real coverage, both measured the same way.
• Test-type classification. Every test path classified asunit/integration/e2e by regex on the file path (cypress/, playwright/, e2e/, integration-tests/, *.cy.*, *.e2e.*). CI workflow files scanned for known runners (jest, vitest, pytest, go test, cargo test, rspec, etc.) — theci_workflow signal fires if any runner is wired.
• Stage 6 wires it in. Stage 6 metrics tries to import the privatefaultlines-test-coverage provider; falls back to the OSS density-only path when absent. Output:coverage_pct per feature + per flow on every scan.

Every metric we emit — sell-yourself catalog

One scan produces all of these. Each card answers four questions: what it measures, what gradations mean and why, how to raise it, and what you actually do with the number.

Runtime integrations — Sentry + PostHog

Two optional integrations enrich every feature with how it actually behaves in production. Zero SDK changes. Read-only API tokens. We match events to features through the same path_index the engine already computes.

Hotspots

Hotspot = feature whosebug_fix_ratio exceeds 30% with at least 3 total commits (the floor filters out newly-added features with only 1-2 commits). The landing surfaces the top 5 per repo, sorted by ratio descending. Health score (0-100, composite of churn + bug ratio) is shown alongside.

Honest caveats

• L1 precision on some repos sits at 10-30%. This is not an engine bug — it's theengineering-grain rule in action. The engine emits real per-subsystem features (e.g.scim,audit,secret-rotation for infisical); the truth file enumerates only the maintainer's top-level pitch. We publish the gap honestly rather than retro-fit the truth to match (that would be cheating).
• Flow precision varies widely. Backend-heavy repos can over-emit flows from the same entry-point file; the Stage 3 dedup pass (Sprint S7-B) catches most but not all. Sprint S11 added a dynamic wall-time budget so big repos (chatwoot 330 features, directus 242) don't time out anymore — chatwoot went from 0 / 330 features with flows to 132 / 330 after the fix.
• Sonnet outputs are not strictly deterministic even at temperature defaults. A single re-run can shift L2 P/R by ±5-15pp. Trend over many scans is what matters.
• Truth lists are versioned in our public repo. When we expand a list to reflect engineering-grain reality, the change and the rationale are commit-history visible.
• Independent generalization eval (7 unseen repos) sits within±3pp of the trained corpus average on both L2 P and R — the engine learned the problem class (route extractors, MVC, schema, workspace boundaries) rather than memorising the corpus. We re-run this whenever we ship a Stage that could overfit.