Case study · AI workflow operations
Lighthouse — an AI-orchestrated discovery and evaluation pipeline by Carlos Vasquez
An autonomous pipeline that ingests ~1,000 records per run from eight fragmented public data sources, deduplicates, validates, and scores each on four audited axes, with per-score confidence tags and visible reasoning. Designed, QA’d, and operated by one person directing an AI agent. Three autonomous runs a week, under $2/month.
8
public sources
~1,000
records per run
4
audited scoring axes
temp 0
deterministic evaluation
$0.27
per run · <$2/month
3×/week
autonomous, watchdog-supervised
15+
silent failures caught in QA
The problem
High-volume public data is noisy in specific ways: aggregators re-host records under the wrong identity, landing pages masquerade as real entities, duplicates multiply across sources, and records go stale without announcing it. I wanted a system that ingests from eight sources, proves each record is real before spending evaluation budget on it, scores with reasoning a human can audit, and hands back a short, ranked, finishable queue, with every filtered record logged with its reason. Nothing fails silently.
The system
Multi-source ingestion across structured JSON APIs, aggregator endpoints, and public classifieds, never ToS-violating scraping. A validity gate screens every record before scoring, built on a finding from live testing: extrapolated scoring is most generous exactly when the input is emptiest. A landing page scores 9/10 because nothing contradicts the model’s optimism. Records that survive pass through identity-hash deduplication (preferring primary sources over re-hosts), then a two-part fit model: direct match plus extrapolated capability, scored at temperature 0 for deterministic, reproducible output. A structured disqualifier taxonomy separates hard evidence from soft inference: inference penalizes, only evidence disqualifies, and unknown is never treated as bad. Results land in an interactive HTML dashboard with one-click verdict capture, feeding a weekly scorecard that retunes ranking weights from logged human judgment.
~1,000 records in → short, ranked, finishable queue out · every filtered record logged with its reason
Specimen: one scored record
From a real run. Identity masked; gate verdict, scores, confidence tags, and reasoning are verbatim system output (trust reasoning excerpted).
RECORD u_9f3a17 — via aggregator API · scored v3 · temp 0
VALIDITY GATE ....... PASS (verified primary posting, full body 5,612 chars)
RELEVANCE—DIRECT .......... 8.5 [measured]
“The record explicitly requires advanced production/fabrication expertise — color management, press-ready file preparation, proofing standards, material specs, vendor coordination — which directly aligns with the profile’s expert-level production operations capabilities.”
RELEVANCE—EXTRAPOLATED .... 9.0 [measured]
“Proven at-scale program management (30 concurrent projects, $3.5M aggregate budgets, vendor/stakeholder coordination) maps directly to building the coordination-and-delivery model this record describes. Portfolio provides direct evidence of managing complex production for national brands.”
VIABILITY ................. 9.0 [measured]
“Near-perfect match to core strengths; the pool of profiles with this specific blend is likely thin; geography ideal.”
TRUST ..................... 6.0 [measured]
“Well-established organization — however, a legally required disclosure is omitted from the posting. Flagged under the applicable transparency…”
The operations story
The pipeline was built in under a week by directing an autonomous AI agent through checkpoint gates: small-sample proofs before any source scales, raw-evidence verification of every claim, cost approval before every full run. The QA record is the point: 15+ silent failure modes caught before they corrupted output, including a 300-character truncation starving the evaluation layer, merge logic that silently doubled records, aggregator pages masquerading as source entities, and an API the agent’s own tooling had falsely declared dead. The standing rules that came out of it now govern every run: ground-truth against vendor reference implementations, 10-record raw samples before ingestion, verified backups before destructive operations, watchdog crons on all background jobs.
From the QA log
Four entries, verbatim. Each one bought a standing rule.
[stage2.extract]
WARN: description_len=300 for 209/209 feed records — fetcher truncating at 300 chars; classification layer starved upstream. Root cause: summary field ingested instead of full body. 923 records re-fetched; every downstream score recomputed.
[stage1.fetch]
404 on /search across 3 runs, all path variants — agent declared the API dead. Ground-truth vs vendor reference implementation: 200 OK; endpoint renamed /search-v2. Standing rule added: a tool’s failure is not the world’s state.
[stage3.score]
9.0 awarded to a forum thread; audit found 42/42 search-snippet records lacked posting signatures. Finding: extrapolated scoring is most generous exactly when input is emptiest. Validity gate added — no verified record, no score.
[ops.backup]
Cleanup wrote a partial set over a stale backup — dataset briefly existed nowhere. Recovered via deterministic re-score ($0.27, temp-0 idempotency). Standing rule: verified, timestamped backup before any destructive operation.
The house rule: a tool’s failure is not the world’s state.