CASE STUDIES / OWN PRODUCT / DATA PIPELINE WITH AN AGENT LAYER

A scoring pipeline that tests websites against AI models

18 deterministic checks Measured and 4 live AI engines Measured feed a single readiness score and a written report, with 139 test files Measured guarding every change.

The problem

The question behind the product: can AI assistants read a given website, and do they recommend it when a buyer asks? Answering that by hand means pasting the same questions into several chat apps, copying the answers into a spreadsheet and guessing at a score.

The pipeline does it end to end: it crawls the site, runs the checks, asks each engine the same questions, scores what came back and writes the report a business owner reads.

Constraints

Every score comes from a test that actually ran.
An engine that errors out is left out of the score, never counted as a miss.
A result set with gaps gets no score at all instead of a partial score.
Spend is capped per client and per engine, because every scan calls paid model APIs.
The report has to make sense to a business owner, not only to an engineer.

Who did what

As the product owner

Set the question the product answers.
Set the acceptance bar: no score without a test that ran.

I built

The crawler and the deterministic checks
The query layer across the AI engines, with spend caps and caching
Scoring, the tier rules and the PDF report
The job queue, retries, rate limits and error monitoring
The test suite and CI

My own product, so I was the client. No partner was involved in this build.

Step through a run

Sample run on a made-up bakery website

Crawl and check

Plain web requests, no AI calls. Redirects and response size are capped before anything is parsed.

Crawler rules OK:allow AI crawlers
Sitemap OK:found and valid
Structured data Fails:business details missing
Readable without scripts OK:yes

Ask the engines

The same questions go to every engine as background jobs. A question asked recently comes back from cache at no cost.

Perplexity OK:recommends the bakery
Claude Fails:recommends a competitor
OpenAI OK:recommends the bakery
Gemini Note:engine error, left out

Score

Only engines that answered count toward the score. A result set with gaps gets no score instead of a partial score.

Checks OK:most pass
Engines counted Note:the error is excluded
Tier Note:partly ready

Report

A PDF a business owner can read: what failed, why it matters and the fix, in priority order. Each result record stores the hash of the record before it, so an edited result shows.

Fix first Fails:add business details to the structured data
Then Note:ask the engines again after the fix ships

Timeline

Engine adapters to the first full run: 2 days Measured
First commit to a working self-serve purchase: 7 weeks Measured
Running in production: 6 months and counting Measured

Scope receipt

Build record

Milestone tier: Multi-step agent or workflow
Stack: TypeScript, Next.js and React, Postgres with Drizzle, Convex for the job queue, Upstash Redis for rate limits, Vercel scheduled jobs, Sentry, react-pdf, a Python worker
Evidence basis: Counted from the source and git history of the production repository.

Done-checklist

Every check runs without an AI call.
Every engine call sits under a spend cap that callers cannot raise.
An engine error is excluded, never scored as a miss.
A result set with gaps gets no score.
Repeated questions are served from cache.
A test mode returns fixed answers at no cost, so CI never pays for a model call.
Result records are hash-chained.
Lint, a type check, unit tests, a build and browser tests run on every change.

What I would do differently

A placeholder adapter for a fifth engine shipped inside the live set. It returned made-up answers, and they were counted as real until I caught it. Placeholders are now excluded from scoring by name. I would keep them out of the live set from the start.

A parsing bug stopped reading an error code at its first underscore, so real errors were scored as misses and some scores came out at half their true value. The fix was a test for each error format. I would write those tests before the parser, not after the bad numbers.

Have a client who needs this?

I build work like this for agencies and consultants, under your brand, at a fixed partner price per milestone.

Book a partner call