Agent evaluation platform
Enterprises are deploying AI agents that take real actions, and most of them cannot answer a basic question afterwards: what did it actually do, and was that good? This is infrastructure for answering both.
The problem
Agents are being shipped without a way to judge them
A traditional service either returns the right value or it does not, and a test suite settles the question. An agent chooses a path. It decides which tool to call, how to interpret an ambiguous instruction, and when to stop. Two runs of the same prompt can differ, and both can be defensible.
That breaks the usual release process. Teams cannot tell whether a new model version, a changed prompt, or a new tool made the system better or quietly worse, because there is no stable definition of better and no record of what the agent did on the way to its answer.
In a regulated environment the problem is sharper still. If you cannot reconstruct why an agent did something, you cannot approve it for a workflow that touches patients, payments, or filings.
Architecture
Capture, evaluate, then gate the release
Capture has to be low friction or nobody instruments anything, so the trace format records what the agent did rather than requiring teams to restructure how they built it. Evaluation is deliberately two-track. Gating is where the value actually lands: a score nobody acts on is a report, not infrastructure.
This diagram describes the approach at the level already public in profile.json and on the portfolio. It is not the internal design document, and implementation detail is deliberately absent while the platform is pre-launch. Shridhar should review and extend this before it is treated as a technical description.
Position
Why this is not just a feature of an observability tool
The obvious objection, and the one worth taking seriously, is that this becomes a checkbox inside an existing LLM observability or MLOps platform within a couple of years. Tracing on its own probably does.
The part that resists absorption is the evaluation layer: rubrics that encode what a specific enterprise means by a good outcome, a human review workflow that survives audit, and the comparison logic that turns all of it into a release decision. That is closer to a quality system than a dashboard, and quality systems tend to be bought deliberately rather than inherited from whichever vendor already had the traces.
The bet is not settled. The agent-facing home page publishes a prompt that asks a visitor's own model to argue the opposite case.
Status
Where it actually is
| Stage | MVP in progress. |
|---|---|
| Validation | More than 50 enterprise discovery conversations. |
| Design partners | Three pilots underway. |
| Pricing | Seed-stage pricing validated within six months, meaning prospects named a number they said they would pay. Nobody has paid it yet. |
| Revenue | None. This is pre-revenue by design at this stage. |
| Role | Tech Advisor, alongside the day job at Moderna. |
Next
Open questions
- How much instrumentation will a team accept before they decide it is not worth it. This is the difference between a product and a science project.
- Whether rubric authoring can be made cheap enough that a domain expert writes them rather than an engineer.
- Whether regression detection can be made trustworthy enough that a team lets it block a deploy, which is the point at which this stops being optional.