03 / AI evaluation
A new prompt. A different model. A higher score. Before calling it an improvement, I want to know what changed—and what the score leaves out.
One candidate uses simple rules to route support messages. The other always answers “other.” Both repeat their answers perfectly. Only one follows the expected labels across all four examples.
Choose a message and change what gets scored. The input-level result and the aggregate score answer different questions.
Expected: billing · escalate: yes
This message 100%
This message 0%
Repeat each message three times: both candidates are 100% consistent. Being reliably wrong is still being wrong.
Deterministic rules on four synthetic cases. An adaptation of VeroLoop’s offline example; no live AI calls.
The project
VeroLoop is a Python SDK for evaluating a typed AI task consistently across provider candidates. A task defines the inputs, expected output structure, examples, and repetitions. Candidates describe the provider, model, and prompt.
The evaluation layer then records what came back and evaluates it against the same contract. The current source repository is private.
Design decisions
Pydantic models define the canonical output contract. Provider adapters check whether they can support it. Unsupported schema features are surfaced as errors, rather than quietly changing the task.
Unknown cost stays unknown. Missing usage or pricing information should not be interpreted as a free run.
What exists
The SDK includes provider adapters, asynchronous execution, configurable concurrency and retries, repeated attempts, and console, JSON, and optional MLflow reporting. A deterministic offline provider makes examples and tests runnable without credentials.
Real-provider use requires explicit configuration. The project is still developing; its current scope does not include a model registry, automatic schema fallbacks, caching, or an LLM judge.
Reading this demo
This browser demo adapts the offline example using four synthetic support messages and two deterministic rules. Category accuracy scores one field per message. The combined score gives equal weight to category and escalation, for eight field comparisons in total.
The careful rules get all eight fields right. The always-other rules get one category and three escalation flags right: four of eight fields. The displayed scores are calculated from the cases as you explore them.
Both candidates give identical outputs over three repetitions. These are not measurements of OpenAI, Anthropic, Gemini, or any live model.
Keep exploring